Süsteemihaldus 2019/20 kevad

Some instructions are intentionally not clear and have only generic action description. You should be able to put the pieces together with help from Apache manual and Google search.

During 23rd of March, the contents of this lab may change, as in we might add one additional subsection to this lab.

Please note that we might add additional new nagios tests during this weeks lab (regarding this lab)

0.Make sure you did finish all the tasks of the previous weeks.

We expect here:

• Personal domains are configured. Machine is accessible over <machine_name>.sa.cs.ut.ee inside the University network.
• All the tests on scoring.sa.cs.ut.ee are green.

Intro, this bit is instead of the slides.

This, like every other into into a lab, is again is a simple intro to the lab's topic, basically the basic knowledge in layman's terms. You are strongly encourage to dig into the topic by yourself.

Accessing data on the web

Let's say you want to access www.ut.ee. Your browser makes a request for www.ut.ee over the internet. And it usually receives a web page in return. Now this transaction about ask something and recive something is an over simplification. While all of the communication from asking for resources to reciving resources is in human readable text format, the request and responds do quite a few back and forths before the final web page that you requested ends up in your browser.

Client that sends out a message containing info about wanting something, makes a request, uses what is called a GET method. In short when the client utilises a GET method it means, it wants something (information, obviously) in return. All the the requests are done with a specific URL in mind, so that the client doing the request on your behalf would know which server to contact.

Servers that recive a request send a message back to the client, that is called a response. Response messages use POST method (oppsite of GET method), where the server wants to "give" you data, instead of GET method's "give" me data.

Requests and responds are mostly used to exchange data.

A simple request, utilising the GET method would look something like this:

In this example, the type of method is specified (GET). The HTTP specification is noted (HTTP/1.1). The URL that is the target of the request and additionally what types of languages are expected in a response.

A simple response to a web page request would look something like this:

Starting which HTTP specification is used (HTTP/1.1). Server status code is returned (200 OK). Type of data that gets sent with the response. Finally the page's html. Which usually initiates additional GET method requests for additional resources, like CSS and JS (and these requests get responses from the server in the form of POST requests.

In the following image the client makes several requests (Utilising the GET method) in order to display the whole page.

Data formats

The data that gets exchanged over the web could be in any number of formats. For example, all of the data that gets exchanged could simply be within the URL that gets passed along. We have all seen those URLs that visually look like they are over a meter long. That is also considered data exchange on the web. Sending and receiving files counts as data exchange on the web. So when your browser uses GET method to ask for something, the server that gets contacted might just send back a file, if the request asks for it. Data could also be in JSON and/or XML format. Basically <Key:value> pairs. Information regarding what type of data gets exchanged is always paired with the requests and responses. This bit is in the request/response header, so that the client could tell what kind of information it expects and the server could tell the client, what kind of information it is sending. This avoids situations, like, client asks for XML format data and the server just responds with a .jpg file. It would be a miscommunication.

Headers provide extra information about the request / response. For example a server would specifiy within a header that what type of information gets exchanged next. At the same time a client's header might contain a authentication token, which proves to the sever, that this clients is used by this person (said person would have to be allowd to exchange data with the server).

When a server responds to a request. The response would contain the requested information. The header would contain, what type of data it is. Also, the server would send back a status code. Most commonly a status code 200, which stands for OK success. Probably the most familiar status code is the 404, page not found. The error codes that begin with 4xx are considered mistakes on the client's behalf. Something you as the information requester have to deal with / fix in order to get what you look for.

Again, this previous part was not intended to be all encompassing, but just enough to give you the faintest idea and plenty of keywords for googling about, if it got you interested.

Quick clarification / reminder, if someone missed this bit of information. The stuff in orage boxes is what you have to do to:

The stuff in green boxes are the stuff you check if you did the stuff in orange boxes correctly.

At the end of the labs you might notice just a green box without a orange one preceding one, that box is there to let you check if all of the previously done work still works as expected.

Let's get started!

1.Setting up personal web server

Installing the updates

It has been a while we did not pay attention at the announced updates, let's check if there are any.

Understanding DNS CNAME records

Sometimes there is a need for one system to have multiple DNS names, all pointing to the same IP. This could be the case when hosting multiple virtual web servers or running multiple services on the same IP using service-specific host names, like:

• www.<vm_name>.sa.cs.ut.ee
• ftp.<vm_name>.sa.cs.ut.ee
• myservice.<vm_name>.sa.cs.ut.ee

We could create a number of A type records pointing to the same IP address. However the good style is to have a PTR type record for each A or AAAA type record, and if multiple A records point to the same IP, the reverse can be do only for one of them. Therefore good style in this case is to select only A record and assign it's PTR correspondingly, like:

• A : student-test.sa.cs.ut.ee -> 193.40.154.247
• PTR : 193.40.154.247 -> student-test.sa.cs.ut.ee

... and the rest of the host names which are pointing to the same IP address should be assigned using CNAME aliases:

• CNAME: www.student-test.sa.cs.ut.ee -> student-test.sa.cs.ut.ee
• CNAME: ftp.student-test.sa.cs.ut.ee -> student-test.sa.cs.ut.ee
• CNAME: myservice.student-test.sa.cs.ut.ee -> student-test.sa.cs.ut.ee

A CNAME record maps a name (alias) to another name (real name). Resolving CNAME record takes two steps - first, the alias is resolved to the real name, then the real name is resolved to the IP address (using A record). If a (real) name has multiple CNAME aliases pointing to it, one just needs to change the real name's type A record to "move" all the names to a new IP address.

There can be however not reverse record for CNAME aliases, the corresponding IP will be always resolve to the actual A record.Additionally the MX records and NS records can be only in use with A or AAAA records, we cannot assign them pointing to CNAME records.

More details about Canonical Name Record (CNAME) with examples you can read here ... http://www.zytrax.com/books/dns/ch8/cname.html.

Virtual Web Hosts Using Apache Web Server

Apache is the most popular web server software in use (as of March 2012). Apache supports wide variety of features and can be extended with modules. We will use the virtual host functionality to set up multiple web hosts on the same web server.

In CentOS 8, the main configuration file for Apache is /etc/httpd/conf/httpd.conf. Despite the fact that it is not really important, which conf file you use, it is recommended to keep different aspects of the setup in different .conf files. The additional conf files are kept in /etc/httpd/conf.d directory, i.e. ports.conf. This also offers better flexibility, when adding and removing different web applications and modules. When some parts of the configuration are kept in separate /etc/httpd/conf.d/*.conf files it is easier to navigate the setup and enable/disable blocks of instructions. For an example, directive Include mods-enabled/*.conf will load every .conf file in this directory and their instructions will be automatically included in the Apache configuration, everything else will be ignored. This way, adding or removing a bit of functionality can be just a matter of creating an appropriate my-functionality.conf file or changing the file extension to something other than .conf. For example - to my-functionality.conf.disabled without editing the actual .conf files

The log files for the Apache web server are located in the /var/log/httpd/ directory. Query (access_log) logs and error logs (error_log) are kept separately. It is recommended to have a separate log file for each virtual host (vhost).

Extra reading:

• The Apache HTTP server project
• Apache version 2.4 Documentation
• Red Hat specifics (Apache configuration guide)

During this lab we will configure Apache to use name-based virtual hosts. With name-based virtual hosting, the server relies on the client to report the hostname as part of the HTTP headers. Using this technique, many different hosts can share the same IP address.

First, configure the actual virtual host. We will create a virtual host for www.<vm_name>.sa.cs.ut.ee.

Apache httpd documentation:Virtual Host and more specifically Name-based Virtual Host

<VirtualHost *:80>
    ServerName InsertRightValueHere
    DocumentRoot /var/www/html/www.<vm_name>/public_html
    # Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
    LogLevel warn

    ErrorLog /var/log/httpd/InsertRightValueHere-error.log
    CustomLog /var/log/httpd/InsertRightValueHere-access.log combined
</VirtualHost>

If all was done correctly, your website default canvas should be ready. It should look something like this:

Try accessing your web server now:

• http://test.<yourdomain>.sa.cs.ut.ee
• You should see an empty page saying Index of /

You may have noticed that your page is empty. The next step is to create some content for the virtual hosts. In order to do that you need to be familiar with basic HTML tags and how web page is constructed. If needed refer to very good public tutorials here https://www.w3schools.com/html/html_intro.asp and https://www.w3schools.com/html/html_basic.asp .

Now it is time to restart the Apache httpd server.

There are situations when you want to restart your Apache server, but can't interrupt it's work. Imagine that you have few hundred clients currently downloading files from your server and you need to avoid disconnecting them. In such situations you can use the following command:

# apachectl graceful

This will gracefully restart your Apache server with new configuration without affecting your client's connections.

Utilizing webserver as a proxy

Very often, a web server is not utilized to serve files directly, but used for forwarding requests to another service that is a web server itself, but much less known and secure. A good example is a Python (Flask, Tornado), NodeJS or Go web applications. This is called proxying.

This allows a systems administrator to have a single point of control over security settings, without having to delve into the applications code.

In this example we will use Python Flask, but you can use anything that is capable of serving web. We will start a service on localhost port 5000, and proxy that to the internet.

Now you have the necessary libraries to run a flask program. You can now create a random file, for an example /root/server.py, and add the following code there:

#!/bin/env python3

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run(port=5000)

Once you have run the python program, test if it works by doing curl localhost:5000. It should answer with whatever the program is programmed to do.

Now, to set up the proxy, do the following steps:

<VirtualHost *:80>
    ServerName proxy.<vm_name>.sa.cs.ut.ee # Sets the name to listen for with requests
    ErrorLog /var/log/httpd/proxy-error_log
    CustomLog /var/log/httpd/proxy-access_log common

    ProxyPreserveHost On

    ProxyPass / http://localhost:5000/
    ProxyPassReverse / http://localhost:5000/
</VirtualHost>

Now, because you cannot keep a terminal open constantly to keep this web service up, nor is it a good practice to run things like this, we are going to use our service manager, to keep our python service up for us. This also allows the service to start again when the machine reboots.

 
# systemd unit file for the Python Proxy Service

[Unit]

# Human readable name of the unit
Description=Python Proxy Service


[Service]

# Command to execute when the service is started
ExecStart=/usr/bin/python3 /usr/local/lib/server.py

# Disable Python's buffering of STDOUT and STDERR, so that output from the
# service shows up immediately in systemd's logs
Environment=PYTHONUNBUFFERED=1

# Automatically restart the service if it crashes
Restart=on-failure

# Our service will notify systemd once it is up and running
Type=simple

# Use a dedicated user to run our service
User=proxy

[Install]

# Tell systemd to automatically start this service when the system boots
# (assuming the service is enabled)
WantedBy=default.target

Let's add a default WordPress setup to our current setup

Let's install the prerequsite package necessary for a default WordPress setup.

Start up the mariadb service

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none): 
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] Y
New password: 
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] Y
 ... skipping.

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

Now let's download the wordpress tarball, unpack it and move it to the a proper webroot.

Let's grant user apache the right to work in /var/www/html/wordpress directory. And change the file SELinux security context.

Now repeat the steps we did earlier for www.<vm_name>.sa.cs.ut.ee.

The final part of the WordPress setup is actually setting up the page, what we have done so far has been laying down the foundation, so to speak.