0.Make sure you did finish all the tasks of the previous weeks.

We expect here:

• Personal domains are configured. Machine is accessible over <machine_name>.sa.cs.ut.ee inside the University network.
• All the tests on scoring.sa.cs.ut.ee are green.

General web intro

Accessing data on the web

Let's say you want to access www.ut.ee. Your browser makes a request for www.ut.ee over the internet. And it usually receives a web page in return. Now this transaction about ask something and recive something is an oversimplification. While all of the communication from asking for resources to receiving resources is in human readable text format, the request and responds do quite a few back and forths before the final web page that you requested ends up in your browser.

Client that sends out a message containing info about wanting something, makes a request, uses what is called a GET method. In short when the client utilises a GET method it means, it wants something (information, obviously) in return. All the the requests are done with a specific URL in mind, so that the client doing the request on your behalf would know which server to contact.

After a web server receives a request, it sends a message back to the client - a response. Response messages contain the return-code, extra data about the format of the response and the actual data. The response can be as simple as a a text paragraph, image or an authentication token

A simple request, utilising the GET method would look something like this:

In this example, the type of method is specified (GET). The HTTP specification is noted (HTTP/1.1). The URL that is the target of the request and additionally what types of languages are expected in a response.

A simple response to a web page request would look something like this:

Starting which HTTP specification is used (HTTP/1.1). Server status code is returned (200 OK), then the type of data that gets sent with the response. Finally the page's html. Which usually initiates additional GET method requests for additional resources, like CSS and javascript.

In the following image the client makes several requests (Utilising the GET method) in order to display the whole page. Pay attention that not all of the data comes from the same server, content in the web is usually hosted on multiple servers.

Data formats

The data that gets exchanged over the web could be in any number of formats. For example, all of the data that gets exchanged could simply be within the URL that gets passed along. We have all seen those URLs that visually look like they are over a meter long. That is also considered data exchange on the web. Sending and receiving files counts as data exchange on the web. So when your browser uses GET method to ask for something, the server that gets contacted might just send back a file, if the request asks for it. Data could also be in JSON and/or XML format. Basically <Key:value> pairs. Information regarding what type of data gets exchanged is always paired with the requests and responses. This bit is in the request/response header, so that the client could tell what kind of information it expects and the server could tell the client, what kind of information it is sending. This avoids situations, like, client asks for XML format data and the server just responds with a .jpg file, which would be a miscommunication.

Headers provide extra information about the request / response. For example a server would specifiy within a header that what type of information gets exchanged next. At the same time a client's header might contain a authentication token, which proves to the sever, that this clients is used by this person (said person would have to be allowd to exchange data with the server).

When a server responds to a request. The response would contain the requested information. The header would contain, what type of data it is. Also, the server would send back a status code, most commonly a status code 200, which stands for OK success. Probably the most familiar status code is the 404, not found. The error codes that begin with 4xx are considered mistakes on the client's behalf. Something you as the information requester have to deal with / fix in order to get what you look for, especially if working with multiple origins on one page.

Again, this previous part was not intended to be all encompassing, but just enough to give you the faintest idea and plenty of keywords for googling about, if it got you interested.

Quick clarification / reminder, if someone missed this bit of information. The stuff in orage boxes is what you have to do to:

The stuff in green boxes are the stuff you check if you did the stuff in orange boxes correctly.

At the end of the labs you might notice just a green box without a orange one preceding one, that box is there to let you check if all of the previously done work still works as expected.

Let's get started!

1.Setting up personal web server

Understanding DNS CNAME records

Sometimes there is a need for one system to have multiple DNS names, all pointing to the same IP. This could be the case when hosting multiple virtual web servers or running multiple services on the same IP using service-specific host names, like:

• www.<vm_name>.sa.cs.ut.ee
• ftp.<vm_name>.sa.cs.ut.ee
• myservice.<vm_name>.sa.cs.ut.ee

We could create a number of A type records pointing to the same IP address. However the good style is to have a PTR type record for each A or AAAA type record, and if multiple A records point to the same IP, the reverse can be do only for one of them. Therefore good style in this case is to select only A record and assign it's PTR correspondingly. So as an example, subdomain named `student-test` would do the following:

• A : student-test.sa.cs.ut.ee -> 193.40.154.247
• PTR : 193.40.154.247 -> student-test.sa.cs.ut.ee

... and the rest of the host names which are pointing to the same IP address should be assigned using CNAME aliases:

• CNAME: www.student-test.sa.cs.ut.ee -> student-test.sa.cs.ut.ee
• CNAME: ftp.student-test.sa.cs.ut.ee -> student-test.sa.cs.ut.ee
• CNAME: myservice.student-test.sa.cs.ut.ee -> student-test.sa.cs.ut.ee

A CNAME record maps a name (alias) to another name (real name). Resolving CNAME record takes two steps - first, the alias is resolved to the real name, then the real name is resolved to the IP address (using A record). If a (real) name has multiple CNAME aliases pointing to it, one just needs to change the real name's type A record to "move" all the names to a new IP address.

There can be however not reverse record for CNAME aliases, the corresponding IP will be always resolve to the actual A record. Additionally the MX records and NS records can be only in use with A or AAAA records, we cannot assign them pointing to CNAME records.

More details about Canonical Name Record (CNAME) with examples you can read here ... http://www.zytrax.com/books/dns/ch8/cname.html.

Virtual Web Hosts Using Apache Web Server

Apache is the most popular web server software in use (as of February 2021). Apache supports wide variety of features and can be extended with modules. We will use the virtual host functionality to set up multiple web hosts on the same web server.

In CentOS 8, the main configuration file for Apache is /etc/httpd/conf/httpd.conf. Despite the fact that it is not really important which config file you use, it is recommended to keep different aspects of the setup in different .conf files. The additional site and module configuration files are kept in /etc/httpd/conf.d directory, i.e. ports.conf. This also offers better flexibility, when adding and removing different web applications and modules. When some parts of the configuration are kept in separate /etc/httpd/conf.d/*.conf files it is easier to navigate the setup and enable/disable blocks of instructions. For an example, directive Include mods-enabled/*.conf will load every .conf file in this directory and their instructions will be automatically included in the Apache configuration, everything else will be ignored. This way, adding or removing a bit of functionality can be just a matter of creating an appropriate my-functionality.conf file or changing the file extension to something other than .conf. For example - to my-functionality.conf.disabled without editing the actual .conf files

The log files for the Apache web server are located in the /var/log/httpd/ directory, where query (access_log) logs and error logs (error_log) are kept separately. It is recommended to have a separate log file for each virtual host as well (vhost).

Extra reading:

• The Apache HTTP server project
• Apache version 2.4 Documentation
• Red Hat specifics (Apache configuration guide)

During this lab we will configure Apache to use name-based virtual hosts. With name-based virtual hosting, the server relies on the client to report the hostname as part of the HTTP headers. Using this technique, many different hosts can share the same IP address.

First, configure the actual virtual host. We will create a virtual host for www.<vm_name>.sa.cs.ut.ee.

Apache httpd documentation:Virtual Host and more specifically Name-based Virtual Host

<VirtualHost *:80>
    ServerName InsertRightValueHere
    DocumentRoot /var/www/html/www.<vm_name>/public_html
    # Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
    LogLevel warn

    ErrorLog /var/log/httpd/InsertRightValueHere-error.log
    CustomLog /var/log/httpd/InsertRightValueHere-access.log combined
</VirtualHost>

If all was done correctly, your website default canvas should be ready. It should look something like this:

Try accessing your web server now:

• http://www.<yourdomain>.sa.cs.ut.ee
• You should see an empty page saying Index of /

You may have noticed that your page is empty. The next step is to create some content for the virtual hosts. In order to do that you need to be familiar with basic HTML tags and how web page is constructed. If needed, refer to very good public tutorials here https://www.w3schools.com/html/html_intro.asp and https://www.w3schools.com/html/html_basic.asp .

Now it is time to restart the Apache httpd server.

There are situations when you want to restart your Apache server, but can't interrupt it's work. Imagine that you have few hundred clients currently downloading files from your server and you need to avoid disconnecting them. In such situations you can use the following command:

# apachectl graceful

This will gracefully restart your Apache server with new configuration without affecting your client's connections.

Utilizing web server as a proxy

Very often, a web server is not utilized to serve files directly, but used for forwarding requests to another service that is a web server itself, but is either less secure or does not support settings a proper webserver does. A good example is a Python (Flask, Tornado), NodeJS or Go web applications. This is called proxying.

This allows a systems administrator to have a single point of control over security settings, without having to delve into the application configuration or code.

In this example we will use Python Flask, but you can use anything that is capable of serving web. We will start a service on localhost port 5000, and proxy that to the internet.

Now you have the necessary libraries to run a flask program. You can now create a random file, for an example /root/server.py, and add the following code there:

#!/bin/env python3

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run(port=5000)

Once you have run the python program, test if it works by doing curl localhost:5000. It should answer with whatever the program is programmed to do.

Now, to set up the proxy, do the following steps:

<VirtualHost *:80>
    ServerName proxy.<vm_name>.sa.cs.ut.ee 
    # ServerName sets the name to listen for with requests
    ErrorLog /var/log/httpd/proxy-error_log
    CustomLog /var/log/httpd/proxy-access_log common

    ProxyPreserveHost On

    ProxyPass / http://localhost:5000/
    ProxyPassReverse / http://localhost:5000/
</VirtualHost>

Now, because you cannot keep a terminal open constantly to keep this web service up, nor is it a good practice to run things like this, we are going to use our service manager, to keep our python service up for us. This also allows the service to start again when the machine reboots.

 
# systemd unit file for the Python Proxy Service

[Unit]

# Human readable name of the unit
Description=Python Proxy Service


[Service]

# Command to execute when the service is started
ExecStart=/usr/bin/python3 /usr/local/lib/server.py

# Disable Python's buffering of STDOUT and STDERR, so that output from the
# service shows up immediately in systemd's logs
Environment=PYTHONUNBUFFERED=1

# Automatically restart the service if it crashes
Restart=on-failure

# Our service will notify systemd once it is up and running
Type=simple

# Use a dedicated user to run our service
User=proxy

[Install]

# Tell systemd to automatically start this service when the system boots
# (assuming the service is enabled)
WantedBy=default.target

Let's add a default WordPress setup to our current setup

Let's install the prerequisite package necessary for a default WordPress setup.

Start up the mariadb service

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none): 
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] Y
New password: 
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] Y
 ... skipping.

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

Now let's download the wordpress tarball, unpack it and move it to the a proper webroot.

Let's grant user apache the right to work in /var/www/html/wordpress directory. And change the file SELinux security context.

Now let's consolidate our wordpress php logs to one place.

Now repeat the steps we did earlier for www.<vm_name>.sa.cs.ut.ee.

The final part of the WordPress setup is actually setting up the page, what we have done so far has been laying down the foundation, so to speak.

Configuring apache modules

While apache2 on it's own is a very powerful tool, its functionality can be expanded by loading extra modules. Currently we will be looking at loading some simple logging modules, but apache modules are used to security-harden your websites, add TLS support, rewrite URL paths, handle downloads, caching etc.

Each of these modules usually starts with a prefix mod_ and we can see that we are already loading and using some modules, for ex. our proxy website already uses a directive named ProxyPass that is actually loaded from /etc/httpd/conf.modules.d/00-proxy.conf. Using a modular structure like this allows developers and administrators to configure extra functionality without rebuilding whole environments.

Forensic logging

Default Apache2 logging is done by mod_log_config and we are already familiar with this by the usage of the ErrorLog and CustomLog directives we configured for all of our virtualhosts. Looking at our access logs defined in CustomLog we notice some pretty simple info - request origin IP, datetime of the request and some basic information about the request and response. While this is fine in most cases, when working with developers chasing an illusive bug they might require extra information about the nature of the request. This is where mod_log_forensic comes in, as forensic logs offer a substantial increase in the amount of information apache provides, mainly the ability to determine which requests are part of one session.

+YDvAdxfNz6bRQrk@IWUKUQAAAMA|GET / HTTP/1.1|Host:localhost|User-Agent:Mozilla/5.0 (X11; Linux x86_64; rv%3a84.0) Gecko/20100101 Firefox/84.0|Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8|Accept-Language:en-US,en;q=0.5|Accept-Encoding:gzip, deflate|DNT:1|Connection:keep-alive|Cookie:utbslang=et|Upgrade-Insecure-Requests:1|Cache-Control:max-age=0
-YDvAdxfNz6bRQrk@IWUKUQAAAMA

Security-hardening our webserver using mod_security

While basic website security should come from the developers side, system administrators have a few tricks to make sure malicious requests are filtered at the webserver level. For this mod_security is used in apache2 to filter requests that might not always inherently be malicious but can often be exploited.

ModSecurity is a Web Application Firewall (WAF) that is basically a list of rules that get matched against each HTTP request. These rules try to determine whether the request is malicious in nature, for an example trying to access system files like `/etc/shadow`, and if they do, then the request gets blocked.

Not all websites are always able to work behind a WAF. For an example `courses.cs.ut.ee` needed heavy WAF custom configuration, because every edit to a page seemed like a bunch of code for a WAF, and it denied all requests.

# default action when matching rules
SecDefaultAction "phase:2,deny,log,status:406"

# [etc/passwd] is included in request URI
SecRule REQUEST_URI "etc/passwd" "id:'500001'"

# [../] is included in request URI
SecRule REQUEST_URI "\.\./" "id:'500002'"

# [<SCRIPT] is included in arguments
SecRule ARGS "<[Ss][Cc][Rr][Ii][Pp][Tt]" "id:'500003'"

GET / HTTP/1.1
User-Agent: nc/0.0.1
Host: www.<vmname>.sa.cs.ut.ee
Accept: */*

Thankfully, with nowadays default webserver configuration, most of these kinds of attacks are impossible if you have not made a severe configuration error, but it still good to have at least two layers of configuration to prevent these kinds of problems.

Ansible tips

Tags

Previously, we have asked you to define your roles with tags like - role { role: <role name>, tags: <tag name> }. Tags are used for running specific parts of your playbook, instead of the whole thing.

For example, you have edited some zone files under your dns role and want to apply them. Running ansible playbook playbook.yml will run all of the roles that you have set up there, this will get cumbersome if you have tens of huge roles that will be executed. Running these extra roles won't break anything if your playbook is well written, but in some cases it will take a lot of time to run a whole playbook for a single configuration change. In this instance, it can be avoided by using ansible-playbook --tags=dns playbook.yml, which will run only the roles that have a dns tag. The main purpose is to avoid the hassle of creating multiple playbooks for singular roles, editing the main playbook to comment something our and all-in-all makes using Ansible a better experience. More information about tags can be found at the Ansible documentation tags page

Ansible modules

Ansible modules are discreet sets of code that give ansible the functionality that it has. You have already used a few modules in your playbook tasks, like user: for user management, file: for creating files and modifying their permissions dnf: for installing packages, templates etc. Ansible has a lot of modules built in that can greatly improve your playbook execution.

Ansible documentation always shows specific examples on every module usage, most of them even match with what we are trying to do in this lab. For example we can take them from seboolean and sefcontext to get:

- name: Seboolean | this equates to 'setsebool -P httpd_can_network_connect=1'
  seboolean:
    name: httpd_can_network_connect
    state: yes
    persistent: yes

- name: Sefcontext | this equates to 'chcon -t httpd_sys_rw_content_t /var/www/html/wordpress -R'
  sefcontext:
    target: '/var/www/html/wordpress(/.*)?'
    setype: httpd_sys_rw_content_t
    state: present

Handlers

Ansible tasks have three main states of completion:

• ok - task ran successfully and nothing was changed
• changed - task ran successfully and something was changed
• failed - executing the task failed

There are some tasks that you may only want to run when a change is made. Mostly this is used to restart services only when a configuration file has been updated, not on every run. Just as it was said in the guide above, you want to avoid unnecessary restarts to apache as much as you can.

Handlers have their own subdirectory roles/<rolename>/handlers/main.yml

An example use of a handler in a task declaration

- name: Template | website config
  template:
    src: vhost.conf.j2
    dest: /etc/httpd/conf.d/www.{{ hostname }}.conf
  notify:
    - restart httpd

When this task returns changed, then it will call out the handler that is listening for "restart httpd"

An example of a httpd handler in handlers/main.yml:

- name: restart httpd
  systemd:
    daemon_reload: yes
    name: httpd
    state: restarted
    enabled: yes
  listen: "restart httpd"

The ansible documentation for handlers has many great examples on how to run handlers. Now when a handler is notified, it will run at the end of your playbook to flush all of the changed you have done all at one time.

Automating lab 4

Keeping your Ansible repository safe in our Gitlab

And as always push your stuff to our Gitlab.