Süsteemihaldus - Kursused - Arvutiteaduse instituut

0. Before we begin the TLS lab

Let's change the flavor for the VMs so that we won't run into running of RAM issues, which can manifest themselves in our websites becoming unavailable or our VM having received an OOM kill.

First let's verify that we have 1 CPU and 1 GB of RAM. Use htop for that, in the upper left corner you should be greeted with similar output as:

Let's give the VM more resources by changing it's flavor.

ETAIS -> Resoruces -> VMs -> Actions -> Stop -> Wait for it to stop -> Actions -> Change Flavor -> From the New flavor drop-down menu, choose m2.tiny (2 vCPU, 4 GB RAM, 20 GB storage) -> Submit -> Wait for the flavor change to take place (refresh the page, the State should be SHUTOFF after a minute or so) -> Actions -> Start

Log back into your vm and utilise htop again to check that you have 2 CPUs available with 4 GB of RAM. Like this:

Now we have mitigated potential problems that might arise in the upcoming labs due to memory usage..

1. Overview

This weeks topic is Transport Layer Security. We will be talking about what it is, what it's used for and how to enable it in our already setup services.

This lab is composed of following topics:

Description of TLS
Enabling TLS with Apache web server
Enabling TLS in Postfix
Enabling TLS in Dovecot

2. Transport Layer Security (TLS)

Nowadays, everything is on the internet. Not only cat pictures, but also financial institutions, mail service, schools and even the government.

The problem is, that as a person, you usually do not want someone else to read your private emails or access your bank account. Especially if that someone has malicious intent.

This is entirely the job of System Administrators. They try to protect the privacy of data. Thankfully, to make their job easier, mathematicians have come up with a technology called asymmetric cryptography.

Asymmetric cryptography allows you to generate a pair of certificates - usually called certificate and key. The key is secret, you never let anyone know the key. But you do let them know the certificate.

Then, anyone that wants to send you anything private, takes your certificate, and encrypts the data s/he wants to send with this certificate. Now, after encryption, the only way to decrypt this data, is using your key. This is the same idea SSH keys use.

The same concept can be used, when you are hosting your own services. You can take a certificate/key pair, and tell Apache web server to use this pair. This means, that any time someone connects to Apache, they are given a public certificate to encrypt their information with. When they encrypt and send their data to the server, the server takes the private key, and decrypts the data. This allows you to exchange data with a server, while remaining completely safe from prying eyes.

You can see this public certificate by doing: openssl s_client -connect google.com:443 Under the server certificate section, there is the public certificate of google.com.

Now that certificates have entered the picture, there is yet another security issue. If everyone could generate a certificate/key pair for any web service, then it would be very easy to fake a web server, for an example a bank's. To prevent this, these certificate/key pairs are done in the following steps:

Generate a private key
Create a certificate request based on this key
Send the certificate request to a Certificate Authority
Validate why you should be able to generate a certificate for this domain
- Done over email, phone, DNS secret or HTTP secret
Wait until you receive your email and start using it

This process allows a client to check if the certificate/key pair is actually trusted. Most computer systems have a built in Certificate Authority list with their keys, and can cryptographically check the validity of all certificate/key pairs. Even your browser has one built in, and you can actually check it.

Because this process takes time, and either costs (CAs like Digicert, Namecheap etc.) or requires a service on the public web (Lets Encrypt), then we are going to use a CA made by the teachers, and we will sign our own certificates.

3. TLS in Apache (HTTPS)

Using certificate/key pair in Apache is fairly straightforward, but first we have to generate our certificates.

Thing to remember is that the HTTPS port is 443.

Make /etc/pki/tls/CA directory, and change your working directory to it ( PS! All following commands are supposed to be executed at /etc/pki/tls/CA folder).
# mkdir certs crl newcerts private
# echo 01 > serial
# touch index.txt
# vim /etc/pki/tls/openssl.cnf
Find the block [ CA_default ] section and under it configure
- dir = /etc/pki/tls/CA
- unique_subject = no (uncomment)
- default_md = sha512
- private_key = $dir/private/cakey.pem (Make sure there is a space between this and the comment)
Find the block [ req ] section and under it configure:
- default_bits = 4096
Find the block [ req_distinguished_name ] section and under it configure:
- countryName_default = EE
- stateOrProvinceName_default = Tartumaa
- localityName_default = Tartu (you must add it manually to correct place)
- 0.organizationName_default = System Administration course
- organizationalUnitName_default = Institute of Computer Science
Find the block [ usr_cert ] section and under it configure:
- subjectAltName=@alternate_names
Add the block at the end of file [ alternate_names ]

[ alternate_names ]
DNS.0 = *.<vm_name>.sa.cs.ut.ee
DNS.1 = <vm_name>.sa.cs.ut.ee
DNS.2 = mail.<vm_name>.sa.cs.ut.ee

Setting this alternate names block, means that our TLS certificate is valid for all the domain names specified in that block.

Save the changes.

Everybody will use the same CA root certificates to sign your server certificate with. Teachers created it for you with a command ... (DO NOT execute this command it is here for informational purpose only $openssl req -new -x509 -keyout /etc/ssl/cacert/private/cakey.pem -out /etc/ssl/cacert/cacert.pem -config /etc/ssl/cacert/openssl.cnf).

In the real world, you would have to make a certificate request, send it to an official CA and in a few days you would get an answer with your signed certificate. To speed things up we will give you a CA private key that normally is very-super-extra SECRET thing, as with that anyone can sign requests, but as our CA certificates are only used for educational purposes, we can afford to make that "BIG" mistake for now.

# cd /etc/pki/tls/CA - makes sure you are in right folder
# wget -O /etc/pki/tls/CA/private/cakey.pem http://scoring.sa.cs.ut.ee/files/cakey.pem - Download CA secret private key.
# wget -O cacert.pem http://scoring.sa.cs.ut.ee/files/cacert.pem - Download ca public key

Let' generate the new private key first:

# openssl genrsa -out newkey.pem 4096

Now lets make certificate request for your domain

# openssl req -new -key newkey.pem -out newreq.pem -days 360 -config /etc/pki/tls/openssl.cnf
- If asked use ENTER to select default value from openssl.cnf or if needed provide a new one
- Country = EE
- State = Tartumaa
- Locality = Tartu
- Organization = System Administration course
- Organizational Unit = Institute of Computer Science
- Common Name = *.<vm_name>.sa.cs.ut.ee - THIS MUST BE CHANGED
- Email = root@<vm_name>.sa.cs.ut.ee
- A challenge password []: - leave empty (hit ENTER)
- An optional company name []:- leave empty (hit ENTER)

Now you should sign a certificate with CA private key you downloaded earlier

# openssl ca -config /etc/pki/tls/openssl.cnf -policy policy_anything -out newcert.pem -infiles newreq.pem

It should ask for a CA private key password that would be "2daysuperadmin" and 2 times Y to confirm signing process.

Copy newly created keys to proper folders and fix file permissions

# cp newcert.pem /etc/pki/tls/certs/server.crt
# cp cacert.pem /etc/pki/tls/certs/cacert.crt
# cp newkey.pem /etc/pki/tls/private/server.key

TLS related site configuration directives can initially be found in the /etc/httpd/conf.d/ssl.conf file. We will use this as the template and create the configuration for the virtual hosts in their appropriate site configuration files.

You should be familiar with the following Apache httpd configuration directives before moving to the next task:

Now, let's enable TLS with our web services.

Install the mod_ssl package using the package manager.
- This package is actually a httpd webserver module, that allows it to deal with HTTPS connections.
For each of the virtualhosts you made in the webserver lab, do the following:
- In the virtualhost file, duplicate the listening virtualhost.
- Instead of the new virtualhost listening on port 80, set it to listen on port 443 (HTTPS port).
- Add an SSLEngine directive, and set it to on.
- Add the following SSL settings:

    SSLCertificateFile /etc/pki/tls/certs/server.crt
    SSLCertificateKeyFile /etc/pki/tls/private/server.key
    SSLCACertificateFile /etc/pki/tls/certs/cacert.crt

Repeat this for each virtualhost. There was 3 of them.
Make sure to open port 443 on all firewalls.
Restart your webserver.

Now you should be able to access your HTTPS virtual host in the web browser:

Open for example https://www.<vm_name>.sa.cs.ut.ee with your browser
- you must create a security exception for your self-signed certificate
  - This is intended, as we have not imported our private CA's root certificate to our browsers yet, so it does not know to trust this web page.
- Nevertheless, you should see the same content as on the http://www.<your-domain>.sa.cs.ut.ee when you accept the exception.
You must download cacert.pem file from http://scoring.sa.cs.ut.ee/files/cacert.pem to your host machine (not your vm, any folder is good) and then import it into browser.
- In Firefox import cacert.pem by Edit -> Preferences -> Advanced -> Certificates -> View Certificates -> Authorities -> Import -> Select File -> ...
- In Chrome Setting -> Show advanced settings ... -> Manage certificates -> trusted Root Certificate Authorities -> Import -> Next -> File name: cacert.pem ; type: All Files *.* -> Next -> Place all certificates in the following store; Certificate store: Trusted Root Certification Authorities -> Finish -> Yes -> OK
- Close browser and try again (error "Not Secure" should be removed.
  - In order to see Certificate info in Chrome press CTRL+SHIFT+i and in the windows that opens choose security tab and View certificate -> Details.
Now try accessing the HTTPS site again, it should not throw any errors.
If the other virtual hosts also work like https://proxy.<vm_name>.sa.cs.ut.ee and https://wordpress.<vm_name>.sa.cs.ut.ee, then you have configured them correctly.

4. TLS in Postfix

Last week we set up Postfix in a non-secure way. Anybody either in the client's network, or your personal VM's network, can eavesdrop on network traffic, and see any passwords and/or emails your email server sends or receives.

To remediate that, the solution is, again, using TLS. Enabling TLS means the following in Postfix:

In postfix main.cf configuration file, change the following:
- smtpd_tls_security_level = may
- smtpd_tls_cert_file=/etc/pki/tls/certs/postfix.pem
- smtpd_tls_key_file=/etc/pki/tls/private/postfix.key
- smtpd_tls_loglevel = 1
- smtp_tls_loglevel = 1
Also, copy the certificate and key to in place of the files we specified above.
In postfix master.cf configuration file:
- Find the submission port declaration:

submission inet n       -       n       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_tls_auth_only=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_client_restrictions=$mua_client_restrictions
  -o smtpd_helo_restrictions=$mua_helo_restrictions
  -o smtpd_sender_restrictions=$mua_sender_restrictions
  -o smtpd_recipient_restrictions=
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

(Half of it might be disabled from LAB 5.)

Enable the submission block (uncomment the header line)
- In the options list (lines starting with -o ) in the submission block make sure the following options are enabled:
  - -o syslog_name=postfix/submission
  - -o smtpd_tls_security_level=encrypt
  - -o smtpd_sasl_auth_enable=yes
  - -o smtpd_sasl_path=private/auth
  - -o smtpd_sasl_security_options=noanonymous
  - -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  - -o milter_macro_daemon_name=ORIGINATING

(In case some of the options are missing, you should add them)

Now let's also enable secure submission over explicit TLS (SMTPS). We will use the same limitation policies as in submission block

Again in the master.cf file.
- Find the smtps port declaration:

#smtps     inet  n       -       n       -       -       smtpd
#  -o syslog_name=postfix/smtps
#  -o smtpd_tls_wrappermode=yes
#  -o smtpd_sasl_auth_enable=yes
#  -o smtpd_reject_unlisted_recipient=no
#  -o smtpd_client_restrictions=$mua_client_restrictions
#  -o smtpd_helo_restrictions=$mua_helo_restrictions
#  -o smtpd_sender_restrictions=$mua_sender_restrictions
#  -o smtpd_recipient_restrictions=
#  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING

(It is disabled by default)

Enable the submission block (uncomment the header line)
- In the options list (lines starting with -o ) in the submission block make sure the following options are enabled:
  - -o syslog_name=postfix/smtps
  - -o smtpd_tls_wrappermode=yes
- SMTPS connection runs in TLS pipe (See submission block options) so we do not have to enroll any in-line TLS encryption, therefore we omit the tls_security_level option here.
- The rest of the options you may configure similarly to the submission block
  - -o smtpd_sasl_auth_enable=yes
  - -o smtpd_sasl_path=private/auth
  - -o smtpd_sasl_security_options=noanonymous
  - -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  - -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  - -o milter_macro_daemon_name=ORIGINATING
Save the file
Restart the postfix service
Make sure to open ports 465 and 587 in all the firewalls.

You can check whether your changes work, by doing the following:

openssl s_client -connect mail.<vm_name>.sa.cs.ut.ee:587 -starttls smtp -CAfile <path_to>/cacert.pem
openssl s_client -connect mail.<vm_name>.sa.cs.ut.ee:465 -CAfile <path_to>/cacert.pem

The most important part is the Verify return code field. If that is 0, everything is working, and CA validation also works.

If it is complaining about "self signed certificate", then either CA certificate is not working, you're using wrong certificate, or something else is wrong.

5. TLS in Dovecot

Encrypting the traffic of the mailserver itself is useless, if the authentication credentials can be still seen by malicious people when logging into the mail server.

This is why we also need to encrypt traffic in Dovecot.

In the /etc/dovecot/conf.d/10-ssl.conf file, set the following parameters:
- ssl = yes
- ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
- ssl_key = </etc/pki/dovecot/private/dovecot.pem
Also make sure to copy the certificate into those files, again.
Restart your dovecot service.
Open port 993 in firewalls.

You can check whether everything works again, by utilizing openssl command:

openssl s_client -connect <vm_name>.sa.cs.ut.ee:993 -CAfile <path_to>/cacert.pem

The most important part is the Verify return code field. If that is 0, everything is working, and CA validation also works.

If it is complaining about "self signed certificate", then either CA certificate is not working, you're using wrong certificate, or something else is wrong.

Self-signed certificate in a nutshell.

Source for the image. Published on 11.01.2017 by E.William in The Barbed Wire.

6. Wireshark

This part is not mandatory, but it will help you to understand why we were doing what we were doing.

We will try to see if we can sniff some traffic.

First, go install yourself a tool called wireshark . (To your own computer)

For windows:

For Linux:

install it from your package manager
e.g. yum install wireshark

For Mac OSX:

brew install wireshark

Once you have it installed, start it up with the highest permissions in your machine (Administrator in Windows, root in Linux and Mac).

You should get a window like this:

Attach:wireshark_interface.png Δ

From there, you need to choose the proper interface. This depends on which interface your University VPN tunnel is working. If you hit the correct interface, and doubleclick it, you should be getting a lot of traffic on the screen. If not, you're using the wrong interface.

Then, instead of the "Apply a display filter" line, write the following: ip.dst == <your_vm_ip> and (http or tls) And press enter.

After having done that, go and access the web pages on your machine. Use both http:// and https:// pages. You should be getting traffic on Wireshark.

After having done that, go and click on the traffic lines. See how different is TLS and HTTP information. With HTTP, you can see all the information that was exchanged. With TLS you cannot.

Just HTTP traffic:

Versus TLS traffic:

You can also try it with other pages on the world wide web, just remove the ip.dst filter.

7. Ansible and TLS

Putting the whole lab into Ansible is not something that would result in a idempotent playbook. So we will list the things that can be automated in an idempotent way, but some part of the labs will require manual intervention by the user. The following suggestions and steps are just guidelines, if you think you can do better, feel free to do so.

Creating directories CA, certs, crl, newcerts and private can and should be written into ansible.
Creating the file serial with content can be copied over with copy module same goes for index.txt file.
openssl.cnf is a file that should be templated with Jinja2.
Downloading CA secret private key can and should be automate.
Issuing openssl commands can be automated, but probably shouldn't. As that will take you down the ansible playbook advanced features rabbithole (expect module and so forth). Continue reading.

Just because it is reasonable to do some bits manually doesn't mean you can't still use the playbook even if manual intervention is expected in the middle of the playbook. Read about the pause module from ansible. Which allows you to stop your playbook execution, do things manually on the host in the meantime and then continue exectuing the playbook by simply pressing Enter in the terminal. This pause module is an option for issuing the openssl commands from the lab manual.

After the manual part one can use the Copy module to copy the newly created certificates and keys around. Figure out how to move files inside remote host.
Installing a package with ansible is given of course.
Use the file module (or template) for the Virtualhost files for Httpd. Restarting a service with Ansible is also a given.
Using file or template module on main.cf and master.cf for Postfix.
Using file or template module for 10-ssl.conf for Dovecot.
Opening necessary ports with ansible.
Use pause module for the openssl check commands.

8. Keep your playbook safe

As per usual always push your playbook to course's Gitlab.

In your ansible playbook directory:
git add .
git commit -m "TLS lab"
git push -u origin master

Go to your gitlab page in to see all of your latest push reached the git repository. If you play around with your repository and have made changes to the ansible that you wish to utilize also in the future, always remember to commit and push them.

Süsteemihaldus 2020/21 kevad