Some instructions are intentionally not clear and have only generic action description. You should be able to put the pieces together with help from Apache manual and Google search.
During 23rd of March, the contents of this lab may change, as in we might add one additional subsection to this lab.
Please note that we might add additional new nagios tests during this weeks lab (regarding this lab)
0.Make sure you did finish all the tasks of the previous weeks.
We expect here:
- Personal domains are configured. Machine is accessible over <machine_name>.sa.cs.ut.ee inside the University network.
- All the tests on scoring.sa.cs.ut.ee are green.
Before you start, you need to set your personal machine's (the computer you are using - NOT THE VM) DNS servers.
They should be set to ns1.ut.ee and ns2.ut.ee. (IPs: 193.40.5.39, 193.40.5.76).
Without this you will not be able to check your websites from the browser later on in the lab.
Intro, this bit is instead of the slides.
This, like every other into into a lab, is again is a simple intro to the lab's topic, basically the basic knowledge in layman's terms. You are strongly encourage to dig into the topic by yourself.
Accessing data on the web
Let's say you want to access www.ut.ee. Your browser makes a request for www.ut.ee over the internet. And it usually receives a web page in return. Now this transaction about ask something and recive something is an over simplification. While all of the communication from asking for resources to reciving resources is in human readable text format, the request and responds do quite a few back and forths before the final web page that you requested ends up in your browser.
Client that sends out a message containing info about wanting something, makes a request, uses what is called a GET method. In short when the client utilises a GET method it means, it wants something (information, obviously) in return. All the the requests are done with a specific URL in mind, so that the client doing the request on your behalf would know which server to contact.
Servers that recive a request send a message back to the client, that is called a response. Response messages use POST method (oppsite of GET method), where the server wants to "give" you data, instead of GET method's "give" me data.
Requests and responds are mostly used to exchange data.
A simple request, utilising the GET method would look something like this:
![](https://courses.cs.ut.ee/2020/sa/spring/Main/Apache?action=download&upname=simple_get.png)
In this example, the type of method is specified (GET). The HTTP specification is noted (HTTP/1.1). The URL that is the target of the request and additionally what types of languages are expected in a response.
A simple response to a web page request would look something like this:
![](https://courses.cs.ut.ee/2020/sa/spring/Main/Apache?action=download&upname=simple_post.png)
Starting which HTTP specification is used (HTTP/1.1). Server status code is returned (200 OK). Type of data that gets sent with the response. Finally the page's html. Which usually initiates additional GET method requests for additional resources, like CSS and JS (and these requests get responses from the server in the form of POST requests.
In the following image the client makes several requests (Utilising the GET method) in order to display the whole page.
![](https://courses.cs.ut.ee/2020/sa/spring/Main/Apache?action=download&upname=many_gets.png)
Data formats
The data that gets exchanged over the web could be in any number of formats. For example, all of the data that gets exchanged could simply be within the URL that gets passed along. We have all seen those URLs that visually look like they are over a meter long. That is also considered data exchange on the web. Sending and receiving files counts as data exchange on the web. So when your browser uses GET method to ask for something, the server that gets contacted might just send back a file, if the request asks for it. Data could also be in JSON and/or XML format. Basically <Key:value> pairs. Information regarding what type of data gets exchanged is always paired with the requests and responses. This bit is in the request/response header, so that the client could tell what kind of information it expects and the server could tell the client, what kind of information it is sending. This avoids situations, like, client asks for XML format data and the server just responds with a .jpg file. It would be a miscommunication.
Headers provide extra information about the request / response. For example a server would specifiy within a header that what type of information gets exchanged next. At the same time a client's header might contain a authentication token, which proves to the sever, that this clients is used by this person (said person would have to be allowd to exchange data with the server).
When a server responds to a request. The response would contain the requested information. The header would contain, what type of data it is. Also, the server would send back a status code. Most commonly a status code 200, which stands for OK success
. Probably the most familiar status code is the 404, page not found
. The error codes that begin with 4xx are considered mistakes on the client's behalf. Something you as the information requester have to deal with / fix in order to get what you look for.
Again, this previous part was not intended to be all encompassing, but just enough to give you the faintest idea and plenty of keywords for googling about, if it got you interested.
Quick clarification / reminder, if someone missed this bit of information. The stuff in orage boxes is what you have to do to:
- Install vim
# yum install vim
The stuff in green boxes are the stuff you check if you did the stuff in orange boxes correctly.
- Use vim to modify file A.
# vim /etc/A
At the end of the labs you might notice just a green box without a orange one preceding one, that box is there to let you check if all of the previously done work still works as expected.
Let's get started!
1.Setting up personal web server
Installing the updates
It has been a while we did not pay attention at the announced updates, let's check if there are any.
Understanding DNS CNAME records
Sometimes there is a need for one system to have multiple DNS names, all pointing to the same IP. This could be the case when hosting multiple virtual web servers or running multiple services on the same IP using service-specific host names, like:
- www.<vm_name>.sa.cs.ut.ee
- ftp.<vm_name>.sa.cs.ut.ee
- myservice.<vm_name>.sa.cs.ut.ee
We could create a number of A
type records pointing to the same IP address. However the good style is to have a PTR
type record for each A
or AAAA
type record, and if multiple A
records point to the same IP, the reverse can be do only for one of them. Therefore good style in this case is to select only A
record and assign it's PTR
correspondingly, like:
- A : student-test.sa.cs.ut.ee -> 193.40.154.247
- PTR : 193.40.154.247 -> student-test.sa.cs.ut.ee
... and the rest of the host names which are pointing to the same IP address should be assigned using CNAME aliases:
- CNAME: www.student-test.sa.cs.ut.ee -> student-test.sa.cs.ut.ee
- CNAME: ftp.student-test.sa.cs.ut.ee -> student-test.sa.cs.ut.ee
- CNAME: myservice.student-test.sa.cs.ut.ee -> student-test.sa.cs.ut.ee
A CNAME
record maps a name (alias) to another name (real name). Resolving CNAME
record takes two steps - first, the alias is resolved to the real name, then the real name is resolved to the IP address (using A
record). If a (real) name has multiple CNAME
aliases pointing to it, one just needs to change the real name's type A
record to "move" all the names to a new IP address.
There can be however not reverse record for CNAME
aliases, the corresponding IP will be always resolve to the actual A
record.Additionally the MX
records and NS
records can be only in use with A
or AAAA
records, we cannot assign them pointing to CNAME
records.
More details about Canonical Name Record (CNAME) with examples you can read here ... http://www.zytrax.com/books/dns/ch8/cname.html.
- Create a CNAME record for
www
inside/etc/named/<vm_name>.sa.cs.ut.ee
zone file.www IN CNAME <vm_name>.sa.cs.ut.ee.
- Reload the DNS and make sure the CNAME was added correctly.
# journalctl -r -u named
- To make sure everything new gets read in, did you update the
Serial
?
Virtual Web Hosts Using Apache Web Server
Apache is the most popular web server software in use (as of March 2012). Apache supports wide variety of features and can be extended with modules. We will use the virtual host functionality to set up multiple web hosts on the same web server.
- Install Apache Web Server
# yum install httpd
- Add the default Security group in
ETAIS
calledweb
withtcp
port80
to your Virtual machine, if necessary. - Also, either add port
80
or servicehttp
as a rule to yourfirewalld
service. - Afterwards you can start HTTPD service with default configuration
- systemctl start httpd
- Additionally let's install console based web browser
lynx
, this has been moved to a new repo, hence the unusual yum install command# yum --enablerepo=PowerTools install lynx
Check if you can access the default web page:
- Install text-based browser called
lynx
and enter a commandlynx http://<vm_name>.sa.cs.ut.ee
. You should see a text-based version of Apache default page. - Check if you can ping your virtual machine from personal computer (laptop)
$ ping 172.17.XX.XX
or$ ping <vm_name>.sa.cs.ut.ee
(last will work if yourbind
configuration is right AND you have set your personal computers DNS servers to be 193.40.5.39 and 193.40.5.76) - If previous steps are successful then in your web browser type address
http://172.17.XX.XX
orhttp://<vm_name>.sa.cs.ut.ee
In CentOS 8, the main configuration file for Apache is /etc/httpd/conf/httpd.conf
. Despite the fact that it is not really important, which conf file you use, it is recommended to keep different aspects of the setup in different .conf files. The additional conf files are kept in /etc/httpd/conf.d
directory, i.e. ports.conf
.
This also offers better flexibility, when adding and removing different web applications and modules. When some parts of the configuration are kept in separate /etc/httpd/conf.d/*.conf
files it is easier to navigate the setup and enable/disable blocks of instructions. For an example, directive Include mods-enabled/*.conf
will load every .conf
file in this directory and their instructions will be automatically included in the Apache configuration, everything else will be ignored. This way, adding or removing a bit of functionality can be just a matter of creating an appropriate my-functionality.conf
file or changing the file extension to something other than .conf
. For example - to my-functionality.conf.disabled
without editing the actual .conf files
The log files for the Apache web server are located in the /var/log/httpd/
directory. Query (access_log) logs and error logs (error_log) are kept separately. It is recommended to have a separate log file for each virtual host (vhost).
Extra reading:
- The Apache HTTP server project
- Apache version 2.4 Documentation
- Red Hat specifics (Apache configuration guide)
During this lab we will configure Apache to use name-based virtual hosts. With name-based virtual hosting, the server relies on the client to report the hostname as part of the HTTP headers. Using this technique, many different hosts can share the same IP address.
First, configure the actual virtual host. We will create a virtual host for www.<vm_name>.sa.cs.ut.ee
.
Apache httpd documentation:Virtual Host and more specifically Name-based Virtual Host
- Create a virtual host webroot directory. The contents of this directory will be published to web via HTTP. For
www.<vm_name>.sa.cs.ut.ee
,/var/www/html
directory is created during Apache installation for default page (usually web pages are located in /var/www folder or /var/www/html/folder). Into that directory createwww.<vm_name>
directory. Into which you should create another directory namedpublic_html
. So the end result should be something like/var/www/html/www.<vm_name>/public_html/
. - Create a configuration file for said virtual host. The file must be in the
/etc/httpd/conf.d/
directory. As the file name, usewww.<vm_name>.conf
- e.g.<vm_name>.conf
,www.<vm_name>.conf
(name does not actually matter, it just makes it easier for you to keep track of things) - Into the virtual host configuration file just created, add the configuration directives to define a new name-based virtual host. This is just an example (given below), you should insert correct values
<VirtualHost *:80> ServerName InsertRightValueHere DocumentRoot /var/www/html/www.<vm_name>/public_html # Possible values include: debug, info, notice, warn, error, crit, alert, emerg. LogLevel warn ErrorLog /var/log/httpd/InsertRightValueHere-error.log CustomLog /var/log/httpd/InsertRightValueHere-access.log combined </VirtualHost>
- Overview of the important parameters/directives for the virtual host you will need to set:
ServerName
- the full DNS name for virtual host i.e.www.student-test.sa.cs.ut.ee
DocumentRoot
- Directory from where Apache servers domain related files, for www would be/var/www/html/www.<vm_name>/public_html
ErrorLog
,CustomLog
- error log and access (query) log for the virtual host - the name and location of log files can be set to any value, but we recommend to stick to common values e.g.:${APACHE_LOG_DIR}/webmail-error.log
and${APACHE_LOG_DIR}/webmail-access.log combined
- For easier troubleshooting change
LogLevel
todebug
If all was done correctly, your website default canvas should be ready. It should look something like this:
![](https://courses.cs.ut.ee/2020/sa/spring/Main/Apache?action=download&upname=default_canvas.png)
This default page contains necessary information for you.
- Find the location of the default Apache HTTP server page.
- Follow the instructions inside it.
Try accessing your web server now:
http://test.<yourdomain>.sa.cs.ut.ee
- You should see an empty page saying Index of /
![](https://courses.cs.ut.ee/2020/sa/spring/Main/Apache?action=download&upname=website_empty.png)
You may have noticed that your page is empty. The next step is to create some content for the virtual hosts. In order to do that you need to be familiar with basic HTML tags and how web page is constructed. If needed refer to very good public tutorials here https://www.w3schools.com/html/html_intro.asp and https://www.w3schools.com/html/html_basic.asp .
- In the root directory of your virtual host, create an
index.html
file. The content of this file can be freely chosen. - For the
www.<your-domain>.sa.cs.ut.ee
virtual host, create anindex.html
file that contains your name in a Heading 1 format.- This index.html should also have a string of
www.<vm_name>.sa.cs.ut.ee
somewhere inside it for Nagios to test.
- This index.html should also have a string of
Use the # apachectl configtest
to test the configuration syntax or if all the configuration files are visible and can be loaded by Apache.
The output should be Syntax OK
.
Now it is time to restart the Apache httpd server.
- Enable and start/restart the Apache web server (service name is
httpd
)# apachectl restart
You do not need to restart Apache service everytime as the pages will be re-read each time. But since we just created a new index.html
file a restart to make sure everything is in order, does not hurt at this point.
If you visit your page again you should see your personal demo page now.
Sometimes modern web browsers don't understand page has been changed and just displays local cache version therefore if in doubt always refresh with <CTRL> + F5
key.
There are situations when you want to restart your Apache server, but can't interrupt it's work. Imagine that you have few hundred clients currently downloading files from your server and you need to avoid disconnecting them. In such situations you can use the following command:
# apachectl graceful
This will gracefully restart your Apache server with new configuration without affecting your client's connections.
Test by accessing the web pages:
- View the web pages for the virtual host you created, make sure that you are getting the right content for the virtual host.
- try
http://www.<your-domain>.sa.cs.ut.ee
- try
- In the
/var/log/httpd/
directory, look at the access and error logs for your virtual host. ** Understanding error messages are important part of system administration. Some future tasks (e.g. exam) may require you to get the troubleshooting information from the Apache logs independently.
Utilizing webserver as a proxy
Very often, a web server is not utilized to serve files directly, but used for forwarding requests to another service that is a web server itself, but much less known and secure. A good example is a Python (Flask, Tornado), NodeJS or Go web applications. This is called proxying.
This allows a systems administrator to have a single point of control over security settings, without having to delve into the applications code.
In this example we will use Python Flask, but you can use anything that is capable of serving web. We will start a service on localhost
port 5000, and proxy that to the internet.
- Install
PIP
(a tool to manage python libraries.)yum install python3-pip
- Install Flask libraries.
pip3 install flask
Now you have the necessary libraries to run a flask program. You can now create a random file, for an example /root/server.py, and add the following code there:
#!/bin/env python3 from flask import Flask app = Flask(__name__) @app.route("/") def hello(): return "Hello World!" if __name__ == "__main__": app.run(port=5000)
Following steps require two terminals. Run this Python application in one of the terminals, by doing python3 /root/flask.py
. The other terminal we use to continue setting up appropriate settings for proxy.
Once you have run the python program, test if it works by doing curl localhost:5000
. It should answer with whatever the program is programmed to do.
Now, to set up the proxy, do the following steps:
- Set up a domain name proxy.<vm_name>.sa.cs.ut.ee, that points to your machine.
- Run the command
setsebool -P httpd_can_network_connect=1
. This allows HTTPD to connect to your python program. - Add the following code into a new HTTPD config file, called
proxy.conf
.
<VirtualHost *:80> ServerName proxy.<vm_name>.sa.cs.ut.ee # Sets the name to listen for with requests ErrorLog /var/log/httpd/proxy-error_log CustomLog /var/log/httpd/proxy-access_log common ProxyPreserveHost On ProxyPass / http://localhost:5000/ ProxyPassReverse / http://localhost:5000/ </VirtualHost>
After doing this, you should be able to check whether your python program is available from the name proxy.<vm_name>.sa.cs.ut.ee
.
Try accessing your page by:
- curl proxy.<vm_name>.sa.cs.ut.ee.
- Using a web browser.
Now, because you cannot keep a terminal open constantly to keep this web service up, nor is it a good practice to run things like this, we are going to use our service manager, to keep our python service up for us. This also allows the service to start again when the machine reboots.
- Make a
proxy
user. - Move your python file to /usr/local/lib/server.py.
- Give ownership of your python file to
proxy
user. - Create a file
/etc/systemd/system/proxy.service
with following contents:
# systemd unit file for the Python Proxy Service [Unit] # Human readable name of the unit Description=Python Proxy Service [Service] # Command to execute when the service is started ExecStart=/usr/bin/python3 /usr/local/lib/server.py # Disable Python's buffering of STDOUT and STDERR, so that output from the # service shows up immediately in systemd's logs Environment=PYTHONUNBUFFERED=1 # Automatically restart the service if it crashes Restart=on-failure # Our service will notify systemd once it is up and running Type=simple # Use a dedicated user to run our service User=proxy [Install] # Tell systemd to automatically start this service when the system boots # (assuming the service is enabled) WantedBy=default.target
- Reload systemctl service files
systemctl daemon-reload
- Start a service called
proxy
. - Enable a service called
proxy
to be started on boot.
- See if something is listening on port
5000
. - Test if your website is working.
- Check if service starts on machine reboot.
Let's add a default WordPress setup to our current setup
Let's install the prerequsite package necessary for a default WordPress setup.
# yum install php-mysqlnd php-fpm mariadb-server httpd tar curl php-json
Start up the mariadb service
# systemctl start mariadb
And let's enable it, so it starts up every time the VM boots up.
# systemctl enable mariadb
If you haven't already done so, do the same for httpd
service.
The following command starts up a command line based procedure. The purpose of this is to override the default mariadb
settings and make it more secure by removing the default settings.
When you are asked to 'Set root password?'
write it down for future purposes. Put it somewhere you know to look for. Root password can be changed when you have root access to said machine, but that means you would have to google for a solution yourself (There are plethora of guides on the internet that can help you achieve your goal).
Otherwise using recommended settings, is recommended, pun intended. You will recognize recommended option by the capitalized letter, i.e. [Y/n], here Y (Yes is recommended).
# mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY! In order to log into MariaDB to secure it, we'll need the current password for the root user. If you've just installed MariaDB, and you haven't set the root password yet, the password will be blank, so you should just press enter here. Enter current password for root (enter for none): OK, successfully used password, moving on... Setting the root password ensures that nobody can log into the MariaDB root user without the proper authorisation. Set root password? [Y/n] Y New password: Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success! By default, a MariaDB installation has an anonymous user, allowing anyone to log into MariaDB without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? [Y/n] Y ... Success! Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] Y ... skipping. By default, MariaDB comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] Y - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] Y ... Success! Cleaning up... All done! If you've completed all of the above steps, your MariaDB installation should now be secure. Thanks for using MariaDB!
Login into mysql database with the password you just created for user @@root@.
# mysql -u root -p
Let's create a database for wordpress, naming it wordpress
.
mysql> CREATE DATABASE wordpress;
Now we need to create a new user admin
, with and really insecure password pass
.
mysql> CREATE USER `admin`@`localhost` IDENTIFIED BY 'pass';
Let's grant user admin
access to the database we created.
mysql> GRANT ALL ON wordpress.* TO `admin`@`localhost`;
Now, since we have just granted user admin
access to a new database wordpress
, we need to reload the grant tables, with the following command:
mysql> FLUSH PRIVILEGES;
mariadb
bit is done for now.
mysql> exit
Now let's download the wordpress
tarball, unpack it and move it to the a proper webroot.
# curl https://wordpress.org/latest.tar.gz --output wordpress.tar.gz
The tar command unpacks the wordpress.tar.gz into a directory wordpress
in the current working directory.
# tar xf wordpress.tar.gz
We can place the freshly unpacked wordpress
directory into /var/www/html
. We could just as easly create a new direcotry wordpress.<vm_name>
for more structured look (like we did for www previously), but it is not critical, as long as you know what you are doing.
# cp -r wordpress /var/www/html
Let's grant user apache the right to work in /var/www/html/wordpress
directory. And change the file SELinux security context.
# chown -R apache:apache /var/www/html/wordpress
# chcon -t httpd_sys_rw_content_t /var/www/html/wordpress -R
Now repeat the steps we did earlier for www.<vm_name>.sa.cs.ut.ee.
- Create a virtual host pointing at wordpress.
- Create a CNAME for wordpress.<vm_name>.sa.cs.ut.ee
# systemctl reload httpd
The final part of the WordPress setup is actually setting up the page, what we have done so far has been laying down the foundation, so to speak.
Go to wordpress.<vm_name>.sa.cs.ut.ee
and perform the actual WordPress installation. Follow the instructions provided there and complete the installation. Again all the passwords and authentications you set there, you should remember or write down. Otherwise it is a one way trip to google.