System Administration - Courses - Institute of Computer Science

Overview

This week's topic is practical Docker and DevOps. We will be looking into how to deploy Docker applications so, that they are available to be used, how to debug them and how to proxy them so, that putting applications into containers does not bring you to management hell.

Please make sure you did the previous lab, especially editing the /etc/docker/daemon.json file. You can delete last week's containers, but not that file, as it is necessary for running the Docker daemon on your VM.

This lab is composed of following topics:

Introduction
Running a starting container with open ports
Debugging containers
Linking containers
Properly publishing a container

Introduction

Last week was an introductory step into Docker. Even though you set up quite a few containers, none of them were actually usable properly. This is because for proper deployments to happen, you need to step into sysadmin shoes, and use your experience for the most appropriate deployment method.

Usually you want to make your container usable somehow. Depending on the software in it, there's different ways to do that.

For scripts/one-off-programs that have been put into containers, you run them and they do their job, and eventually exit.
- Example: Nikto (https://github.com/sullo/nikto)
For software that you publish to the network - webservers, databases, etc. You want them to listen on a host port, or to be proxied somehow.
- Example: apache, databases
Some software is just supporting infrastructure for other containers, that you only want to publish internally inside the machine to other containers.
- Example: redis, databases

Before starting any containers, please run the following command firewall-cmd --permanent --zone=trusted --add-interface=docker0 and reload your firewall

This makes sure, that packets are allowed to be routed between your public and docker interfaces. Otherwise this lab will start throwing errors at some point.

Running a starting container with open ports

Let's start with a very tiny web service, that prints out the OS information and HTTP request upon accessing it: https://github.com/nginxinc/NGINX-Demos/tree/master/nginx-hello.

Running this container is as simple as:

docker run -d -p 50000:80 --name iamfoo nginxdemos/hello
- The -d option makes it run in background.
- -p 50000:80 option makes it so, that the port 50000 on your VM gets directed into port 80 inside the container.
- --name iamfoo just gives it a name, so you cannot start a new container with the same name unless you delete previous one first.

Make sure you open all the firewalls on port 50000. If you have, then accessing port 50000 on your machine should do something like this:

Debugging containers

Often enough, things do not work them as you want them do. Either requests do not return what you would expect, the container itself seems to be missing data, there's some dependency you did not account for, etc.

For these reasons, it's important one knows how to find debug information with containers.

Logs

The problem is, that even though docker daemon writes logs:

journalctl -r -u docker

You'll notice that these logs are only about the docker service itself. You care more about the container logs. These can be checked doing:

docker logs <container ID|name>

So, for an example, doing docker logs iamfoo should return you this:

192.168.251.1 - - [11/May/2020:03:47:16 +0000] "GET / HTTP/1.1" 200 7234 "-" "curl/7.66.0" "-"
192.168.251.1 - - [11/May/2020:03:48:36 +0000] "GET / HTTP/1.1" 200 7234 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:75.0) Gecko/20100101 Firefox/75.0" "-"

Which are basically the same as the access logs for Apache. All the container errors get put here as well, if the software and dockerfile is set up properly. You can also read the logs information from stopped (not deleted) containers, which is especially handy for finding problems why they stopped.

Internals

Logs are not always useful. Sometimes you want to debug something inside the container, or test network connectivity.

For this reason, you can (sometimes, when creator of the Dockerfile has left appropriate binaries in) open a shell inside a container, and use it as your own VM.

Try doing this: docker exec -ti iamfoo sh

exec means execute a command inside container
-t means open a tty (shell), and -i makes it interactive, so you can type there
iamfoo is the container name
sh is the command to be executed

If this worked, you should be dropped into a new shell. If you check the files, IP address, it is a completely different machine. You can check the configuration that is being used to run nginx (cat /etc/nginx/nginx.conf), install software (apk update && apk add vim), and ping the network to test networking.

Some things to remember:

If you make changes to this container from the inside, then it will be there only for the lifetime of this particular container. The moment you delete this one, it's gone. For this reason it's never a good practice to make changes inside the container, but always the Dockerfile.
localhost now means the container itself.
- If you do curl localhost:80, now you see the same information on port 80 as you did on port 50000 before.
- Port 50000 gives an error, because there is nothing listening inside the container on port 50000.
- If you want to access your VM host, not the container, you need to use an IP address. If you check your containers IP address, it should something like 192.168.67.X. You can access your VM on 192.168.67.1 (e.g. curl 192.168.67.1:50000).

Linking containers

Sometimes, for security reasons, you want to hide your running service from the outside world, but still allow it to be accessible by some containers.

Good example is a database. A database does not need to be accessible from the outside world, as it contains important information, that is very easy to access - you only need to know right username and password.

This is why Docker uses networks internally, which is one of the most complicated aspect of Docker. We are going to make our own network, and make one container available only from inside another.

First, let us set up a new container: https://github.com/containous/whoami

docker run -d --name whoami containous/whoami
- As you can notice, we did not set up a port for it. It only has a service listening on port 80 inside the container, nothing outside.
- Go try to find a way to access it. (Clue: there is a way, but it is annoying)
- Let's also try to ping it from our previous container.
  - docker exec -ti iamfoo sh
  - ping whoami
    - What happens?
    - Also check how many network interfaces this container has.

So, as you can see, the two containers cannot talk to each other.

Let's now make a new network, and add our containers to it:

docker network create test --subnet 192.168.150.0/24
- NB! The IP address is mandatory, if you do not specify this you lose the access to your machine over the internet.
- You can check the networks by using docker network ls and docker network inspect test.
docker network connect test whoami
docker network connect test iamfoo

Now, if you go back inside the iamfoo container, check the network interfaces again. Let's try to ping the other container.

docker exec -ti iamfoo sh
ping whoami

As you can see, now the container pings the other container, and also, if you have specified a name for it, it can also utilize the name!
NB! pinging with name only works when you specify a --name <smth> parameter when running the container. If you do not, it gets an auto-assigned name and IP address, and it is your responsibility to know what you need to connect to.

There are tools to make this easier for you, for an example docker-compose, but we find that you cannot use tools properly unless you know what actually happens, when you do use those tools.

Properly publishing a container

Even though using the ports method for publishing things to the internet would work.. technically, then there are huge problems with that approach:

You need to remember which service is on which port.
You cannot scale services, as you cannot put multiple of them listening on the same port.
If you use a service provider, then it is very often that only some ports from the internet are allowed. (e.g. in public internet, only ports 80 and 443)
Firewall and security configuration becomes complicated.
You have no overview about how often or how you service is used, unless the software inside your container provides that information. (it usually does not)

One of the solutions would be to do a localhost proxy, like we did in the web lab. The problem with this is, it would solve only points 2, 3 and 5. Thankfully, there are genius solutions out there, that are capable of proxying without using any docker ports.

One of these services is called Traefik. We will be setting up a Traefik proxy to a container, without dabbling with the fancy buttons and dials Traefik has (automatic encryption, metrics, logging, complicated routing). This is left as homework, if interested.

Make a directory at /etc/traefik and copy the following inside /etc/traefik/traefik.toml.

[global]
  checkNewVersion = true
  sendAnonymousUsage = true

[entryPoints]
  [entryPoints.web]
    address = ":80"

[log]

[api]
  insecure = true
  dashboard = true

[ping]

[providers.docker]

This configuration opens up port 80 and 8080 inside the container. Port 80 for accessing websites/containers, and port 8080 for a fancy dashboard.

Now run Traefik: docker run -d -p 50080:80 -p 58080:8080 -v /var/run/docker.sock:/var/run/docker.sock:ro -v /etc/traefik/traefik.toml:/traefik.toml:ro --restart=always traefik:v2.1.8

We map port 80 to host port 50080, because there is already apache listening on port 80.
We map port 8080 to host port 58080 to comply with previous standard.
Traefik needs access to host /var/run/docker.sock to find containers we are going to start up. NB! Do this only with services you trust! One can read security information from this socket!
Also mount /etc/traefik/traefik.toml as a configuration file inside the container.

After running this and opening port 50080 and 58080, you should be able to access them from the internet. Check the 58080 port.

Now we have a working proxy, but we have not told it to proxy anything. Let's fix that problem.

Run a container like this:

docker run -d --name whoami-traefik --label traefik.enable=true --label 'traefik.http.routers.router.rule=Host(`<machine_name>.sa.cs.ut.ee`)' --label 'traefik.http.routers.router.entrypoints=web' containous/whoami

Where you replace appropriate bits.

traefik.enable=true tells traefik to proxy this container
traefik.http.routers.router.rule=Host(`<machine_name>.sa.cs.ut.ee`) tells which name to route to this container
traefik.http.routers.router.entrypoints=web says which entrypoint to use (we have only one)
--name whoami-traefik we set a name to the container so you cannot run multiple instances of the container with these settings - traefik does not like that.

If you go to page <machine_name>.sa.cs.ut.ee:50080, you should see the output of whoami container. You can check the logs and see how it works, also there should be information about this container on the Traefik dashboard.

System Administration 2019/20 spring