Süsteemihaldus 2017/18 kevad

0.Make sure you did finish all the tasks of the previous weeks.

This manual is tested and should work. If you find any errors or have a question about it report to alo.peets@ut.ee

1.Setting up File Server

In distributed systems a files server is one (or many) hosts attached to a network and providing a file storage service that the workstations or remote clients can use. The interaction between the client and server in this case is organized by the file server protocol. Multiple protocols exist in the domain of file services and can be categorized into groups:

• LAN side vs. Internet side protocols
  • LAN side protocols usually offer tighter bindings between the server side storage and client side. The client in this case has the remote storage explicitly visible as a local file system ( we call it mapping or mounting the remote file system). The protocols here: NFS and CIFS/SMB. In addition to FS-level transparency the LAN side protocols usually offer a service discovery helping the clients to discover the file services in the scope of LAN.
  • Internet side protocols are mostly reduced to request/response protocol design (FTP,SFTP,HTTP) and therefore are focusing on download,upload,delete etc. primitive actions (as opposed to NFS where we can edit the file directly - in FTP we have to download the file first).
    • A special subset here are protocols like DropBox and OwnCloud (and SSHFS) which are in fact Internet side protocols but do offer remote directory mapping (to local file systems). However when editing the file on the mapped directory - the changes are stored locally first and then synced to the remote storage by the corresponding client software. Here the software on client side is determined to watch for the changes on both remote and local files and sync them correspondingly).
• Standalone vs. Network File System vs. Distributed File System
  • How the actual data is stored:
    • on single server standalone
    • on multiple servers hierarchically
    • on multiple servers with block-level redundancy

In case of NFS and CIFS/SMB the redundancy is usually achieved by having different physical servers contributing to different portions of so called common file tree. Just like we have different partitions of hard drive contributing to one root tree:

• /dev/sda1 -> /
• /dev/sda2 -> /var

... we can have different NFS and SMB servers contributing to local root tree:

• smb://10.9.8.7/common/opt -> /opt
• nfs://10.9.8.5/common/home -> /home
• nfs://10.9.8.5/common/share -> /usr/share

Here we see clear resource distribution but we do not see redundancy yet as each subtree is still stored on only one physical server. Redundancy in this case can be achieved by applying a distributed file system (like GlusterFS) to aggregate the storages of multiple servers using file-based replicating image.
Finally the systems like HDFS allows as to have automatic block-level redundancy and distribution: image

!!! Make a backup! In this lab there is a higher than normal chance that you will break your machine (errors on boot) so you should make a backup.

File System Quotas

A disk quota is a limit set by a system administrator that restricts certain aspects of file system usage on modern operating systems. The function of using disk quotas is to allocate fair distribution of storage resources and to protect against accidental filling of the file system.

First we need a disk that we can use without breaking things. When we created a machine in ETAIS we also specified one small separate 1GB disk. Now its time to start using it. Do not mix vda and vdb in this lab otherwise you might break your whole machine and in worst case scenario you break everything thus must start labs from Week 3.

• Identify correct path and name of your secondary @@1GB disk
  • # lsblk
  • 
• Now we need to configure (format) it before we can use it.
  • # fdisk /dev/vdb
  • hit o to create a new empty DOS partition table
  • hit n to add a new partition
  • hit p to select primary type
  • hit 1 - Partition number (1-4, default 1):
  • hit 2048 - First sector (2048-2097151, default 2048)
  • hit ENTER if asked Last sector, +sectors or +size{K,M,G,T,P} (2048-2097151, default 2097151):
  • hit w to write table to disk and exit
• # mkfs.ext4 /dev/vdb1
• # mkdir /mnt/vdb1
• # mount -t ext4 /dev/vdb1 /mnt/vdb1

Our manual is based on public more detailed manual How To Enable User and Group Quotas : https://www.digitalocean.com/community/tutorials/how-to-enable-user-and-group-quotas

• Install quota package
• Initialize the user quotas on newly created file system
  • Edit the /etc/fstab and add mount point to /dev/vdb1 with usrquota parameter enabled.

  • /dev/vdb1       /mnt/vdb1       ext4    errors=remount-ro,usrquota,grpquota 0 1
    

• remount the /mnt/vdb1 file system with mount -o remount /mnt/vdb1 command
• Perform a quotacheck wit quotacheck -cugm /mnt/vdb1 command
• Turn on quotas by running quotaon /mnt/vdb1

• Create new user account called dataguy (use adduser command)

Before we continue with setting the quota for our new user. Let's figure out what is its UID ? A UID (user identifier) is a number assigned by Linux to each user on the system. This number is used to identify the user to the system and to determine which system resources the user can access. UIDs are stored in the /etc/passwd file. Groups in Linux are defined by GIDs (group IDs). Just like with UIDs, the first 100 GIDs are usually reserved for system use. The GID of 0 corresponds to the root group and the GID of 100 usually represents the users group. GIDs are stored in the /etc/groups file.
Let's first figure out what is current UID of a newly created user dataguy.

Later in this Lab we are going to setup NFS server and export file system for remote usage. In particular we are going to rely on user dataguy as NFS expects the UID will be the same on both NFS server and NFS client hosts. Usually centralized user directory server (DB or LDAP) is in use with NFS to guarantee the uniqueness of user/UID pairs. For our test-setup we just agree we force the dataguy user to be set with static UID.

uid=2001(dataguy) gid=2001(dataguy) groups=2001(dataguy),100(users)

After the UID of a dataguy user is fixed we may finally proceed to quota settings:

Setting the quotas for the user, root and mailuser is not recommended at this moment.

Installing Samba

Samba is a free software re-implementation of the SMB/CIFS networking protocol, and was originally developed by Andrew Tridgell. Samba provides file and print services for various Microsoft Windows clients and can integrate with a Microsoft Windows Server domain, either as a Domain Controller (DC) or as a domain member. As of version 4, it supports Active Directory and Microsoft Windows NT domains.

Samba runs on most Unix, OpenVMS and Unix-like systems, such as Linux, Solaris, AIX and the BSD variants, including Apple's macOS Server, and macOS client (Mac OS X 10.2 and greater). Samba is standard on nearly all distributions of Linux and is commonly included as a basic system service on other Unix-based operating systems as well. Samba is released under the terms of the GNU General Public License. The name Samba comes from SMB (Server Message Block), the name of the standard protocol used by the Microsoft Windows network file system. Source:Wikipedia.org

# apt install samba smbclient

# nano /etc/samba/smb.conf

 [smbshare]
    comment = Some files from folder /mnt/vdb1/data/rw
    writable = yes
    locking = no
    path = /mnt/vdb1/data/rw
    guest ok = no
    browseable = yes

 [smbreadonly]
   comment = Some files from folder /mnt/vdb1/data/ro
   read only = yes
   locking = no
   path = /mnt/vdb1/data/ro
   guest ok = yes
   browseable = yes

# mkdir -p /mnt/vdb1/data/rw
# mkdir -p /mnt/vdb1/data/ro
# chown -R dataguy:users /mnt/vdb1/data/rw
# chown -R dataguy:users /mnt/vdb1/data/ro
# chmod -R ug+rwx,o+rx-w /mnt/vdb1/data/rw
# chmod -R u+rwx,go+rx-w /mnt/vdb1/data/ro

# usermod -a -G users dataguy

# smbpasswd -a dataguy

pdbedit -w -L

$ smbclient -U dataguy //localhost/dataguy
$ smbclient -U smbtester //localhost/smbreadonly

Installing NFS

NFS is a network file system protocol allowing a client system access files over a network. The file system is presented to the end-user in a manner similar to a local file system. NFS allows the administrator to share sub-trees of a local file system to the network. The process of sharing is called "exporting".

 /mnt/vdb1/data
 /mnt/vdb1/data/ro
 /mnt/vdb1/data/rw

 /mnt/vdb1/data 172.17.64.0/22(rw,sync,fsid=0,crossmnt,subtree_check)
 /mnt/vdb1/data/ro 172.17.64.0/22(ro,nohide,insecure,sync,subtree_check)
 /mnt/vdb1/data/rw 172.17.64.0/22(rw,nohide,insecure,sync,subtree_check)

 You should see:

    program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  33446  status
... many more lines

 /media/nfs_ro
 /media/nfs_rw

 mount -vvv -t nfs4 teacher.est:/ro -oro,vers=4,proto=tcp,port=2049,sec=sys /media/nfs_ro/
 mount -vvv -t nfs4 teacher.est:/rw -orw,vers=4,proto=tcp,port=2049,sec=sys /media/nfs_rw/

To be continued by:

• LDAP authentication
• Nextcloud installation

New lab manual is separate file ... LDAP & Nextcloud