• We will be using a virtual machine in the faculty's cloud.
• When creating a virtual machine in the Launch Instance window:
  • Select Boot from image in Instance Boot Source section
  • Select SCGC Template in Image Name section
  • Select a flavor that is at least m1.medium.
• The username for connecting to the VM is student
• First, download the laboratory archive:

  [student@scgc ~] $ cd scgc
  [student@scgc ~/scgc] $ wget --user=<username> --ask-password https://repository.grid.pub.ro/cs/scgc/laboratoare/lab-07.zip
  [student@scgc ~/scgc] $ unzip lab-07.zip

After unzipping you should have a KVM image file (puppet.qcow2) and a script used to start the VM (lab07-start-kvm).

To start the VM, run the startup script:

student@scgc:~/scgc$ ./lab07-start-kvm

The KVM virtual machine for the lab will boot (can take up to 2-3 minutes).

In order to access the VM, use the following IP address (the password is student):

student@scgc:~/scgc$ ssh student@10.0.0.2

Puppet is a configuration management tool. In order to describe the necessary configurations, Puppet uses its own declarative language. Puppet can manage both Linux and Windows systems.

Puppet uses a resource as an abstraction for most entities and operations to be performed on a system. As an example, the state of a service (running/stopped) is defined in Puppet as a resource.

Use the puppet resource service command to see the system services from Puppet's perspective.

[root@puppet ~]# puppet resource service
service { 'apparmor':
  ensure => 'running',
  enable => 'true',
}
service { 'apparmor.service':
  ensure => 'running',
  enable => 'true',
}
service { 'autovt@':
  ensure => 'stopped',
  enable => 'true',
}
service { 'autovt@.service':
  ensure => 'stopped',
  enable => 'true',
}
service { 'bootlogd.service':
  ensure => 'stopped',
  enable => 'false',
}
service { 'bootlogs.service':
  ensure => 'stopped',
  enable => 'false',
}
...

The previous command syntax is:

• puppet command - used for accessing most of Puppet's features
• resource subcommand - interacts with available Puppet resources
• service parameter - type of the resources to be shown

Besides services, other Puppet resource examples are:

• users
• files or directories
• packages (software)

Show the resource representing the root user account, using the command: puppet resource user root

[root@puppet ~]# puppet resource user root
user { 'root':
  ensure             => 'present',
  comment            => 'root',
  gid                => 0,
  home               => '/root',
  password           => '$6$SWpfJK2ozbQ.bFA9$/[...]',
  password_max_age   => 99999,
  password_min_age   => 0,
  password_warn_days => 7,
  shell              => '/bin/bash',
  uid                => 0,
}

The resource structure contains the following elements:

• Type of resource: for this example, user
• Name of the resource: 'root'
• Attributes of the resource: ensure, comment, gid, home etc.
• Each attribute has a certain value

The previous syntax forms the “resource declaration”.

Besides services and users, Puppet implements a lot of other types of resources. In order to show them, use the command puppet describe –list

[root@puppet ~]# puppet describe --list
These are the types known to puppet:
augeas          - Apply a change or an array of changes to the  ...                                                                                          
computer        - Computer object management using DirectorySer ...                                                                                          
cron            - Installs and manages cron jobs                                                                                                             
exec            - Executes external commands                                                                                                                 
file            - Manages files, including their content, owner ...                                                                                          
filebucket      - A repository for storing and retrieving file  ...                                                                                          
group           - Manage groups
...

Using the puppet resource command, we can create new resources. Generic syntax is:

puppet resource type name attr1=val1 attr2=val2

If we want to create the user gigel so that:

• the user home directory is /home/gigel
• the default shell is /bin/sh

The Puppet command for this is:

[root@puppet ~]# puppet resource user gigel ensure=present shell="/bin/sh" home="/home/gigel"
Notice: /User[gigel]/ensure: created
user { 'gigel':
  ensure => 'present',
  home   => '/home/gigel',
  shell  => '/bin/sh',
}

Open the /etc/passwd file and check if the user has been created.

In order to remove a resource, the ensure attribute must be set to absent.

As an example, to remove the user gigel that we have previously created:

[root@puppet ~]# puppet resource user gigel ensure=absent
Notice: /User[gigel]/ensure: removed
user { 'gigel':
  ensure => 'absent',
}

Chech the /etc/passwd file to see if the user was actually removed.

Even though we can create, modify or remove resources from the command line, using puppet resource commands, this is not a scalable approach and not appropriate for complex scenarios.

A better solution would be:

• declaring resources in a (text) file
• applying the described modifications using Puppet

Files containing Puppet resource declarations are called manifests and usually have the .pp file extension.

We are going to write a manifest that describes a (text) file resource. The file is going to have the following properties:

• name and path: /tmp/my_file
• access rights: 0604
• file content: “File created using Puppet”

Resource declaration has the following syntax:

file {'my_file':
  path    => '/tmp/my_file',
  ensure  => present,
  mode    => '0640',
  content => "File created using Puppet.",
}

Save the previously described code in a manifest file called my_file_manif.pp

Applying a manifest is done with the command: puppet apply

[root@puppet ~]# puppet apply my_file_manif.pp

Notice: Compiled catalog for puppet in environment production in 0.18 seconds
Notice: /Stage[main]/Main/File[my_file]/ensure: defined content as '{md5}b4fdf30d694de5a5d7fe7a50cda27851'
Notice: Finished catalog run in 0.38 seconds

Check that the file has been created and the content and access rights are correct.

Try to apply the same manifest one more time:

[root@puppet ~]# puppet apply my_file_manif.pp

Notice: Compiled catalog for puppet in environment production in 0.16 seconds
Notice: Finished catalog run in 0.38 seconds

Notice that if the resource is already in the state described by the manifest, Puppet does not execute any action.

Change the access rights of the file to 755 and then apply the manifest again.

[root@puppet ~]# chmod 755 /tmp/my_file 
[root@puppet ~]# puppet apply my_file_manif.pp

Notice: Compiled catalog for puppet in environment production in 0.18 seconds
Notice: /Stage[main]/Main/File[my_file]/mode: mode changed '0755' to '0640'
Notice: Finished catalog run in 0.38 seconds

Change the content of the file and then apply the manifest again.

[root@puppet ~]# echo "This is not my file" > /tmp/my_file 
[root@puppet ~]# puppet apply my_file_manif.pp

Notice: Compiled catalog for puppet in environment production in 0.18 seconds
Notice: /Stage[main]/Main/File[my_file]/content: content changed '{md5}7225302b0d15d4a2562c2ab55e45d4cc' to '{md5}b4fdf30d694de5a5d7fe7a50cda27851'
Notice: Finished catalog run in 0.41 seconds

Notice that if the attributes of the resource are different from the ones described in the manifest, applying the manifest brings the resource back to the desired state.

The ensure attribute usually specifies if the resource:

• must exist (ensure ⇒ present)
• must NOT exist (ensure ⇒ absent)

Some types of resources define additional states for this attribute. File resources can have, in addition, the following values for ensure:

• directory
• link
• file

Define a manifest that creates a symbolic link to the /tmp/my_file file.

In a manifest, define a resource with the type ssh_authorized_key.

The resource must allow the user student from the physical machine to authenticate as the student user on the VM, without a password.

A Puppet manifest can contain declarations for multiple resources, but the order in which they are applied is not strictly enforced.

There are cases in which we have to make sure that a resource is applied before another (as an example, a package is installed before starting the service).

In these situations, we have to define resource dependencies.

We modify the previously created manifest:

file {'my_file':
  path    => '/tmp/my_file',
  ensure  => present,
  mode    => '0640',
  content => "File created using Puppet.",
}

notify {'my_notify':
  message => "File /tmp/my_file has been synced",
  require => File['my_file'],
}

• The notify resource defines a message that will be shown when its declaration is evaluated
• The dependency between resources is defined with the require attribute. In the previous example, the my_file resource is evaluated before the my_notify resource.

Modify the /tmp/my_file file and then apply the manifest described above. Notice the order in which the resources are evaluated.

An equivalent syntax would be to use the before attribute in the my_file resource:

file {'my_file':
  path    => '/tmp/my_file',
  ensure  => present,
  mode    => '0640',
  content => "File created using Puppet.",
  before  => Notify['my_notify'],
}

notify {'my_notify':
  message => "File /tmp/my_file has been synced",
}

For some resources we need a “refresh” action (as an example, a service that has to be restarted).

If in addition to resource dependency, we want to “refresh” the second resource when the first one is changed, we must:

• use notify instead of before, or
• use subscribe instead of require

An example would be restarting the SSH service when its configuration file has been changed:

file { '/etc/ssh/sshd_config':
  ensure => file,
  mode   => '0600',
  source => '/root/config-files/sshd_config',
}
service { 'sshd':
  ensure    => running,
  enable    => true,
  subscribe => File['/etc/ssh/sshd_config'],
}

Create a Puppet manifest with the previous code, then modify the /etc/ssh/sshd_config file and apply the manifest.

Instead of before / require or notify / subscribe, we can use the operators: ”->” or ”~>”

Example:

file {'my_file':
  path    => '/tmp/my_file',
  ensure  => present,
  mode    => '0640',
  content => "File created using Puppet.",
}
->
notify {'my_notify':
  message => "File /tmp/my_file has been synced",
}

In many situations, Puppet is used to make sure that a certain system service is installed, started and with the appropriate configuration.

The above use case can be implemented with 3 resources:

• package
• file
• service

Between the first 2 we have a “before / require” relation, and between the last 2 there is a “notify / subscribe”.

Create the following manifest which implements this design pattern for the SSH service, and then apply the manifest:

package { 'openssh-server':
  ensure => present,
}
->
file { '/etc/ssh/sshd_config':
  ensure => file,
  mode   => '600',
  source => '/root/config-files/sshd_config',
}
~>
service { 'sshd':
  ensure     => running,
  enable     => true, 
}

Modify various states of the “package / file / service” triplet and reapply the manifest. Examples:

• uninstall/purge the packet
• change the configuration file
• stop the service

Create a “package / file / service” manifest for the Apache service.

In order to define a variable in Puppet, we use the syntax $variable, both for assignment and referencing.

We change the manifest for the my_file file, defining the contents of the file as a variable.

$my_content = "File created using Puppet."

file {'my_file':
  path    => '/tmp/my_file',
  ensure  => present,
  mode    => '0640',
  content => $my_content,
}

In addition to user-defined variables, Puppet defines some system variables. These are called facts. In order to see all of these variables, we use the command facter.

[root@puppet ~]# facter 
disks => {
  fd0 => {
    size => "4.00 KiB",
    size_bytes => 4096
  },
  sda => {
    model => "QEMU HARDDISK",
    size => "8.00 GiB",
    size_bytes => 8589934592,
    vendor => "ATA"
  },
  sr0 => {
    model => "QEMU DVD-ROM",
    size => "1.00 GiB",
    size_bytes => 1073741312,
    vendor => "QEMU"
  }
}
dmi => {
  bios => {
    release_date => "04/01/2014",
    vendor => "SeaBIOS",
    version => "1.10.2-1ubuntu1"
  },
...

An example of using system variables is when taking decisions based on the value of some of them.

The following manifest ensures that the NTP service:

• is started if the system is a physical machine
• is stopped if the system is a virtual machine

The decision is taken based on the value of the $is_virtual system variable.

if str2bool("$is_virtual") {
  service {'ntp':
    ensure => stopped,
    enable => false,
  }
}
else {
  service { 'ntp':
    name       => 'ntp',
    ensure     => running,
    enable     => true,
    hasrestart => true,
    require => Package['ntp'],
  }
}

Apply the manifest and notice the state of the NTP service.

First, uninstall the NTP server from the virtual machine.

Then, write a manifest that:

• installs the NTP server packet
• ensures that the NTP server is started (the service name depends on the Linux distro)

Use the case conditional statement.

Ansible is a configuration management and provisioning tool, similar to Puppet. It uses SSH to connect to servers and run the configured tasks.

As opposed to Puppet, where each host manages its own data (services, users, files, etc.), and can optionally connect to a remote host to retrieve manifest files for configuration, Ansible is used to push the configuration from a central system to other hosts. An advantage of Ansible is that it does not require a specific service daemon to be installed before being able to configure the hosts. Operation is achieved through Python scripts for remote Linux hosts, or Powershell scripts for remote Windows hosts.

On the SCGC VM we are going to install and configure Ansible.

student@scgc:~$ sudo apt update
student@scgc:~$ sudo apt install -y ansible
# Required to use password authentication. By default, ansible requires authentication through SSH keys
student@scgc:~$ sudo apt install -y sshpass

Check that the package was successfully installed by running the command:

student@scgc:~/scgc$ ansible --version
ansible 2.5.1
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/student/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.17 (default, Nov  7 2019, 10:07:09) [GCC 7.4.0]

Ansible has a default inventory file used to define which servers it will be managing. After installation, there's an example one you can reference at /etc/ansible/hosts (initially fully commented out).

We are going to add 2 labels to the hosts file. One containing the local address, named thishost and another one of the previously used VM, named remote.

student@scgc:~$ cat /etc/ansible/hosts | grep -A 5 thishost
--
--
[thishost]
127.0.0.1

[remote]
10.0.0.2
--
--

Let's start running Tasks against a server.

Running against localhost:

student@scgc:~$ ansible thishost --connection=local -m ping
127.0.0.1 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

Running against the remote VM:

student@scgc:~$ ansible --ask-pass --user=student remote -m ping
SSH password: 
10.0.0.2 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

In either case, we can see the output we get from Ansible is some JSON which tells us if the Task (our call to the ping module) made any changes and the result.

Let's cover these commands:

• thishost, remote - Use the servers defined under this label in the hosts inventory file. The all argument can be used to run the ruleset on all defined hosts.
• --connection=local - Run commands on the local server, not over SSH
• -m ping - Use the “ping” module, which checks if the host can be accessed. Using ping with --connection=local does not make sense, as the option is used when running commands on the host that is issuing commands. It normally attempts to connect to the host via SSH.
• --ask-pass --user=student - SSH connection parameters: interactive password input, login as student user

Ansible has a fact gathering system similar to Puppet. To extract facts about the remote host we can use the setup module. The information is returned as Python dictionaries, where values can be strings, arrays, or other dictionaries.

student@scgc:~$ ansible --ask-pass --user=student remote -m setup
SSH password:
10.0.0.2 | SUCCESS => {
    "ansible_facts": {
        "ansible_all_ipv4_addresses": [
            "10.0.0.2"
        ], 
        "ansible_all_ipv6_addresses": [
            "fe80::5054:ff:fe12:3451"
        ], 
        "ansible_apparmor": {
            "status": "enabled"
        }, 
        "ansible_architecture": "x86_64", 
        "ansible_bios_date": "04/01/2014", 
        "ansible_bios_version": "1.10.2-1ubuntu1", 
...

The information in the facter, can be used in playbooks - configuration files written in YAML that act as scripts for ansible. The syntax used to expand all variables - including those created by the facter - is "{{ variable }}". For example, "{{ ansible_facts.hostname }}" will be expanded to the hostname, as identified by the facter.

We plan to enable the use of two-factor authentication for SSH through the use of Google's Authenticator mobile application. To do this, we need to create a Google Authenticator configuration file on the host. To create one with sensible defaults, you can use the following commands:

student@scgc:~$ sudo apt install libpam-google-authenticator qrencode
student@scgc:~$ echo -e "y\ny\ny\nn\ny" | google-authenticator

The commands above will create a configuration file for the authenticator, that will generate time-based codes, will update the ~/.google-authenticator file, disallow multiple users and enable rate limiting. The fourth option (the n in the string passed to the google-authenticator binary ) disables longer-lasting codes (this option is only useful when the phone and/or the server's time sync protocols are not working properly). For more details, consult DigitalOcean's tutorial on how to set it up here.

After running the command, the terminal will display the secret key as both a large QR code, and text. Please open the Google Authenticator app on your phone, and scan the QR code or enter it manually.

We will copy the Google Authenticator's configuration file, the configuration for the SSH daemon, and the PAM configuration file for the SSH service:

student@scgc:~$ mkdir config-files
student@scgc:~$ cd config-files
student@scgc:~/config-files$ cp /home/student/.google_authenticator .
student@scgc:~/config-files$ cp /etc/pam.d/sshd .
student@scgc:~/config-files$ cp /etc/ssh/sshd_config .

We will use the files created above as templates to replicate on the server(s). This example will only use the 10.0.0.2 VM as a target machine. We must set up the configuration files to use password + the a One Time Password (OTP) generated by the authenticator. Make sure the configuration files for sshd and PAM look as below:

student@scgc:~/config-files$ grep -B 5 -A 3 'pam_google_authenticator.so' sshd
# PAM configuration for the Secure Shell service
 
# Standard Un*x authentication.
@include common-auth
# 2-FA authentication with Google Authenticator
auth       required     pam_google_authenticator.so
 
# Disallow non-root logins when /etc/nologin exists.
account    required     pam_nologin.so

The changes to the PAM configuration file above make using the Google Authenticator module mandatory. It is placed after 'common-auth', so the code will be required after entering the password.

student@scgc:~/config-files$ grep -B 5 -A 3 '^ChallengeResponseAuthentication' sshd_config 
#PasswordAuthentication yes
#PermitEmptyPasswords no
 
# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication yes
AuthenticationMethods publickey keyboard-interactive
 
# Kerberos options

The changes to the configuration file above make using the challenge response to allow PAM to use multiple modules with challenge responses (i.e., password and authentication code in our case); the use of keyboard-interactive authentication is mandatory if more than just the password is required.

Ansible can use privilege escalation using the become keyword at certain tasks, or all tasks. If the user cannot run sudo with a password, the ansible_become_password variable must be set. To do this we will use a vault - a type of file that encrypts strings through a password - to store the password, instead of adding it as plain text to the playbook. To create a vault, use the following command, and write the key-value pair for the password in the file it opens using the default editor:

student@scgc:~/config-files$ ansible-vault create puppet.vault
New Vault password: # Enter vault password
Confirm New Vault password: # Confirm vault password
 
# In the opened file
ansible_become_password: student

After closing the vault, you can see that the information in it is encrypted.

To install the SSH daemon on the remote machine, and set it up for use with the Google Authenticator we will use the following playbook, saved as sshd.yaml:

---
- hosts: remote
  remote_user: student

  tasks:
  # include sudo password vault
  - name: Set host variables
    include_vars: "{{ ansible_facts.hostname }}.vault"
 
  # Install sshd and make sure it is at the latest version using the package
  # manager identified by ansible
  - name: Ensure sshd is at the latest version
    package:
      name: openssh-server
      state: latest
    become: yes
 
  # Install google authenticator module and make sure it is at the latest version
  - name: Ensure google-authenticator is at the latest version
    package:
      name: libpam-google-authenticator
      state: latest
    become: yes
 
  # Copy the google authenticator configuration file
  # The file MUST be located in the user's home directory with permissions 0600
  - name: Copy Google Authenticator config file
    copy:
      src: /home/student/config-files/.google_authenticator
      dest: /home/student/.google_authenticator
      mode: 0600
      owner: student
      group: student
 
  # Overwrite sshd configuration file. Make sure the challenge response setting
  # is enabled, and keyboard-interactive is a valid authentication method
  - name: Write the sshd configuration file
    template:
      src: /home/student/config-files/sshd_config
      dest: /etc/ssh/sshd_config
    become: yes
    notify:
      - restart sshd
 
  # Overwrite the PAM configuration file. Make sure that authentication through
  # google authenticator is required
  - name: Write the PAM configuration file
    template:
      src: /home/student/config-files/sshd
      dest: /etc/pam.d/sshd
    become: yes
    notify:
      - restart sshd

  handlers:
  # Handlers that are invoked when the configuration files change -
  # restart the sshd service
  - name: restart sshd
    service:
      name: sshd
      state: restarted
    become: yes

The playbook attempts to include the file named {{ ansible_facts.hostname }}.vault - which resolves to puppet.vault for the VM. To run it, we use the ansible-playbook command, with the --ask-pass, to ask for the SSH authentication password, and the --ask-vault-pass to provide the decryption password for the vault.

student@scgc:~/config-files$ ansible-playbook --ask-vault-pass --ask-pass sshd.yml
SSH password:
Vault password:
 
PLAY [remote] *********************************************************************
 
TASK [Gathering Facts] ************************************************************
ok: [10.0.0.2]
 
TASK [Set host variables] *********************************************************
ok: [10.0.0.2]
 
TASK [Ensure sshd is at the latest version] ***************************************
ok: [10.0.0.2]
 
TASK [Ensure google-authenticator is at the latest version] ***********************
changed: [10.0.0.2]
 
TASK [Copy Google Authenticator config file] **************************************
changed: [10.0.0.2]
 
TASK [Write the sshd configuration file] ******************************************
changed: [10.0.0.2]
 
TASK [Write the PAM configuration file] *******************************************
changed: [10.0.0.2]
 
RUNNING HANDLER [restart sshd] ****************************************************
changed: [10.0.0.2]
 
PLAY RECAP ************************************************************************
10.0.0.2                   : ok=8    changed=5    unreachable=0    failed=0

You should now be able to login using the password and the Google Authenticator.

student@scgc:~$ ssh student@10.0.0.2
Password: 
Verification code: 
Password: 
Verification code: 
Linux puppet 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64
 
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
 
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
student@puppet:~$