Hashicorp Vault - Logging

Shenzhen, China

Configure Logging
- Override
- Verify
Log Auditing
- Log Backup
  - Testing
- Setting up Auto-Sync
  - Debugging

Configure Logging

Define the desired log_level (Trace, Debug, Info, Warn or Error) in your Vault server config:

nano /etc/vault.d/config.hcl

storage "consul" {
    address = "192.168.2.110:8500"
    path = "vault/"
}
listener "tcp" {
    address = "192.168.2.110:8200"
    tls_disable = 1
}
api_addr = "http://192.168.2.110:8200"
cluster_addr = "https://192.168.2.110:8201"
log_level = "Debug"
ui = true

service vault restart
service vault status

Override

You can set an environment variable to override the log level set in the Vault config - either add it to your bash config or export it directly so it only remains active for the running session:

export VAULT_LOG_LEVEL=trace

So far I have been using the Vault UI to unseal Vault after every system reboot. To do it with the CLI run the command vault operator unseal three times with three different operator keys.

Verify

I can verify that logging is enabled by following along with journalctl:

sudo journalctl -f -b --no-pager -u vault

-- Logs begin at Sat 2021-09-18 09:32:14 HKT. --
Sep 18 12:15:06 consul-master vault[20295]: 2021-09-18T12:15:06.177+0800 [DEBUG] identity: groups collected: num_existing=0
Sep 18 12:15:06 consul-master vault[20295]: 2021-09-18T12:15:06.177+0800 [INFO]  identity: groups restored
Sep 18 12:15:06 consul-master vault[20295]: 2021-09-18T12:15:06.177+0800 [DEBUG] expiration: leases collected: num_existing=5
Sep 18 12:15:06 consul-master vault[20295]: 2021-09-18T12:15:06.180+0800 [INFO]  expiration: lease restore complete
Sep 18 12:15:06 consul-master vault[20295]: 2021-09-18T12:15:06.180+0800 [DEBUG] core: request forwarding setup function
Sep 18 12:15:06 consul-master vault[20295]: 2021-09-18T12:15:06.180+0800 [DEBUG] core: clearing forwarding clients
Sep 18 12:15:06 consul-master vault[20295]: 2021-09-18T12:15:06.180+0800 [DEBUG] core: done clearing forwarding clients
Sep 18 12:15:06 consul-master vault[20295]: 2021-09-18T12:15:06.180+0800 [DEBUG] core: leaving request forwarding setup function
Sep 18 12:15:06 consul-master vault[20295]: 2021-09-18T12:15:06.181+0800 [INFO]  core: usage gauge collection is disabled
Sep 18 12:15:06 consul-master vault[20295]: 2021-09-18T12:15:06.182+0800 [INFO]  core: post-unseal setup complete

Log Auditing

To collect those logs on a separate server I can enable the Vault audit function:

sudo mkdir /var/log/vault
vault audit enable file file_path=/var/log/vault/audit.log
Success! Enabled the file audit device at: file/

Verify that the log file is being written to:

cat /var/log/vault/audit.log | jq

{
  "time": "2021-09-18T05:02:41.431179781Z",
  "type": "request",
  "auth": {
    "token_type": "default"
  },
  "request": {
    "id": "fe5f6943-0a7b-4a21-05f3-5c5bb47e94e3",
    "operation": "update",
    "namespace": {
      "id": "root"
    },
    "path": "sys/audit/test"
  }
}
{
  "time": "2021-09-18T05:02:41.447768489Z",
  "type": "response",
  "auth": {
    "client_token": "hmac-sha256:b4ff29e865438f90b797f4cad389faf79c6a8093fc61f5b0f300b9568afa1524",
    "accessor": "hmac-sha256:79961a93a15a45e8e6b556f67c3003b9eebc809b52d4f1d2364394c3651b2c79",
    "display_name": "root",
    "policies": [
      "root"
    ],
    "token_policies": [
      "root"
    ],
    "token_type": "service",
    "token_issue_time": "2021-09-12T19:18:36+08:00"
  },
  "request": {
    "id": "d4590f68-9598-f870-39cd-ba3e42ed6a02",
    "operation": "update",
    "mount_type": "system",
    "client_token": "hmac-sha256:b4ff29e865438f90b797f4cad389faf79c6a8093fc61f5b0f300b9568afa1524",
    "client_token_accessor": "hmac-sha256:79961a93a15a45e8e6b556f67c3003b9eebc809b52d4f1d2364394c3651b2c79",
    "namespace": {
      "id": "root"
    },
    "path": "sys/audit/file",
    "data": {
      "description": "hmac-sha256:d3a21d3083e102f9231bf9946addf6bbf3ae4611e46b88d81698cb6f322dc3df",
      "local": false,
      "options": {
        "file_path": "hmac-sha256:c3ca67685459c365d181c445c3203c314b0d33150b71f94261f1a13cd4abbbca"
      },
      "type": "hmac-sha256:a8f9437a7116d5b9d0ba1ebb1ad9e836337bf24d4eccff441e49475c41c3c702"
    },
    "remote_address": "192.168.2.110"
  },
  "response": {
    "mount_type": "system"
  }
}

Now I could use Logstash or Elastic Filebeat to consume the logfile. Or just copy it to another server to be ingested on demand.

Log Backup

I will need an SSH keypair to be able to setup rsync to synchronize the logfile on my Vault Server (192.168.2.110) with an identical file on the Logging Server (192.168.2.111):

sudo ssh-keygen

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.

And copy the content of the public key:

cat ~/.ssh/id_rsa.pub

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7+KtKrz1OcGqfR6yCf/iWMkyva7cJNEEWF8axAZPQ4Y0p1A1dgkk2Tj2WZ5yskcGB8p7f2k1fCrelTnIOCemeVoxEZJcYmWvFyzt30Ih0XzGJWa3LuCWOxbyB5lALlwRhWKMxzeAGPeDQwsBgo5U3M3vS3mZ400fhodg56+P7/VM2GGxQIY4QWkqNZeb1xbkymYWpfnUiqa867kZoGslGAU8sGpODxSjCccsAXV70OtaYA2eNHXkjkznWxFs9SjeESyEApKn0WwTkF38iSuCvdLl1VpjXqrV7tr5nZhB4IShLTpjJgpOT2QTQICDDB4lu9LB4BUwR6E+k9ijpjU7cCflOnz89t7UZExFM1b8zhF1do4/6zvNJECvGoNZq5W105BNEXrN8m9HcU4IUzIvRq4FISLt+5wCJnnN8cQ4Atkj97Xc3qeBRT+2z8bv5oDCheA8rfjxFC5Nw+RwEELMHgVH8tkFUcmKoid3ZhBNs+CgQs01aA+Zm0phvrdc2AsU= myuser@consul-master

Into the authorized keys on the Logging Server:

nano ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 644 ~/.ssh/authorized_keys

Or us ssh_copy to add your users public key:

sudo ssh-copy-id root@192.168.2.111
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.2.111's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@192.168.2.111'"
and check to make sure that only the key(s) you wanted were added.

And uncomment the line PubkeyAuthentication yes in the:

nano /etc/ssh/sshd_config
systemctl restart sshd

Testing

To test the connection I can simply run ssh root@192.168.2.111 and this now logs me in to my logging server without asking for a password. But when testing rsync by creating a file test.txt on the Vault server I ran into 2 error messages:

sudo rsync -a ~/test.txt root@192.168.2.111:/opt/test.txt
zsh:1: command not found: rsync
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.3]

For the rsync Error I checked rsync --version on my logging server and saw that the software was not installed:

apt-get install rsync

rsync --version
rsync  version 3.2.3  protocol version 31

Re-running the test now was successful:

sudo rsync -a ~/test.txt root@192.168.2.111:/opt/test.txt

And the text file appeared on my logging server:

cat /opt/test.txt
hello

Setting up Auto-Sync

To use rsync to update the logfile on our logging server whenever Vault adds a line to it we can use incron:

apt install incron
service incrond start
service incrond status

And configure it:

sudo nano /etc/incron.allow

Using incrontab commands you can list (-l), edit (-e), and remove (-r) incrontab entries:

incrontab -l
incrontab -e
incrontab -r

And add the user you are going to use - in my case just root. Now I can create the crontab for the watch task:

sudo incrontab -e

Syntax:

<path>  <mask>  <command>

Here:

path is absolute path of the directory to watch.
mask is event mask(in symbolic or numerical form).
Event Symbols (Masks):
- IN_ACCESS: File was accessed (read).
- IN_ATTRIB: Metadata changed (permissions, timestamps, extended attributes, etc.).
- IN_CLOSE_WRITE: File opened for writing was closed.
- IN_CLOSE_NOWRITE: File not opened for writing was closed.
- IN_CREATE: File/directory created in watched directory.
- IN_DELETE: File/directory deleted from watched directory.
- IN_DELETE_SELF: Watched file/directory was itself deleted.
- IN_MODIFY: File was modified.
- IN_MOVE_SELF: Watched file/directory was itself moved.
- IN_MOVED_FROM: File moved out of watched directory.
- IN_MOVED_TO: File moved into watched directory.
- IN_OPEN: File was opened.
command is executable file (or script) with its arguments.
- The following wildcards may be used inside the command specification.
  - $$ Prints a dollar sign
  - $@ Add the watched filesystem path
  - $# Add the event-related file name
  - $% Add the event flags (textually)
  - $& Add the event flags (numerically)

/var/log/vault/audit.log        IN_MODIFY       rsync -a /var/log/vault/audit.log root@192.168.2.111:/opt/vault/audit.log

Make sure the job was saved:

sudo incrontab -l
/var/log/vault/audit.log        IN_MODIFY       rsync -a /var/log/vault/audit.log root@192.168.2.111:/opt/vault/audit.log

Debugging

It works... I just had a typo in my cron job... I keep the debugging section below ~ it lead me to the issue.

This did not work at first. So I checked the configuration file:

cat /etc/incron.conf

The allowed users were set correctly:

# Parameter:   allowed_users
# Meaning:     allowed users list file
# Description: This file contains users allowed to use incron.
# Default:     /etc/incron.allow

I read that you have to create an empty file with your user name as file name in /etc/incron/allow but this seems to be happening automatically when you added your username to /etc/incron.allow.

Then I re-checked the service status:

service incrond status

Sep 18 16:28:13 consul-master systemd[1]: Starting Inotify System Scheduler...
Sep 18 16:28:13 consul-master incrond[64272]: loading system tables
Sep 18 16:28:13 consul-master systemd[1]: Started Inotify System Scheduler.
Sep 18 16:28:13 consul-master incrond[64272]: loading user tables
Sep 18 16:28:13 consul-master incrond[64272]: loading table for user root
Sep 18 16:28:13 consul-master incrond[64272]: access denied on /var/log/vault_audit.log - events will be discarded silently
Sep 18 16:28:13 consul-master incrond[64272]: cannot create watch for user root: (2) No such file or directory
Sep 18 16:28:13 consul-master incrond[64272]: ready to process filesystem events

And saw that the access to /var/log/vault_audit.log was denied. I made sure that the root user had access to this file and even set it to mode 777 - no chance.

So I created a second job for the test.txt file from earlier sudo incrontab -e:

/home/myuser/test.txt   IN_MODIFY    rsync -a /home/myuser/test.txt root@192.168.2.111:/opt/test.txt

After restarting the service service incrond restart I was able to edit this file and see the changes in /opt/test.txt on my logging server - so it is working after all.

So I decided to change the log folder for Vault:

vault audit disable file
Success! Disabled audit device (if it was enabled) at: file/

mkdir /opt/vault
vault audit enable file file_path=/opt/vault/audit.log
Success! Enabled the file audit device at: file/

sudo incrontab -l
/opt/vault/audit.log        IN_MODIFY       sudo rsync -a /opt/vault/audit.log root@192.168.2.111:/opt/vault/audit.log

Configure Logging​

Override​

Verify​

Log Auditing​

Log Backup​

Testing​

Setting up Auto-Sync​

Debugging​