Hashicorp Consul - Vault Cert Management Part 3

Shen Zhen, China

Configure consul-template
- Create Template Files
- Create consul-template Configuration
  - Server Configuration
  - Client Configuration
Start consul-template
- Nomad Automation

Configure consul-template

Create Template Files

We can now use Consul-Template to automate certificate renewal for Consul when the TTL is expired. We can instruct consul-template to generate and retrieve those files from Vault using the following templates:

mkdir /opt/consul/templates

/opt/consul/templates/agent.crt.tpl (Master only)

Careful: The official documentation uses the common_name for the default datacenter consul.dc1. I use the name consul here and have to change the variable consul.consul accordingly. Otherwise the cert verification will fail. This name will be used a couple of times in the following commands - you need to change all of them according to your setup.

{{ with secret "pki_int/issue/consul-consul" "common_name=server.consul.consul" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}
{{ .Data.certificate }}
{{ end }}

The template will use the pki_int/issue/consul-consul endpoint that Vault exposes to generate new certificates.

/opt/consul/templates/agent.key.tpl (Master only)

{{ with secret "pki_int/issue/consul-consul" "common_name=server.consul.consul" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}
{{ .Data.private_key }}
{{ end }}

/opt/consul/templates/ca.crt.tpl (All Nodes)

{{ with secret "pki_int/issue/consul-consul" "common_name=server.consul.consul" "ttl=24h"}}
{{ .Data.issuing_ca }}
{{ end }}

Not sure if this is necessary - but since the Consul agent is running with the consul user (Master node only) I will change the user rights:

chown -R consul:consul /opt/consul/templates

Create consul-template Configuration

Now we need to create a configuration file consul_template.hcl that will instruct Consul-Template to retrieve the files needed for the agent, client and server, to configure TLS encryption:

Server Configuration

/opt/consul/templates/consul_template.hcl (Master only)

# This denotes the start of the configuration section for Vault. All values
# contained in this section pertain to Vault.
vault {
  # This is the address of the Vault leader. The protocol (http(s)) portion
  # of the address is required.
  address      = "http://localhost:8200"

  # This value can also be specified via the environment variable VAULT_TOKEN.
  # I am using the admin token created earlier
  token        = "s.MdNlboI0nff3Xpo97d1TfIxd"

  unwrap_token = false

  renew_token  = false
}

# This block defines the configuration for a template. Unlike other blocks,
# this block may be specified multiple times to configure multiple templates.
template {
  # This is the source file on disk to use as the input template. This is often
  # called the "consul-template template".
  source      = "/opt/consul/templates/agent.crt.tpl"

  # This is the destination path on disk where the source template will render.
  # If the parent directories do not exist, consul-template will attempt to
  # create them, unless create_dest_dirs is false.
  destination = "/opt/consul/agent-certs/agent.crt"

  # This is the permission to render the file. If this option is left
  # unspecified, consul-template will attempt to match the permissions of the
  # file that already exists at the destination path. If no file exists at that
  # path, the permissions are 0644.
  perms       = 0700

  # This is the optional command to run when the template is rendered. The
  # command will only run if the resulting template changes.
  command     = "sh -c 'date && consul reload'"
}

template {
  source      = "/opt/consul/templates/agent.key.tpl"
  destination = "/opt/consul/agent-certs/agent.key"
  perms       = 0700
  command     = "sh -c 'date && consul reload'"
}

template {
  source      = "/opt/consul/templates/ca.crt.tpl"
  destination = "/opt/consul/agent-certs/ca.crt"
  command     = "sh -c 'date && consul reload'"
}

The configuration file for the server contains the information to retrieve the CA certificate as well as the certificate/key pair for the server agent:

address : The address of your Vault server. If Vault runs on the same node as Consul you can use http://localhost:8200
token : A valid Vault ACL token with appropriate permissions

Client Configuration

/opt/consul/templates/consul_template.hcl (Minions only)

# This denotes the start of the configuration section for Vault. All values
# contained in this section pertain to Vault.
vault {
  # This is the address of the Vault leader. The protocol (http(s)) portion
  # of the address is required.
  address      = "https://server.consul.consul:8201"

  # This value can also be specified via the environment variable VAULT_TOKEN.
  # I am using the admin token created earlier
  token        = "s.MdNlboI0nff3Xpo97d1TfIxd"

  unwrap_token = false

  renew_token  = false
}

template {
  source      = "/opt/consul/templates/ca.crt.tpl"
  destination = "/opt/consul/agent-certs/ca.crt"
  command     = "sh -c 'date && consul reload'"
}

The configuration file for the server contains the information to retrieve the CA certificate only, the certificates for client agents are automatically generated from Consul when using the auto_encrypt setting:

address : The address of your Vault server. Here you need to use the IP / domain + opened port of your Vault master
token : A valid Vault ACL token with appropriate permissions

To be able to access Vault from an external service I am using a Vault-generated certificate. On my test system, that runs both Consul and Vault on a single server, I can use the Consul cert itself. For this to work I needed to add the Consul server address - which is in my case server.consul.consul - to /etc/hosts to resolve the WAN IP of my Vault server. Now I am still running into issues since the Certificate Authority is not being recognized by my Linux machine. So I have to add it to the trusted-store:

ln -s /opt/consul/agent-certs/ca.crt /usr/local/share/ca-certificates/ca.crt

update-ca-certificates
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

Start consul-template

After configuration is completed, you can start consul-template. You must provide the file with the -config parameter:

consul-template -config "/opt/consul/templates/consul_template.hcl"

THis service will output a `` every 24h - according to our Cert TTL. Meaning that the Consul service has been restarted after the certificate was updated. You can verify the cert validity by running:

openssl x509 -text -noout -in /opt/consul/agent-certs/agent.crt 

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            55:...
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = consul.consul Intermediate Authority
        Validity
            Not Before: Dec  5 06:34:13 2022 GMT
            Not After : Dec  6 06:34:43 2022 GMT
        Subject: CN = server.consul.consul

        ...

Nomad Automation

We can run this job with Nomad on all clients (except the master) using the raw_exec driver. To enable raw exec, the Nomad client configuration must explicitly enable the raw_exec driver in the plugin's options:

/etc/nomad.d/client.hcl

plugin "raw_exec" {
  config {
    enabled = true
  }
}

job "consul-cert-renewal" {
    
    type = "system"

    datacenters = ["All","your","datacenters"]
    
    group "cert-renewal" {

        task "consul-template" {
            driver = "raw_exec"

            config {
                # When running a binary that exists on the host, the path must be absolute/
                command = "/usr/bin/consul-template"
                args    = ["-config", "/opt/consul/templates/consul_template.hcl"]
            }
        }
    }
}

Configure consul-template​

Create Template Files​

Create consul-template Configuration​

Server Configuration​

Client Configuration​

Start consul-template​

Nomad Automation​