Skip to main content

Hashicorp Consul Refresher - Gossip TLS Encryption

Phnom Penh, Cambodia

Consul Agent Communication

The communication between servers in your Consul cluster is not encrypted by default. But Consul supports TLS encryption which can be configured in the server configuration file /etc/consul.d/consul.hcl.

TLS Encryption Settings

The following parameter have to be defined in all Consul Agent configuration files to enable encryption:

  • verify_server_hostname: true
  • verify_incoming: true
  • verify_outgoing: true

When verify_server_hostname is enabled you need certificates that are valid for your clients SAN (subject alternative-name) that is [servername].[datacenter].[domain]. If you use Consul to create your certs (see below) this will be done automatically.

Certificate Authority

Consul can be used as it's own CA and certificates can be generated through the Consul CLI:

  • Server: consul tls cert create -server
  • Client: consul tls cert create -client
  • CLI: consul tls cert create -cli

The Consul CA is automatically chosen if Connect is enabled without providing an external CA and client certificates are automatically distributed:

sudo nano /etc/consul.d/consul.hcl

Add the following lines to the end of the file and service consul restart:

connect {
  enabled = true

Required Certificates

  • The Consul HTTP API and RPC communication (TLS Certificates)
  • Service Mesh (mTLS Certificates)

TLS Upgrade for an existing Consul Cluster

Add the Encryption Key

  1. Generate Encryption Keys
consul keygen
  1. Add Keys to Keyring
consul keyring -install="+bLh5g5Q8N1RL2QQaY9U9z6I5Qi1WWDw6G3KMgeaTKg="
==> Installing new gossip encryption key...

Configure all Consul Agents

nano /etc/consul.d/consul.hcl
encrypt = "+bLh5g5Q8N1RL2QQaY9U9z6I5Qi1WWDw6G3KMgeaTKg="
encrypt_verify_incoming = false
encrypt_verify_outgoing = false

Restart all Consul Agents

service consul restart

Enable Encryption on all Agents

encrypt = "+bLh5g5Q8N1RL2QQaY9U9z6I5Qi1WWDw6G3KMgeaTKg="
encrypt_verify_incoming = false
encrypt_verify_outgoing = true
service consul restart
encrypt = "+bLh5g5Q8N1RL2QQaY9U9z6I5Qi1WWDw6G3KMgeaTKg="
encrypt_verify_incoming = true
encrypt_verify_outgoing = true
service consul restart

We can verify that encryption is enabled with by checking the journal for Encrypt: Gossip: true:

 journalctl -u consul | grep Gossip

Sep 08 13:57:46 consul-master consul[42418]:            Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
Sep 08 14:10:02 consul-master consul[44340]:            Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false

Key Management with Keyring

Distribute new Keys

This command distributes the key you created with consul keygen around the Consul cluster:

consul keyring -install="+bLh5g5Q8N1RL2QQaY9U9z6I5Qi1WWDw6G3KMgeaTKg="
==> Installing new gossip encryption key...

List Keys

Check the currently available keys:

consul keyring -list
==> Gathering installed encryption keys...

  +bLh5g5Q8N1RL2QQaY9U9z6I5Qi1WWDw6G3KMgeaTKg= [1/1]

instaryun (LAN):
  +bLh5g5Q8N1RL2QQaY9U9z6I5Qi1WWDw6G3KMgeaTKg= [2/2]

Update your Cluster to a new Key

consul keyring -use="+bLh5g5Q8N1RL2QQaY9U9z6I5Qi1WWDw6G3KMgeaTKg="
==> Changing primary gossip encryption key...

Remove old Keys

consul keyring -remove="myoldkey"
==> Removing gossip encryption key...