Hashicorp Consul in Production

Shenzhen, China

Installation
Security Credentials
- Gossip Encryption Key
- Generate TLS certificates for RPC encryption
Configure Consul
- Masters
- Minions
Ports
Run as a Service
Setup Consul environment variables
Enable Consul ACLs
Consul UI

Installation

To install the precompiled binary, download the appropriate package for your system.

wget https://releases.hashicorp.com/consul/1.10.3/consul_1.10.3_linux_amd64.zip
wget https://releases.hashicorp.com/consul/1.10.3/consul_1.10.3_SHA256SUMS

The SHA256SUMS shows me the corresponding check sum for this file:

50afd45daaffd3af5ab67b03ff616117eca9961014ca0ef25ed2aaa27a7be698  consul_1.10.3_linux_amd64.zip

The following command has to give you the same sum - if you downloaded the correct version of the file:

sha256sum consul_1.10.3_linux_amd64.zip
50afd45daaffd3af5ab67b03ff616117eca9961014ca0ef25ed2aaa27a7be698  consul_1.10.3_linux_amd64.zip

Now that we know that the zip container has not been tempered with we can unzip it to a place that is in our system PATH:

echo $PATH

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

unzip ./consul_1.10.3_linux_amd64.zip
rm ./consul_1.10.3_linux_amd64.zip
mv consul /usr/bin/consul

Verify that it is working:

consul -v
Consul v1.10.3
Revision c976ffd2d

The consul command features opt-in autocompletion for flags, subcommands, and arguments (where supported). Enable autocompletion:

consul -autocomplete-install
complete -C /usr/bin/consul consul

Create a unique, non-privileged system user to run Consul and create its data directory:

useradd --system --home /etc/consul.d --shell /bin/false consul
mkdir --parents /opt/consul
chown --recursive consul:consul /opt/consul

Security Credentials

Gossip Encryption Key

Gossip is encrypted with a symmetric key, since gossip between nodes is done over UDP. All agents must have the same encryption key. You can create the encryption key via the Consul CLI even though no Consul agents are running yet. Generate the encryption key:

consul keygen
qDOPBEr+/oUVeOFQOnVypxwDaHzLrD+lvjo5vCEBbZ0=

You will need to add the newly generated key to the encrypt option in the server configuration on all Consul agents. Save your key in a safe location. You will need to reference the key throughout the installation.

Generate TLS certificates for RPC encryption

Start by creating the CA on your admin instance, using the Consul CLI:

mkdir /opt/consul/certs
cd /opt/consul/certs
consul tls ca create

Next create a set of certificates, one for each Consul agent. You will need to select a name for your primary datacenter now, so that the certificates are named properly. First, for your Consul servers, use the following command to create a certificate for each server:

consul tls cert create -server -dc my_dc

Use the following command with the -client flag to create client certificates. The file name increments automatically:

consul tls cert create -client -dc my_dc

Now we need to copy the certificates to all master (server) and minion (client) servers into the /etc/consul.d/ directory:

mkdir /etc/consul.d

Masters

consul-agent-ca.pem
my_dc-server-consul-0-key.pem
my_dc-server-consul-0.pem

Minions

consul-agent-ca.pem
my_dc-client-consul-0-key.pem
my_dc-client-consul-0.pem

Configure Consul

touch /etc/consul.d/consul.hcl
chown --recursive consul:consul /etc/consul.d
chmod 640 /etc/consul.d/consul.hcl
nano /etc/consul.d/consul.hcl

Masters

/etc/consul.d/consul.hcl

datacenter = "my_dc"
node_name = "consul-server"
client_addr = "0.0.0.0"
bind_addr = "172.217.160.110"
advertise_addr = "172.217.160.110"
server    = true
bootstrap = true
ui_config {
  enabled = true
}
data_dir   = "/opt/consul"
log_level  = "INFO"
addresses {
  http = "0.0.0.0"
}
ports {
  https = 8501
}
connect {
  enabled = true
}
key_file = "/etc/consul.d/my_dc-server-consul-0-key.pem"
cert_file = "/etc/consul.d/my_dc-server-consul-0.pem"
ca_file = "/etc/consul.d/consul-agent-ca.pem"
encrypt = "qDOPBEr+/oUVeOFQOnVypxwDaHzLrD+lvjo5vCEBbZ0="
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
retry_join = ["157.240.7.35"]

WARNING The join or retry_join is a required parameter for the systemd process to complete successfully and send its notify signal on LAN join.

Minions

/etc/consul.d/consul.hcl

datacenter = "my_dc"
node_name = "consul-minion"
bind_addr = "157.240.7.35"
advertise_addr = "157.240.7.35"
server    = false
ui_config {
  enabled = false
}
data_dir   = "/opt/consul"
log_level  = "INFO"
addresses {
  http = "0.0.0.0"
}
ports {
  https = 8501
}
connect {
  enabled = true
}
key_file = "/etc/consul.d/my_dc-client-consul-0-key.pem"
cert_file = "/etc/consul.d/my_dc-client-consul-0.pem"
ca_file = "/etc/consul.d/consul-agent-ca.pem"
encrypt = "qDOPBEr+/oUVeOFQOnVypxwDaHzLrD+lvjo5vCEBbZ0="
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true

Ports

Consul requires up to 6 different ports to work properly, some on TCP, UDP, or both protocols. Below we document the requirements for each port:

Use	Default Ports
DNS: The DNS server (TCP and UDP)	8600
HTTP: The HTTP API (TCP Only)	8500
HTTPS: The HTTPs API	disabled (8501)*
gRPC: The gRPC API	disabled (8502)*
LAN Serf: The Serf LAN port (TCP and UDP)	8301
Wan Serf: The Serf WAN port (TCP and UDP)	8302
server: Server RPC address (TCP Only)	8300
Sidecar Proxy Min: Inclusive min port number to use for automatically assigned sidecar service registrations.	21000
Sidecar Proxy Max: Inclusive max port number to use for automatically assigned sidecar service registrations.	21255

*For HTTPS and gRPC the ports specified in the table are recommendations.

Those ports are used for:

DNS Interface Used to resolve DNS queries.
HTTP API This is used by clients to talk to the HTTP API.
HTTPS API (Optional) Is off by default, but port 8501 is a convention used by various tools as the default.
gRPC API (Optional) Currently gRPC is only used to expose the xDS API to Envoy proxies. It is off by default, but port 8502 is a convention used by various tools as the default. Defaults to 8502 in -dev mode.
Serf LAN This is used to handle gossip in the LAN. Required by all agents.
Serf WAN This is used by servers to gossip over the WAN, to other servers. As of Consul 0.8 the WAN join flooding feature requires the Serf WAN port (TCP/UDP) to be listening on both WAN and LAN interfaces.
Server RPC This is used by servers to handle incoming requests from other agents.

firewall-cmd --permanent --zone=public --add-port=8301/tcp --add-port=8301/udp --add-port=8302/tcp --add-port=8302/udp
firewall-cmd --reload
firewall-cmd --zone=public --list-all

ufw allow 8301
ufw allow 8302
ufw reload
ufw status

Run as a Service

Create a Consul service file:

nano /usr/lib/systemd/system/consul.service

or /usr/lib/systemd/user/consul.service

Add this configuration to the Consul service file:

[Unit]
Description="HashiCorp Consul"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl

[Service]
Type=notify
User=consul
Group=consul
ExecStart=/usr/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Check that your configuration file is valid, with the Consul CLI validate command:

consul validate /etc/consul.d/consul.hcl

bootstrap = true: do not enable unless necessary
Configuration is valid!

Enable and start Consul using the systemctl command:

systemctl enable --now  consul
service consul status

Setup Consul environment variables

Masters

export CONSUL_CACERT=/etc/consul.d/consul-agent-ca.pem
export CONSUL_CLIENT_CERT=/etc/consul.d/my_dc-server-consul-0.pem
export CONSUL_CLIENT_KEY=/etc/consul.d/my_dc-server-consul-0-key.pem

Minions

export CONSUL_CACERT=/etc/consul.d/consul-agent-ca.pem
export CONSUL_CLIENT_CERT=/etc/consul.d/my_dc-client-consul-0.pem
export CONSUL_CLIENT_KEY=/etc/consul.d/my_dc-client-consul-0-key.pem

Enable Consul ACLs

Add the ACL configuration to the consul.hcl configuration file and choose a default policy of "allow" (allow all traffic unless explicitly denied) or "deny" (deny all traffic unless explicitly allowed).

acl = {
  enabled = true
  default_policy = "allow"
  enable_token_persistence = true
}
performance {
  raft_multiplier = 1
}

raft_multiplier - An integer multiplier used by Consul servers to scale key Raft timing parameters. Setting this to a value of 1 will configure Raft to its highest-performance mode, equivalent to the default timing of Consul prior to 0.7, and is recommended for production Consul servers.

Generate the bootstrap token from your master server:

consul acl bootstrap

This will return the Consul bootstrap token. You will need the SecretID for all subsequent Consul API requests (including CLI and UI). Ensure that you save the SecretID.

Set CONSUL_MGMT_TOKEN env variable:

export CONSUL_HTTP_TOKEN="<Token SecretID from previous step>"
export CONSUL_MGMT_TOKEN="<Token SecretID from previous step>"

Create a node policy file with write access for nodes related actions and read access for service related actions:

mkdir /etc/consul.d/policies
nano /etc/consul.d/policies/node-policy.hcl

node-policy.hcl

agent_prefix "" {
  policy = "write"
}
node_prefix "" {
  policy = "write"
}
service_prefix "" {
  policy = "read"
}
session_prefix "" {
  policy = "read"
}

Generate the Consul node ACL policy with the newly created policy file:

cd /etc/consul.d/policies

consul acl policy create \
  -token=${CONSUL_MGMT_TOKEN} \
  -name node-policy \
  -rules @node-policy.hcl

ID:           secret
Name:         node-policy
Description:
Datacenters:
Rules:
agent_prefix "" {
  policy = "write"
}
node_prefix "" {
  policy = "write"
}
service_prefix "" {
  policy = "read"
}
session_prefix "" {
  policy = "read"
}

Create the node token with the newly created policy:

consul acl token create \
  -token=${CONSUL_MGMT_TOKEN} \
  -description "node token" \
  -policy-name node-policy

AccessorID:       secret
SecretID:         secret
Description:      node token
Local:            false
Create Time:      2021-10-25 07:28:39.420612336 +0200 CEST
Policies:
   secret - node-policy

On all Consul Servers add the node token.

consul acl set-agent-token \
  -token="<Management Token SecretID>" \
  agent "<Node Token SecretID>"

ACL token "agent" set successfully

Consul UI

I did not forward the HTTP port, but I can still test the Nomad UI by tunnelling the user interface through SSH onto your local server:

ssh myuser@my-server-ip -p ssh-port -L8500:localhost:8500

Access the Nodes tab and you should see both the Master and Minion server connected:

http://localhost:8500/ui/my_dc/nodes

Installation​

Security Credentials​

Gossip Encryption Key​

Generate TLS certificates for RPC encryption​

Configure Consul​

Masters​

Minions​

Ports​

Run as a Service​

Setup Consul environment variables​

Enable Consul ACLs​

Consul UI​