Skip to main content

Hashicorp Consul Refresher - Key Value Store

Mongkok, Hongkong

Consul includes a key value store, which you can use to dynamically configure applications. Everything stored inside the KV Store is not encrypted! I cannot be used for sensitive data like logins, etc. Those have to be stored in Hashicorp Vault.

Add Data

First, insert or PUT some values into the KV store with the consul kv put command. The first entry after the command is the key, and the second entry is the value:

consul kv put redis/config/minconns 1
Success! Data written to: redis/config/minconns

Hashicorp Consul KV

Query Data

Now, query for the value of one of the keys you just entered:

consul kv get redis/config/minconns

You can also get some more -detailed information:

consul kv get -detailed redis/config/minconns

CreateIndex      11763
Flags            0
Key              redis/config/minconns
LockIndex        0
ModifyIndex      11763
Session          -
Value            1

List all the keys in the store using the -recurse options. Results are returned in lexicographical order:

consul kv get -recurse

Delete Data

Delete a key from the Consul KV store, issue a delete call.

consul kv delete redis/config/minconns
Success! Deleted key: redis/config/minconns

Consul lets you interact with keys in a folder-like way. Delete all the keys with the redis prefix using the -recurse option:

consul kv delete -recurse redis
Success! Deleted keys with prefix: redis

consul kv get redis/config/minconns
Error! No key exists at: redis/config/minconns


You can also use the REST API to store values:

curl --request PUT --data 'active'

And recall them with a GET request:

curl --request GET | jq

    "LockIndex": 0,
    "Key": "redis/state",
    "Flags": 0,
    "Value": "YWN0aXZl",
    "CreateIndex": 11924,
    "ModifyIndex": 11924

Values are stored base64 encoded and have to be decoded to become readable:

echo 'YWN0aXZl' | base64 --decode

And everything can be deleted again with:

curl --request DELETE

Working with KV

Monitor KV Changes

You can trigger actions when a value is changed - e.g. triggering a script execution:

consul watch -type=key -key=db/update/postgres /opt/consul/scripts/

Writing to Environment

You can write Key/Values to environment variables on your minion server using envconsul. Download the latest version on your minion:

mv envconsul /usr/local/bin
envconsul -v

envconsul v0.11.0 (9f66f6c5)

We can now add a few values to the Consul KV store on our master server that we need to be available to the local instance of our web app - e.g.:

consul kv put frontend/elasticsearch/connection_string ''
consul kv put frontend/elasticsearch/index 'dev_2021_08_23'
consul kv put frontend/elasticsearch/_type '_doc'

Make sure the minion server has access to those variables:

consul kv get -recurse


Now we can use envconsul to read those entries and write them into the system environment:

envconsul -upcase -prefix frontend/elasticsearch env


Creating Configuration Files

Hashicorp Consul Template is a Template rendering, notifier, and supervisor for @hashicorp Consul and Vault data. This project provides a convenient way to populate values from Consul into the file system using the consul-template daemon.

The daemon consul-template queries a Consul or Vault cluster and updates any number of specified templates on the file system. As an added bonus, it can optionally run arbitrary commands when the update process completes. Please see the examples folder for some scenarios where this functionality might prove useful. Download the latest version and unzip it to /usr/local/bin on your minion server:

mv consul-template /usr/local/bin
consul-template -v

consul-template v0.27.0 (d4af0222)

We can now create a configuration template file that can be populated with data from the Consul K/V store:

nano es_config.json.tmpl
connection_string: {{ key "frontend/elasticsearch/connection_string" }}
index: {{ key "frontend/elasticsearch/index" }}
_type_: {{ key "frontend/elasticsearch/_type" }}

We can now use consul-template to populate the configuration file for us:

consul-template -template "es_config.json.tmpl:config.json" -once

Without the -once flag the configuration will automatically be update every time a value in the Consul KV store is updated!

cat config.json

index: dev_2021_08_23
_type_: _doc