Hashicorp Consul Refresher - Key Value Store

Mongkok, Hongkong

Add Data
Query Data
Delete Data
Consul REST API
Working with KV

Consul includes a key value store, which you can use to dynamically configure applications. Everything stored inside the KV Store is not encrypted! I cannot be used for sensitive data like logins, etc. Those have to be stored in Hashicorp Vault.

Add Data

First, insert or PUT some values into the KV store with the consul kv put command. The first entry after the command is the key, and the second entry is the value:

consul kv put redis/config/minconns 1
Success! Data written to: redis/config/minconns

Hashicorp Consul KV

Query Data

Now, query for the value of one of the keys you just entered:

consul kv get redis/config/minconns
1

You can also get some more -detailed information:

consul kv get -detailed redis/config/minconns

CreateIndex      11763
Flags            0
Key              redis/config/minconns
LockIndex        0
ModifyIndex      11763
Session          -
Value            1

List all the keys in the store using the -recurse options. Results are returned in lexicographical order:

consul kv get -recurse
redis/config/minconns:1

Delete Data

Delete a key from the Consul KV store, issue a delete call.

consul kv delete redis/config/minconns
Success! Deleted key: redis/config/minconns

Consul lets you interact with keys in a folder-like way. Delete all the keys with the redis prefix using the -recurse option:

consul kv delete -recurse redis
Success! Deleted keys with prefix: redis

consul kv get redis/config/minconns
Error! No key exists at: redis/config/minconns

Consul REST API

You can also use the REST API to store values:

curl --request PUT --data 'active' http://192.168.2.110:8500/v1/kv/redis/state
true

And recall them with a GET request:

curl --request GET http://192.168.2.110:8500/v1/kv/redis/state | jq

[
  {
    "LockIndex": 0,
    "Key": "redis/state",
    "Flags": 0,
    "Value": "YWN0aXZl",
    "CreateIndex": 11924,
    "ModifyIndex": 11924
  }
]

Values are stored base64 encoded and have to be decoded to become readable:

echo 'YWN0aXZl' | base64 --decode
active

And everything can be deleted again with:

curl --request DELETE http://192.168.2.110:8500/v1/kv/redis/state
true

Working with KV

Monitor KV Changes

You can trigger actions when a value is changed - e.g. triggering a script execution:

consul watch -type=key -key=db/update/postgres /opt/consul/scripts/update_postgres.sh

Writing to Environment

You can write Key/Values to environment variables on your minion server using envconsul. Download the latest version on your minion:

wget https://releases.hashicorp.com/envconsul/0.11.0/envconsul_0.11.0_linux_amd64.zip
unzip envconsul_0.11.0_linux_amd64.zip
mv envconsul /usr/local/bin
envconsul -v

envconsul v0.11.0 (9f66f6c5)

We can now add a few values to the Consul KV store on our master server that we need to be available to the local instance of our web app - e.g.:

consul kv put frontend/elasticsearch/connection_string 'https://elasticsearch.mydomain.com'
consul kv put frontend/elasticsearch/index 'dev_2021_08_23'
consul kv put frontend/elasticsearch/_type '_doc'

Make sure the minion server has access to those variables:

consul kv get -recurse

frontend/elasticsearch/_type:_doc
frontend/elasticsearch/connection_string:https://elasticsearch.mydomain.com
frontend/elasticsearch/index:dev_2021_08_23

Now we can use envconsul to read those entries and write them into the system environment:

envconsul -upcase -prefix frontend/elasticsearch env

_TYPE=_doc
INDEX=dev_2021_08_23
CONNECTION_STRING=https://elasticsearch.mydomain.com

Creating Configuration Files

Hashicorp Consul Template is a Template rendering, notifier, and supervisor for @hashicorp Consul and Vault data. This project provides a convenient way to populate values from Consul into the file system using the consul-template daemon.

The daemon consul-template queries a Consul or Vault cluster and updates any number of specified templates on the file system. As an added bonus, it can optionally run arbitrary commands when the update process completes. Please see the examples folder for some scenarios where this functionality might prove useful. Download the latest version and unzip it to /usr/local/bin on your minion server:

wget https://releases.hashicorp.com/consul-template/0.27.0/consul-template_0.27.0_linux_amd64.zip
unzip consul-template_0.27.0_linux_amd64.zip
mv consul-template /usr/local/bin
consul-template -v

consul-template v0.27.0 (d4af0222)

We can now create a configuration template file that can be populated with data from the Consul K/V store:

nano es_config.json.tmpl

connection_string: {{ key "frontend/elasticsearch/connection_string" }}
index: {{ key "frontend/elasticsearch/index" }}
_type_: {{ key "frontend/elasticsearch/_type" }}

We can now use consul-template to populate the configuration file for us:

consul-template -template "es_config.json.tmpl:config.json" -once

Without the -once flag the configuration will automatically be update every time a value in the Consul KV store is updated!

cat config.json

connection_string: https://elasticsearch.mydomain.com
index: dev_2021_08_23
_type_: _doc

Add Data​

Query Data​

Delete Data​

Consul REST API​

Working with KV​

Monitor KV Changes​

Writing to Environment​

Creating Configuration Files​