Skip to main content

Installing HashiCorp Consul on Ubuntu Server 20.04

Victoria Harbour, Hong Kong

Consul is a networking tool that provides a fully featured service-mesh control plane and service discovery. Consul also includes a key-value store for service configuration. Learn how to perform common Consul operations locally.

Install Consul on Ubuntu 20.04​

The first thing you need to do in order to use Consul is install it. In a production deployment you would install Consul on every node where you want to register services:

$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -

Add the official HashiCorp Linux repository:

sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"

Update and install.

sudo apt-get update && sudo apt-get install consul

After installing Consul, verify that the installation worked by opening a new terminal session and running the command :

consul
Usage: consul [--version] [--help] <command> [<args>]

Ports Used​

(only additional ports to the Nomad service)

firewall-cmd --permanent --zone=public --add-port=8500/tcp
firewall-cmd --permanent --zone=public --add-port=8301/tcp
firewall-cmd --reload
firewall-cmd --zone=public --list-all

Start the Consul Agent in Dev Mode​

Lets start our local agent in development mode, which is an in memory server mode with some common features enabled (despite security risks) for ease of use, and all persistence options turned off:

consul agent -dev

Check the membership of the Consul datacenter by running the following command in a new terminal window:

consul members

Node         Address         Status  Type    Build  Protocol  DC   Segment
salt-master  127.0.0.1:8301  alive   server  1.8.3  2         dc1  <all>

The output displays your agent, its IP address, its health state, its role in the datacenter, and some version information. You can discover additional metadata by providing the -detailed flag.

consul members --detailed
Node         Address         Status  Tags
salt-master  127.0.0.1:8301  alive   acls=0,build=1.8.3:a9322b9c,dc=dc1,ft_fs=1,id=6f0b4bf5-a294-20f3-a3e1-c4f9d3e75de7,port=8300,raft_vsn=3,role=consul,segment=<all>,vsn=2,vsn_max=3,vsn_min=2,wan_join_port=8302

For a strongly consistent view of the world, query the HTTP API, which forwards the request to the Consul servers.

curl localhost:8500/v1/catalog/nodes
[
    {
        "ID": "6f0b4bf5-a294-20f3-a3e1-c4f9d3e75de7",
        "Node": "salt-master",
        "Address": "127.0.0.1",
        "Datacenter": "dc1",
        "TaggedAddresses": {
            "lan": "127.0.0.1",
            "lan_ipv4": "127.0.0.1",
            "wan": "127.0.0.1",
            "wan_ipv4": "127.0.0.1"
        },
        "Meta": {
            "consul-network-segment": ""
        },
        "CreateIndex": 10,
        "ModifyIndex": 12
    }
]

This will gracefully stop the agent, causing it to leave the Consul datacenter and shut down:

consul leave

Graceful leave complete

When you issue the leave command, Consul notifies other members that the agent left the datacenter. Forcibly killing the agent process indicates to other agents in the Consul datacenter that the node failed instead of left. When a node fails, its health is marked as critical, but it is not removed from the catalog.

Consul Service Discovery​

One of the major use cases for Consul is service discovery. Consul knows where these services are located because each service registers with its local Consul client. Operators can register services manually, configuration management tools can register services when they are deployed, or container orchestration platforms can register services automatically via integrations.

Define a Service​

You can register services either by providing a service definition, which is the most common way to register services, or by making a call to the HTTP API.

First, create a directory for Consul configuration. Consul loads all configuration files in the configuration directory, so a common convention on Unix systems is to name the directory something like:

mkdir /etc/consul.d
nano /etc/consul.d/web.json

Pretend there is a service named "web" running on port 80. This file will contain the service definition: name, port, and an optional tag you can use to find the service later on:

{
  "service": {
    "name": "web",
    "tags": ["frontend"],
    "port": 80
  }
}

Note: We never started a web service in this example. Consul can register services that aren't running yet. It correlates each running service with its registration based on the service's port.

Now, restart the agent, using command line flags to specify the configuration directory and enable script checks on the agent.

Security Warning: Enabling script checks in some configurations may introduce a remote execution vulnerability which is known to be targeted by malware. In production we strongly recommend -enable-local-script-checks instead.

consul agent -dev -enable-script-checks -config-dir=/etc/consul.d

We can use the HTTP API to see if the service was registered:

curl localhost:8500/v1/catalog/services

{
    "consul": [],
    "nomad": [
        "http",
        "serf",
        "rpc"
    ],
    "nomad-client": [
        "http"
    ],
    "web": [
        "frontend"
    ]
}

And get some details for the service itself:

curl http://localhost:8500/v1/catalog/service/web
[
    {
        "ID": "b6a5758f-b955-df07-d7a1-c9f26853641a",
        "Node": "salt-master",
        "Address": "127.0.0.1",
        "Datacenter": "dc1",
        "TaggedAddresses": {
            "lan": "127.0.0.1",
            "lan_ipv4": "127.0.0.1",
            "wan": "127.0.0.1",
            "wan_ipv4": "127.0.0.1"
        },
        "NodeMeta": {
            "consul-network-segment": ""
        },
        "ServiceKind": "",
        "ServiceID": "web",
        "ServiceName": "web",
        "ServiceTags": [
            "frontend"
        ],
        "ServiceAddress": "",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {},
        "ServicePort": 80,
        "ServiceEnableTagOverride": false,
        "ServiceProxy": {
            "MeshGateway": {},
            "Expose": {}
        },
        "ServiceConnect": {},
        "CreateIndex": 13,
        "ModifyIndex": 13
    }
]

The HTTP API lists all nodes hosting a given service. You'll typically want to filter your query for only healthy service instances, which DNS does automatically under the hood. Filter your HTTP API query to look for only healthy instances:

curl 'http://localhost:8500/v1/health/service/web?passing'
[
    {
        "Node": {
            "ID": "b6a5758f-b955-df07-d7a1-c9f26853641a",
            "Node": "salt-master",
            "Address": "127.0.0.1",
            "Datacenter": "dc1",
            "TaggedAddresses": {
                "lan": "127.0.0.1",
                "lan_ipv4": "127.0.0.1",
                "wan": "127.0.0.1",
                "wan_ipv4": "127.0.0.1"
            },
            "Meta": {
                "consul-network-segment": ""
            },
            "CreateIndex": 10,
            "ModifyIndex": 12
        },
        "Service": {
            "ID": "web",
            "Service": "web",
            "Tags": [
                "frontend"
            ],
            "Address": "",
            "Meta": null,
            "Port": 80,
            "Weights": {
                "Passing": 1,
                "Warning": 1
            },
            "EnableTagOverride": false,
            "Proxy": {
                "MeshGateway": {},
                "Expose": {}
            },
            "Connect": {},
            "CreateIndex": 13,
            "ModifyIndex": 13
        },
        "Checks": [
            {
                "Node": "salt-master",
                "CheckID": "serfHealth",
                "Name": "Serf Health Status",
                "Status": "passing",
                "Notes": "",
                "Output": "Agent alive and reachable",
                "ServiceID": "",
                "ServiceName": "",
                "ServiceTags": [],
                "Type": "",
                "Definition": {},
                "CreateIndex": 10,
                "ModifyIndex": 10
            }
        ]
    }
]

We can also query the web service using Consul's DNS interface. The DNS name for a service registered with Consul is NAME.service.consul, where NAME is the name you used to register the service (in this case, web):

dig @127.0.0.1 -p 8600 web.service.consul

; <<>> DiG 9.16.1-Ubuntu <<>> @127.0.0.1 -p 8600 web.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55751
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;web.service.consul.            IN      A

;; ANSWER SECTION:
web.service.consul.     0       IN      A       127.0.0.1

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Sat Aug 29 16:20:51 UTC 2020
;; MSG SIZE  rcvd: 63

As you can verify from the output, an A record was returned containing the IP address where the service was registered. A records can only hold IP addresses. You can also use the DNS interface to retrieve the entire address/port pair as a SRV record:

dig @127.0.0.1 -p 8600 web.service.consul SRV

; <<>> DiG 9.16.1-Ubuntu <<>> @127.0.0.1 -p 8600 web.service.consul SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63249
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;web.service.consul.            IN      SRV

;; ANSWER SECTION:
web.service.consul.     0       IN      SRV     1 1 80 salt-master.node.dc1.consul.

;; ADDITIONAL SECTION:
salt-master.node.dc1.consul. 0  IN      A       127.0.0.1
salt-master.node.dc1.consul. 0  IN      TXT     "consul-network-segment="

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Sat Aug 29 16:21:17 UTC 2020
;; MSG SIZE  rcvd: 146

You can also use the DNS interface to filter services by tags dig @127.0.0.1 -p 8600 frontend.service.consul SRV.

Service Health Check​

Next you'll update the web service by registering a health check for it. Remember that because you never started a service on port 80 where you registered web, the health check you register will fail.

You can update service definitions without any downtime by changing the service definition file and sending a SIGHUP to the agent or running consul reload.

First, edit the registration file by running the following command:

nano /etc/consul.d/web.json

And add the following content:

{
  "service": {
    "name": "web",
    "tags": ["frontend"],
    "port": 80,
    "check": {
      "args": ["curl", "localhost"],
      "interval": "10s"
    }
  }
}

The check stanza of this service definition adds a script-based health check that tries to connect to the web service every 10 seconds via curl. If the command exits with an exit code >= 2, then the check will fail and Consul will consider the service unhealthy. An exit code of 1 will be considered as warning state.

Now reload Consul's configuration to make it aware of the new health check:

consul reload

Configuration reload triggered

You will now start seeing Critical Warnings in the Consul log every 10 seconds since the health check is failing:

2020-08-29T16:34:20.113Z [WARN]  agent: Check is now critical: check=service:web

Consul's DNS server only returns healthy results. Query DNS for the web service again. It shouldn't return any IP addresses since web's health check is failing.

dig @127.0.0.1 -p 8600 web.service.consul

; <<>> DiG 9.16.1-Ubuntu <<>> @127.0.0.1 -p 8600 web.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 5462
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;web.service.consul.            IN      A

;; AUTHORITY SECTION:
consul.                 0       IN      SOA     ns.consul. hostmaster.consul. 1598718956 3600 600 86400 0

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Sat Aug 29 16:35:56 UTC 2020
;; MSG SIZE  rcvd: 97

Also the HTTP API only returns an empty array now:

curl 'http://localhost:8500/v1/health/service/web?passing'

[]

I am going to use httpster to spin up an Node.js based web server on port 80:

mkdir ~/test && cd ~/test

nano index.html  //add a simple "Hello World"

npm install -g httpster  //requires Node.js and NPM to be installed on your server

httpster -p 80

The WebUI​

You can now open your servers IP address and should be able to see your Hello World website. Let's see if the service is now back up and healthy. Open your web browser on your server IP address and port 8500 - e.g. http://192.168.2.110:8500/:

All Service Checks Passing!

Consul Service Discovery