Skip to main content

Hashicorp Nomad - Working with Consul Connect

Shen Zhen, China

Setup Failure

Trying to use Consul Connect in a Nomad job file:

failed to setup alloc: pre-run hook "network" failed: failed to configure networking for alloc: failed to configure network: plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin]

CNI Plugins

Nomad uses CNI plugins to configure the network namespace used to secure the Consul service mesh sidecar proxy. All Nomad client nodes using network namespaces must have CNI plugins installed:

curl -L -o cni-plugins.tgz "$( [ $(uname -m) = aarch64 ] && echo arm64 || echo amd64)"-v1.0.0.tgz
sudo mkdir -p /opt/cni/bin
sudo tar -C /opt/cni/bin -xzf cni-plugins.tgz

Ensure the your Linux operating system distribution has been configured to allow container traffic through the bridge network to be routed via iptables. These tunables can be set as follows:

echo 1 | tee /proc/sys/net/bridge/bridge-nf-call-arptables
echo 1 | tee /proc/sys/net/bridge/bridge-nf-call-ip6tables
echo 1 | tee /proc/sys/net/bridge/bridge-nf-call-iptables

To preserve these settings on startup of a client node, add a file including the following to /etc/sysctl.d/ or remove the file your Linux distribution puts in that directory.

net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

Sidecar Required Ports

Before running Consul, you should ensure the following bind ports are accessible. Below is a complete list of ports that need to be opened inside your Consul cluster:

UseDefault Ports
DNS: The DNS server (TCP and UDP)8600
HTTP: The HTTP API (TCP Only)8500
HTTPS: The HTTPs API disabled(8501)*
gRPC: The gRPC API disabled(8502)*
LAN Serf: The Serf LAN port (TCP and UDP)8301
Wan Serf: The Serf WAN port (TCP and UDP)8302
server: Server RPC address (TCP Only)8300
Sidecar Proxy Min: Inclusive min port number to use for automatically assigned sidecar service registrations.21000
Sidecar Proxy Max: Inclusive max port number to use for automatically assigned sidecar service registrations.21255

* For HTTPS and gRPC the ports specified in the table are recommendations.

E.g. Uncomplicated Firewall:

ufw allow 8300,8301,8302,8500,8502,8600/tcp
ufw allow 8301,8302,8502,8600/udp
ufw allow 21000:21255/tcp

Continues here Hashicorp Nomad - Working with Consul Connect Part II