Hashicorp Nomad - Working with Consul Connect
Setup Failure
Trying to use Consul Connect in a Nomad job file:
failed to setup alloc: pre-run hook "network" failed: failed to configure networking for alloc: failed to configure network: plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin]
CNI Plugins
Nomad uses CNI plugins to configure the network namespace used to secure the Consul service mesh sidecar proxy. All Nomad client nodes using network namespaces must have CNI plugins installed:
curl -L -o cni-plugins.tgz "https://github.com/containernetworking/plugins/releases/download/v1.0.0/cni-plugins-linux-$( [ $(uname -m) = aarch64 ] && echo arm64 || echo amd64)"-v1.0.0.tgz
sudo mkdir -p /opt/cni/bin
sudo tar -C /opt/cni/bin -xzf cni-plugins.tgz
Ensure the your Linux operating system distribution has been configured to allow container traffic through the bridge network to be routed via iptables. These tunables can be set as follows:
echo 1 | tee /proc/sys/net/bridge/bridge-nf-call-arptables
echo 1 | tee /proc/sys/net/bridge/bridge-nf-call-ip6tables
echo 1 | tee /proc/sys/net/bridge/bridge-nf-call-iptables
To preserve these settings on startup of a client node, add a file including the following to /etc/sysctl.d/ or remove the file your Linux distribution puts in that directory.
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
Sidecar Required Ports
Before running Consul, you should ensure the following bind ports are accessible. Below is a complete list of ports that need to be opened inside your Consul cluster:
| Use | Default Ports |
|---|---|
| DNS: The DNS server (TCP and UDP) | 8600 |
| HTTP: The HTTP API (TCP Only) | 8500 |
| HTTPS: The HTTPs API disabled | (8501)* |
| gRPC: The gRPC API disabled | (8502)* |
| LAN Serf: The Serf LAN port (TCP and UDP) | 8301 |
| Wan Serf: The Serf WAN port (TCP and UDP) | 8302 |
| server: Server RPC address (TCP Only) | 8300 |
| Sidecar Proxy Min: Inclusive min port number to use for automatically assigned sidecar service registrations. | 21000 |
| Sidecar Proxy Max: Inclusive max port number to use for automatically assigned sidecar service registrations. | 21255 |
* For HTTPS and gRPC the ports specified in the table are recommendations.
E.g. Uncomplicated Firewall:
ufw allow 8300,8301,8302,8500,8502,8600/tcp
ufw allow 8301,8302,8502,8600/udp
ufw allow 21000:21255/tcp
Continues here Hashicorp Nomad - Working with Consul Connect Part II