Installing HashiCorp Nomad on Ubuntu Server 20.04
Introduction
Nomad simple and flexible workload orchestrator to deploy and manage containers and non-containerized applications across on-prem and clouds at scale.
-
Simple and Lightweight
- Single binary that integrates into existing infrastructure. Easy to operate on-prem or in the cloud with minimal overhead.
-
Flexible Workload Support
- Orchestrate applications of any type - not just containers. First class support for Docker, Windows, Java, VMs, and more.
-
Modernize Legacy Applications without Rewrite
- Bring orchestration benefits to existing services. Achieve zero downtime deployments, improved resilience, higher resource utilization, and more without containerization.
-
Easy Federation at Scale
- Single command for multi-region, multi-cloud federation. Deploy applications globally to any region using Nomad as a single unified control plane.
-
Multi-Cloud with Ease
- One single unified workflow for deploying to bare metal or cloud environments. Enable multi-cloud applications with ease.
-
Native Integrations with Terraform, Consul, and Vault
- Nomad integrates seamlessly with Terraform, Consul and Vault for provisioning, service networking, and secrets management.
Installing Nomad as APT Package
Add the HashiCorp GPG key.
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
Add the official HashiCorp Linux repository.
apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
Update and install.
apt-get update && apt-get install nomad
To verify Nomad was installed correctly, try the nomad command.
nomad
Ports Used
Nomad requires 3 different ports to work properly on servers and 2 on clients, some on TCP, UDP, or both protocols:
-
HTTP API (Default
4646): This is used by clients and servers to serve the HTTP API. TCP only. -
RPC (Default
4647): This is used for internal RPC communication between client agents and servers, and for inter-server traffic. TCP only. -
Serf WAN (Default
4648): This is used by servers to gossip both over the LAN and WAN to other servers. It isn't required that Nomad clients can reach this address. TCP and UDP.
FirewallD
firewall-cmd --permanent --zone=public --add-port=4646-4647/tcp
firewall-cmd --permanent --zone=public --add-port=4648/tcp
firewall-cmd --permanent --zone=public --add-port=4648/udp
firewall-cmd --reload
firewall-cmd --zone=public --list-all
ufw
ufw allow 4646:4648/tcp
ufw allow 4648/udp
ufw reload
Nomad Agent
Nomad relies on a long running agent on every machine in the cluster. The agent can run either in server or client mode. The cluster servers are responsible for managing the cluster. All other agents in the cluster should be in client mode. A Nomad client is a very lightweight process that registers the host machine, performs heartbeating, and runs the tasks that are assigned to it by the servers. The agent must be run on every node that is part of the cluster so that the servers can assign work to those machines.
In this guide, you will start the Nomad agent in development mode. This mode is used to quickly start an agent that is acting as a client and server to test job configurations or prototype interactions. Start a single Nomad agent in development mode with the nomad agent command. Note, this command should not be used in production as it does not persist state.
nomad agent -dev
Wait to continue to the next section until you see the agent has acquired leadership:
2020-08-28T09:23:18.317Z [WARN] nomad.raft: heartbeat timeout reached, starting election: last-leader=
2020-08-28T09:23:18.317Z [INFO] nomad.raft: entering candidate state: node="Node at 127.0.0.1:4647 [Candidate]" term=2
2020-08-28T09:23:18.317Z [DEBUG] nomad.raft: votes: needed=1
2020-08-28T09:23:18.317Z [DEBUG] nomad.raft: vote granted: from=127.0.0.1:4647 term=2 tally=1
2020-08-28T09:23:18.317Z [INFO] nomad.raft: election won: tally=1
2020-08-28T09:23:18.317Z [INFO] nomad.raft: entering leader state: leader="Node at 127.0.0.1:4647 [Leader]"
2020-08-28T09:23:18.317Z [INFO] nomad: cluster leadership acquired
In another terminal, use nomad node status to view the registered nodes of the Nomad cluster.
nomad node status
ID DC Name Class Drain Eligibility Status
425b29b5 dc1 salt-master <none> false eligible ready
The output shows your Node ID, its datacenter, node name, node class, drain mode and current status. The Node ID is a randomly generated UUID. Notice that your node is in the ready state and task draining is currently off.
The agent is also in server mode, which means it is part of the gossip protocol used to connect all the server instances together:
nomad server members
Name Address Port Status Leader Protocol Build Datacenter Region
salt-master.global 127.0.0.1 4648 alive true 2 0.12.3 dc1 global
The output shows your agent, the address it is running on, its health state, some version information, and the datacenter and region. Additional metadata can be viewed by providing the -detailed flag:
nomad server members -detailed
Name Address Port Tags
salt-master.global 127.0.0.1 4648 build=0.12.3,bootstrap=1,role=nomad,vsn=1,rpc_addr=127.0.0.1,mvn=1,raft_vsn=2,region=global,expect=1,id=99ee6e58-43bc-6f5c-e1f7-8e6a3194f433,dc=dc1,port=4647
Jobs
Jobs are the primary configuration that users interact with when using Nomad. A job is a declarative specification of tasks that Nomad should run. The job created by running nomad job init uses the Docker task driver. To run it, you will need a Nomad client available with Docker installed.
To get started, use the job init command which generates a skeleton job file:
mkdir ~/nomad
cd ~/nomad
nomad job init
Example job file written to example.nomad inside your current working directory. This example job file declares a single task named redis, which uses the Docker driver to run the a Redis container.:
job "example" {
...
task "redis" {
driver = "docker"
config {
image = "redis:3.2"
port_map {
db = 6379
}
}
}
...
}
The primary way you interact with Nomad is with the job run command. The run command takes a job file and registers it with Nomad. This is used both to register new jobs and to update existing jobs:
nomad job run example.nomad
==> Monitoring evaluation "9271b1e8"
Evaluation triggered by job "example"
Allocation "9a93c7ff" created: node "425b29b5", group "cache"
Evaluation within deployment: "b5ddfc12"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "9271b1e8" finished with status "complete"
If you receive the following Error message you don't have Docker installed or the Docker daemon is not loaded:
Task Group "cache" (failed to place 1 allocation):
* Constraint "missing drivers": 1 nodes excluded by filter
To inspect the status of your job you use the status command:
nomad status example
ID = example
Name = example
Submit Date = 2020-08-28T11:38:59Z
Type = service
Priority = 50
Datacenters = dc1
Namespace = default
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
cache 0 0 1 0 0 0
Latest Deployment
ID = b5ddfc12
Status = running
Description = Deployment is running
Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
cache 1 1 0 1 2020-08-28T11:48:59Z
Allocations
ID Node ID Task Group Version Desired Status Created Modified
9a93c7ff 425b29b5 cache 0 run running 3m14s ago 14s ago
The last entry Allocation represents the instance described by the task that is now placed on your node:
nomad alloc status 9a93c7ff
ID = 9a93c7ff-1e61-1277-63aa-95ad42d089f7
Eval ID = 9271b1e8
Name = example.cache[0]
Node ID = 425b29b5
Node Name = salt-master
Job ID = example
Job Version = 0
Client Status = running
Client Description = Tasks are running
Desired Status = run
Desired Description = <none>
Created = 6m21s ago
Modified = 3m21s ago
Deployment ID = b5ddfc12
Deployment Health = unhealthy
Task "redis" is "running"
Task Resources
CPU Memory Disk Addresses
2/500 MHz 988 KiB/256 MiB 300 MiB db: 127.0.0.1:27335
Task Events:
Started At = 2020-08-28T11:40:01Z
Finished At = N/A
Total Restarts = 0
Last Restart = N/A
Recent Events:
Time Type Description
2020-08-28T11:41:59Z Alloc Unhealthy Task not running for min_healthy_time of 10s by deadline
2020-08-28T11:40:01Z Started Task started by client
2020-08-28T11:38:59Z Driver Downloading image
2020-08-28T11:38:59Z Task Setup Building Task Directory
2020-08-28T11:38:59Z Received Task received by client
To see the logs of a task, use the alloc logs command:
nomad alloc logs 9a93c7ff redis
1:C 28 Aug 11:40:01.407 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 3.2.12 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 1
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
After modifying the job specification, use the job plan command to invoke a dry-run of the scheduler to see what would happen if you ran the updated job:
nomad job plan example.nomad
The final step in a job lifecycle is stopping the job. This is done with the job stop command:
nomad job stop example
nomad job status example
ID = example
Name = example
Submit Date = 2020-08-28T11:38:59Z
Type = service
Priority = 50
Datacenters = dc1
Namespace = default
Status = dead (stopped)
Periodic = false
Parameterized = false