Hashicorp Waypoint with Nomad
Let's use Waypoint to deploy an application into a Nomad Cluster.
Installation
Get Waypoint by downloading the latest a pre-compiled waypoint binary:
wget https://releases.hashicorp.com/waypoint/0.9.0/waypoint_0.9.0_linux_amd64.zip
unzip waypoint_0.9.0_linux_amd64.zip && rm waypoint_0.9.0_linux_amd64.zip
mv ./waypoint /usr/bin/waypoint
chmod +x /usr/bin/waypoint
Verify the installation:
waypoint -version
CLI: v0.9.0 (2a7b89036)
Use Waypoint with Nomad
I am following the getting started guide by Hashicorp to set up an NodeJS app inside a Docker container using waypoint. The Example code can be found on Github. Start by cloning the Waypoint Examples repository and navigate to the NodeJS example:
git clone https://github.com/hashicorp/waypoint-examples.git
cd waypoint-examples/nomad/nodejs
Persistence
To be able to persist data we first need to add a host volume to our Nomad minion. Add the following configs in your client.hcl file:
client {
enabled = true
servers = ["myhost:port"]
host_volume "waypoint" {
path = "/opt/waypoint_data"
read_only = false
}
}
# Docker Configuration
plugin "docker" {
volumes {
enabled = true
}
}
Create the /opt/waypoint_data directory and I am not sure about the permissions needed here. The documentation says that you have to chown <Nomad User>:
chown nomad:nomad /opt/waypoint_data/
chmod 775 /opt/waypoint_data/
Set up the Waypoint Server
To install Waypoint with host volume persistent storage into the default data center dc1, run:
waypoint install -platform=nomad -accept-tos -nomad-host=https://localhost:4646 -nomad-dc=dc1 -nomad-host-volume=waypoint -nomad-runner-host-volume=waypoint -nomad-consul-service=true -nomad-consul-datacenter=dc1
Error installing the runner: please include '-nomad-csi-volume-provider' or '-nomad-host-volume' Github Issue
You can deactivate the Consul integration if the service is not present on your system.
The docker container is running on my server in the specified data center where I added the waypoint volume to Nomad:
docker ps
IMAGE PORTS
hashicorp/waypoint:latest dc1.server.ip:9701-9702->9701-9702/tcp
And I am able to access the Waypoint UI on port 9702 and I can run the waypoint user token command on my Waypoint server (that is in a different data center) to create an access token.
In addition I can see that Waypoint is using the volume I created on the Nomad minion to store a database file:
ls /opt/waypoint_data
data.db waypoint-restore.db.lock
Initialize the Waypoint Job
Back on the Waypoint server I am still inside the example repository:
ls /opt/waypoint/waypoint-examples/nomad/nodejs
index.js package.json public README.md views waypoint.hcl
Here I have to edit the HCL file since I am not actually use the dc1 data center for my test:
cat waypoint.hcl
project = "nomad-nodejs"
app "nomad-nodejs-web" {
build {
use "pack" {}
registry {
use "docker" {
image = "nomad-nodejs-web"
tag = "1"
local = true
}
}
}
deploy {
use "nomad" {
// these options both default to the values shown, but are left here to
// show they are configurable
datacenter = "dc1"
// namespace = "default"
}
}
}
With these configurations in place, issue the following command in order to initialize Waypoint with this configuration:
waypoint init
✓ Configuration file appears valid
✓ Connection to Waypoint server was successful
✓ Project "nomad-nodejs" and all apps are registered with the server.
Project initialized!
We can now deploy the application to Nomad by running:
waypoint up
The deploy was successful! A Waypoint deployment URL is shown below. This
can be used internally to check your deployment and is not meant for external
traffic. You can manage this hostname using "waypoint hostname."
URL: https://curiously-evolving-muskox.waypoint.run
Deployment URL: https://curiously-evolving-muskox--v1.waypoint.run
Debugging
But I am seeing that the allocation fails in Nomad. The issue seems to be that the Docker image is being build on my Waypoint server but is not pushed to the Nomad minion server:
I do see that the image was build on the Waypoint server:
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nomad-nodejs-web 1 168778d136f0 4 minutes ago 782MB
nomad-nodejs-web latest 168778d136f0 4 minutes ago 782MB
I did open the gRPC port 9702 (see docs). But how is this transfer supposed to happen?
So I pushed the Docker image manually to the minion server, stopped the Nomad allocation and restarted it:
The minion now hosts the application on port 27478:
docker ps
IMAGE PORTS
nomad-nodejs-web:1 dc1.server.ip:27478->3000/tcp
And lo and behold there is the application:
Waypoint Help
Usage: waypoint server install [options]
Alias: waypoint install
Installs a Waypoint server to an existing platform. The platform should be
specified as kubernetes, nomad, ecs, or docker.
This will also install a single Waypoint runner by default. This enables
remote operations out of the box, such as polling a Git repository. This can
be disabled by specifying "-runner=false".
By default, this will also automatically create a new default CLI context
(see "waypoint context") so the CLI will be configured to use the newly
installed server.
This command will require you to accept the Waypoint Terms of Service
and Privacy Policy for the Waypoint URL service by specifying the "-accept-tos"
flag. This only applies to the Waypoint URL service. You may disable the
URL service by manually running the server. If you disable the URL service,
you do not need to accept any terms.
To further customize the server installation, you may pass advanced flag options
specified in the documentation for the 'server run' command. To set these values,
include a '--' after the full argument list for 'install', followed by these
advanced flag options. As an example, to set the server log level to trace
and disable the UI, the command would be:
waypoint install -platform=docker -accept-tos -- -vvv -disable-ui
Global Options:
-app=<string>
App to target. Certain commands require a single app target for Waypoint
configurations with multiple apps. If you have a single app, then this
can be ignored. This is aliased as "-a".
-plain
Plain output: no colors, no animation. The default is false.
-project=<string>
Project to target. This is aliased as "-p".
-workspace=<string>
Workspace to operate in. This is aliased as "-w".
Command Options:
-accept-tos
Pass to accept the Terms of Service and Privacy Policy to use the
Waypoint URL Service. This is required if the URL service is enabled
and you're using the HashiCorp-provided URL service rather than
self-hosting. See the privacy policy at https://hashicorp.com/privacy
and the ToS at https://waypointproject.io/terms The default is false.
-context-create=<string>
Create a context with connection information for this installation. The
default value if not set will be 'install-' and then be suffixed with a
timestamp at the time the command is executed.
-context-set-default
Set the newly installed server as the default CLI context. The default
is true.
-platform=<string>
Platform to install the Waypoint server into.
docker Options:
-docker-odr-image=<string>
Docker image for the Waypoint On-Demand Runners. This will default to
the server image with the name (not label) suffixed with '-odr'.
-docker-server-image=<string>
Docker image for the Waypoint server. The default is
hashicorp/waypoint:latest.
ecs Options:
-ecs-cluster=<string>
Configures the Cluster to install into. The default is waypoint-server.
-ecs-cpu=<string>
Configures the requested CPU amount for the Waypoint server task in ECS.
The default is 512.
-ecs-execution-role-name=<string>
Configures the IAM Execution role name to use. The default is
waypoint-server-execution-role.
-ecs-mem=<string>
Configures the requested memory amount for the Waypoint server task in
ECS. The default is 1024.
-ecs-odr-cpu=<string>
Configures the requested CPU amount for the Waypoint On-Demand runner in
ECS. The default is 512.
-ecs-odr-image=<string>
Docker image for the Waypoint On-Demand Runners. This will default to
the server image with the name (not label) suffixed with '-odr'.
-ecs-odr-mem=<string>
Configures the requested memory amount for the Waypoint On-Demand runner
in ECS. The default is 2048.
-ecs-region=<string>
Configures which AWS region to install into.
-ecs-server-image=<string>
Docker image for the Waypoint server. The default is
hashicorp/waypoint:latest.
-ecs-subnets=<string>
Subnets to install server into.
-ecs-task-role-name=<string>
IAM Execution Role to assign to the on-demand runner. If this is blank,
an IAM role will be created automatically with the default permissions.
The default is waypoint-runner.
kubernetes Options:
-k8s-advertise-internal
Advertise the internal service address rather than the external. This
is useful if all your deployments will be able to access the private
service address. This will default to false but will be automatically
set to true if the external host is detected to be localhost. The
default is false.
-k8s-annotate-service=<key=value>
Annotations for the Service generated.
-k8s-config-path=<string>
Path to the kubeconfig file to use.
-k8s-context=<string>
The Kubernetes context to install the Waypoint server to. If left unset,
Waypoint will use the current Kubernetes context.
-k8s-cpu-limit=<string>
Configures the CPU limit for the Waypoint server in Kubernetes. The
default is 0.
-k8s-cpu-request=<string>
Configures the requested CPU amount for the Waypoint server in
Kubernetes. The default is 0.
-k8s-helm-version=<string>
The version of the Helm chart to use for the Waypoint runner install.
The required version number format is: 'vX.Y.Z'.
-k8s-mem-limit=<string>
Configures the memory limit for the Waypoint server in Kubernetes. The
default is 0.
-k8s-mem-request=<string>
Configures the requested memory amount for the Waypoint server in
Kubernetes. The default is 0.
-k8s-namespace=<string>
Namespace to install the Waypoint server into for Kubernetes.
-k8s-odr-image=<string>
Docker image for the Waypoint On-Demand Runners
-k8s-pull-policy=<string>
Set the pull policy for the Waypoint server image.
-k8s-pull-secret=<string>
Secret to use to access the Waypoint server image on Kubernetes.
-k8s-runner-service-account=<string>
Service account to assign to the on-demand runner. If this is blank,
a service account will be created automatically with the correct
permissions. The default is waypoint-runner.
-k8s-runner-service-account-init
Create the service account if it does not exist. The default is true.
-k8s-secret-file=<string>
Use the Kubernetes Secret in the given path to access the Waypoint
server image.
-k8s-server-image=<string>
Docker image for the Waypoint server. The default is
hashicorp/waypoint:latest.
-k8s-storage-request=<string>
Configures the requested persistent volume size for the Waypoint server
in Kubernetes. The default is 1Gi.
-k8s-storageclassname=<string>
Name of the StorageClass required by the volume claim to install the
Waypoint server image to.
nomad Options:
-nomad-annotate-service=<key=value>
Annotations for the Service generated.
-nomad-auth-soft-fail
Don't fail the Nomad task on an auth failure obtaining server image
container. Attempt to continue without auth. The default is false.
-nomad-consul-datacenter=<string>
The datacenter where Consul is located. The default is dc1.
-nomad-consul-domain=<string>
The domain where Consul is located. The default is consul.
-nomad-consul-service
Create service for Waypoint UI and Server in Consul. The default is
true.
-nomad-consul-service-backend-tags=<string>
Tags for the Waypoint backend service generated in Consul. The 'first'
tag will be used when crafting the Consul DNS hostname for accessing
Waypoint. The default is waypoint.
-nomad-consul-service-hostname=<string>
If set, will use this hostname for Consul DNS rather than the default,
i.e. "waypoint-server.service.consul".
-nomad-consul-service-ui-tags=<string>
Tags for the Waypoint UI service generated in Consul. The default is
waypoint.
-nomad-consul-token=<string>
If set, the passed Consul token is stored in the job before sending to
the Nomad servers. Overrides the CONSUL_HTTP_TOKEN environment variable
if set.
-nomad-csi-external-id=<string>
The ID of the physical volume from the Nomad storage provider.
-nomad-csi-fs=<string>
Nomad CSI volume mount option file system. The default is xfs.
-nomad-csi-parameters=<key=value>
Parameters passed directly to the CSI plugin to configure the volume.
-nomad-csi-plugin-id=<string>
The ID of the CSI plugin that manages the volume, required for volume
type 'csi'.
-nomad-csi-secrets=<key=value>
Secrets to provide for the CSI volume.
-nomad-csi-topologies=<key=value>
Locations from which the Nomad Volume will be accessible.
-nomad-csi-volume-capacity-max=<int>
Nomad CSI volume capacity maximum, in bytes. The default is 2147483648.
-nomad-csi-volume-capacity-min=<int>
Nomad CSI volume capacity minimum, in bytes. The default is 1073741824.
-nomad-csi-volume-provider=<string>
Nomad CSI volume provider, required for volume type 'csi'.
-nomad-dc=<string>
Datacenters to install to for Nomad. The default is dc1.
-nomad-host=<string>
Hostname of the Nomad server to use, like for launching on-demand tasks.
The default is http://localhost:4646.
-nomad-host-volume=<string>
Nomad host volume name to use for the Waypoint server, required for
volume type 'host'.
-nomad-namespace=<string>
Namespace to install the Waypoint server into for Nomad. The default is
default.
-nomad-odr-image=<string>
Docker image for the on-demand runners. If not specified, it defaults to
the server image name + '-odr' (i.e. 'hashicorp/waypoint-odr:latest')
-nomad-policy-override
Override the Nomad sentinel policy for enterprise Nomad. The default is
false.
-nomad-region=<string>
Region to install to for Nomad. The default is global.
-nomad-runner-cpu=<string>
CPU required to run this task in MHz. The default is 200.
-nomad-runner-csi-volume-capacity-max=<int>
Waypoint runner Nomad CSI volume capacity maximum, in bytes. The default
is 2147483648.
-nomad-runner-csi-volume-capacity-min=<int>
Waypoint runner Nomad CSI volume capacity minimum, in bytes. The default
is 1073741824.
-nomad-runner-csi-volume-provider=<string>
Name of the CSI volume provider to use for the Waypoint runner.
-nomad-runner-host-volume=<string>
Name of the host volume to use for the Waypoint runner.
-nomad-runner-memory=<string>
MB of Memory to allocate to the runner job task. The default is 600.
-nomad-server-cpu=<string>
CPU required to run this task in MHz. The default is 200.
-nomad-server-image=<string>
Docker image for the Waypoint server. The default is
hashicorp/waypoint:latest.
-nomad-server-memory=<string>
MB of Memory to allocate to the Server job task. The default is 600.