Skip to main content

Hashicorp Waypoint with Nomad

Shen Zhen, China

Let's use Waypoint to deploy an application into a Nomad Cluster.

Installation

Get Waypoint by downloading the latest a pre-compiled waypoint binary:

wget https://releases.hashicorp.com/waypoint/0.9.0/waypoint_0.9.0_linux_amd64.zip
unzip waypoint_0.9.0_linux_amd64.zip && rm waypoint_0.9.0_linux_amd64.zip
mv ./waypoint /usr/bin/waypoint
chmod +x /usr/bin/waypoint

Verify the installation:

waypoint -version
CLI: v0.9.0 (2a7b89036)

Use Waypoint with Nomad

I am following the getting started guide by Hashicorp to set up an NodeJS app inside a Docker container using waypoint. The Example code can be found on Github. Start by cloning the Waypoint Examples repository and navigate to the NodeJS example:

git clone https://github.com/hashicorp/waypoint-examples.git
cd waypoint-examples/nomad/nodejs

Persistence

To be able to persist data we first need to add a host volume to our Nomad minion. Add the following configs in your client.hcl file:

client {
  enabled = true
  servers = ["myhost:port"]
  host_volume "waypoint" {
    path = "/opt/waypoint_data"
    read_only = false
  }
}

# Docker Configuration
plugin "docker" {
    volumes {
      enabled = true
    }
}

Create the /opt/waypoint_data directory and I am not sure about the permissions needed here. The documentation says that you have to chown <Nomad User>:

chown nomad:nomad /opt/waypoint_data/
chmod 775 /opt/waypoint_data/

Set up the Waypoint Server

To install Waypoint with host volume persistent storage into the default data center dc1, run:

waypoint install -platform=nomad -accept-tos -nomad-host=https://localhost:4646 -nomad-dc=dc1 -nomad-host-volume=waypoint -nomad-runner-host-volume=waypoint -nomad-consul-service=true -nomad-consul-datacenter=dc1

Error installing the runner: please include '-nomad-csi-volume-provider' or '-nomad-host-volume' Github Issue

You can deactivate the Consul integration if the service is not present on your system.

Hashicorp Waypoint

Hashicorp Waypoint

The docker container is running on my server in the specified data center where I added the waypoint volume to Nomad:

docker ps
IMAGE                      PORTS
hashicorp/waypoint:latest  dc1.server.ip:9701-9702->9701-9702/tcp

And I am able to access the Waypoint UI on port 9702 and I can run the waypoint user token command on my Waypoint server (that is in a different data center) to create an access token.

In addition I can see that Waypoint is using the volume I created on the Nomad minion to store a database file:

ls /opt/waypoint_data
data.db  waypoint-restore.db.lock

Initialize the Waypoint Job

Back on the Waypoint server I am still inside the example repository:

ls /opt/waypoint/waypoint-examples/nomad/nodejs
index.js  package.json  public  README.md  views  waypoint.hcl

Here I have to edit the HCL file since I am not actually use the dc1 data center for my test:

cat waypoint.hcl

project = "nomad-nodejs"

app "nomad-nodejs-web" {

  build {
    use "pack" {}
    registry {
      use "docker" {
        image = "nomad-nodejs-web"
        tag   = "1"
        local = true
      }
    }
  }

  deploy {
    use "nomad" {
      // these options both default to the values shown, but are left here to
      // show they are configurable
      datacenter = "dc1"
      // namespace  = "default"
    }
  }
}

With these configurations in place, issue the following command in order to initialize Waypoint with this configuration:

waypoint init

✓ Configuration file appears valid
✓ Connection to Waypoint server was successful
✓ Project "nomad-nodejs" and all apps are registered with the server.

Project initialized!

We can now deploy the application to Nomad by running:

waypoint up

Hashicorp Waypoint

The deploy was successful! A Waypoint deployment URL is shown below. This
can be used internally to check your deployment and is not meant for external
traffic. You can manage this hostname using "waypoint hostname."

           URL: https://curiously-evolving-muskox.waypoint.run
Deployment URL: https://curiously-evolving-muskox--v1.waypoint.run

Debugging

But I am seeing that the allocation fails in Nomad. The issue seems to be that the Docker image is being build on my Waypoint server but is not pushed to the Nomad minion server:

Hashicorp Waypoint

I do see that the image was build on the Waypoint server:

docker images
REPOSITORY           TAG      IMAGE ID       CREATED         SIZE
nomad-nodejs-web     1        168778d136f0   4 minutes ago   782MB
nomad-nodejs-web     latest   168778d136f0   4 minutes ago   782MB

I did open the gRPC port 9702 (see docs). But how is this transfer supposed to happen?

So I pushed the Docker image manually to the minion server, stopped the Nomad allocation and restarted it:

Hashicorp Waypoint

Hashicorp Waypoint

Hashicorp Waypoint

The minion now hosts the application on port 27478:

docker ps
IMAGE              PORTS
nomad-nodejs-web:1 dc1.server.ip:27478->3000/tcp

And lo and behold there is the application:

Hashicorp Waypoint

Waypoint Help

Usage: waypoint server install [options]
Alias: waypoint install

  Installs a Waypoint server to an existing platform. The platform should be
  specified as kubernetes, nomad, ecs, or docker.

  This will also install a single Waypoint runner by default. This enables
  remote operations out of the box, such as polling a Git repository. This can
  be disabled by specifying "-runner=false".

  By default, this will also automatically create a new default CLI context
  (see "waypoint context") so the CLI will be configured to use the newly
  installed server.

  This command will require you to accept the Waypoint Terms of Service
  and Privacy Policy for the Waypoint URL service by specifying the "-accept-tos"
  flag. This only applies to the Waypoint URL service. You may disable the
  URL service by manually running the server. If you disable the URL service,
  you do not need to accept any terms.

  To further customize the server installation, you may pass advanced flag options
  specified in the documentation for the 'server run' command. To set these values,
  include a '--' after the full argument list for 'install', followed by these
  advanced flag options. As an example, to set the server log level to trace
  and disable the UI, the command would be:

    waypoint install -platform=docker -accept-tos -- -vvv -disable-ui

Global Options:

  -app=<string>
      App to target. Certain commands require a single app target for Waypoint
      configurations with multiple apps. If you have a single app, then this
      can be ignored. This is aliased as "-a".

  -plain
      Plain output: no colors, no animation. The default is false.

  -project=<string>
      Project to target. This is aliased as "-p".

  -workspace=<string>
      Workspace to operate in. This is aliased as "-w".

Command Options:

  -accept-tos
      Pass to accept the Terms of Service and Privacy Policy to use the
      Waypoint URL Service. This is required if the URL service is enabled
      and you're using the HashiCorp-provided URL service rather than
      self-hosting. See the privacy policy at https://hashicorp.com/privacy
      and the ToS at https://waypointproject.io/terms The default is false.

  -context-create=<string>
      Create a context with connection information for this installation. The
      default value if not set will be 'install-' and then be suffixed with a
      timestamp at the time the command is executed.

  -context-set-default
      Set the newly installed server as the default CLI context. The default
      is true.

  -platform=<string>
      Platform to install the Waypoint server into.

docker Options:

  -docker-odr-image=<string>
      Docker image for the Waypoint On-Demand Runners. This will default to
      the server image with the name (not label) suffixed with '-odr'.

  -docker-server-image=<string>
      Docker image for the Waypoint server. The default is
      hashicorp/waypoint:latest.

ecs Options:

  -ecs-cluster=<string>
      Configures the Cluster to install into. The default is waypoint-server.

  -ecs-cpu=<string>
      Configures the requested CPU amount for the Waypoint server task in ECS.
      The default is 512.

  -ecs-execution-role-name=<string>
      Configures the IAM Execution role name to use. The default is
      waypoint-server-execution-role.

  -ecs-mem=<string>
      Configures the requested memory amount for the Waypoint server task in
      ECS. The default is 1024.

  -ecs-odr-cpu=<string>
      Configures the requested CPU amount for the Waypoint On-Demand runner in
      ECS. The default is 512.

  -ecs-odr-image=<string>
      Docker image for the Waypoint On-Demand Runners. This will default to
      the server image with the name (not label) suffixed with '-odr'.

  -ecs-odr-mem=<string>
      Configures the requested memory amount for the Waypoint On-Demand runner
      in ECS. The default is 2048.

  -ecs-region=<string>
      Configures which AWS region to install into.

  -ecs-server-image=<string>
      Docker image for the Waypoint server. The default is
      hashicorp/waypoint:latest.

  -ecs-subnets=<string>
      Subnets to install server into.

  -ecs-task-role-name=<string>
      IAM Execution Role to assign to the on-demand runner. If this is blank,
      an IAM role will be created automatically with the default permissions.
      The default is waypoint-runner.

kubernetes Options:

  -k8s-advertise-internal
      Advertise the internal service address rather than the external. This
      is useful if all your deployments will be able to access the private
      service address. This will default to false but will be automatically
      set to true if the external host is detected to be localhost. The
      default is false.

  -k8s-annotate-service=<key=value>
      Annotations for the Service generated.

  -k8s-config-path=<string>
      Path to the kubeconfig file to use.

  -k8s-context=<string>
      The Kubernetes context to install the Waypoint server to. If left unset,
      Waypoint will use the current Kubernetes context.

  -k8s-cpu-limit=<string>
      Configures the CPU limit for the Waypoint server in Kubernetes. The
      default is 0.

  -k8s-cpu-request=<string>
      Configures the requested CPU amount for the Waypoint server in
      Kubernetes. The default is 0.

  -k8s-helm-version=<string>
      The version of the Helm chart to use for the Waypoint runner install.
      The required version number format is: 'vX.Y.Z'.

  -k8s-mem-limit=<string>
      Configures the memory limit for the Waypoint server in Kubernetes. The
      default is 0.

  -k8s-mem-request=<string>
      Configures the requested memory amount for the Waypoint server in
      Kubernetes. The default is 0.

  -k8s-namespace=<string>
      Namespace to install the Waypoint server into for Kubernetes.

  -k8s-odr-image=<string>
      Docker image for the Waypoint On-Demand Runners

  -k8s-pull-policy=<string>
      Set the pull policy for the Waypoint server image.

  -k8s-pull-secret=<string>
      Secret to use to access the Waypoint server image on Kubernetes.

  -k8s-runner-service-account=<string>
      Service account to assign to the on-demand runner. If this is blank,
      a service account will be created automatically with the correct
      permissions. The default is waypoint-runner.

  -k8s-runner-service-account-init
      Create the service account if it does not exist. The default is true.

  -k8s-secret-file=<string>
      Use the Kubernetes Secret in the given path to access the Waypoint
      server image.

  -k8s-server-image=<string>
      Docker image for the Waypoint server. The default is
      hashicorp/waypoint:latest.

  -k8s-storage-request=<string>
      Configures the requested persistent volume size for the Waypoint server
      in Kubernetes. The default is 1Gi.

  -k8s-storageclassname=<string>
      Name of the StorageClass required by the volume claim to install the
      Waypoint server image to.

nomad Options:

  -nomad-annotate-service=<key=value>
      Annotations for the Service generated.

  -nomad-auth-soft-fail
      Don't fail the Nomad task on an auth failure obtaining server image
      container. Attempt to continue without auth. The default is false.

  -nomad-consul-datacenter=<string>
      The datacenter where Consul is located. The default is dc1.

  -nomad-consul-domain=<string>
      The domain where Consul is located. The default is consul.

  -nomad-consul-service
      Create service for Waypoint UI and Server in Consul. The default is
      true.

  -nomad-consul-service-backend-tags=<string>
      Tags for the Waypoint backend service generated in Consul. The 'first'
      tag will be used when crafting the Consul DNS hostname for accessing
      Waypoint. The default is waypoint.

  -nomad-consul-service-hostname=<string>
      If set, will use this hostname for Consul DNS rather than the default,
      i.e. "waypoint-server.service.consul".

  -nomad-consul-service-ui-tags=<string>
      Tags for the Waypoint UI service generated in Consul. The default is
      waypoint.

  -nomad-consul-token=<string>
      If set, the passed Consul token is stored in the job before sending to
      the Nomad servers. Overrides the CONSUL_HTTP_TOKEN environment variable
      if set.

  -nomad-csi-external-id=<string>
      The ID of the physical volume from the Nomad storage provider.

  -nomad-csi-fs=<string>
      Nomad CSI volume mount option file system. The default is xfs.

  -nomad-csi-parameters=<key=value>
      Parameters passed directly to the CSI plugin to configure the volume.

  -nomad-csi-plugin-id=<string>
      The ID of the CSI plugin that manages the volume, required for volume
      type 'csi'.

  -nomad-csi-secrets=<key=value>
      Secrets to provide for the CSI volume.

  -nomad-csi-topologies=<key=value>
      Locations from which the Nomad Volume will be accessible.

  -nomad-csi-volume-capacity-max=<int>
      Nomad CSI volume capacity maximum, in bytes. The default is 2147483648.

  -nomad-csi-volume-capacity-min=<int>
      Nomad CSI volume capacity minimum, in bytes. The default is 1073741824.

  -nomad-csi-volume-provider=<string>
      Nomad CSI volume provider, required for volume type 'csi'.

  -nomad-dc=<string>
      Datacenters to install to for Nomad. The default is dc1.

  -nomad-host=<string>
      Hostname of the Nomad server to use, like for launching on-demand tasks.
      The default is http://localhost:4646.

  -nomad-host-volume=<string>
      Nomad host volume name to use for the Waypoint server, required for
      volume type 'host'.

  -nomad-namespace=<string>
      Namespace to install the Waypoint server into for Nomad. The default is
      default.

  -nomad-odr-image=<string>
      Docker image for the on-demand runners. If not specified, it defaults to
      the server image name + '-odr' (i.e. 'hashicorp/waypoint-odr:latest')

  -nomad-policy-override
      Override the Nomad sentinel policy for enterprise Nomad. The default is
      false.

  -nomad-region=<string>
      Region to install to for Nomad. The default is global.

  -nomad-runner-cpu=<string>
      CPU required to run this task in MHz. The default is 200.

  -nomad-runner-csi-volume-capacity-max=<int>
      Waypoint runner Nomad CSI volume capacity maximum, in bytes. The default
      is 2147483648.

  -nomad-runner-csi-volume-capacity-min=<int>
      Waypoint runner Nomad CSI volume capacity minimum, in bytes. The default
      is 1073741824.

  -nomad-runner-csi-volume-provider=<string>
      Name of the CSI volume provider to use for the Waypoint runner.

  -nomad-runner-host-volume=<string>
      Name of the host volume to use for the Waypoint runner.

  -nomad-runner-memory=<string>
      MB of Memory to allocate to the runner job task. The default is 600.

  -nomad-server-cpu=<string>
      CPU required to run this task in MHz. The default is 200.

  -nomad-server-image=<string>
      Docker image for the Waypoint server. The default is
      hashicorp/waypoint:latest.

  -nomad-server-memory=<string>
      MB of Memory to allocate to the Server job task. The default is 600.