Skip to main content

Hashicorp Nomad Sidecar Pattern

Shen Zhen, China

Based on the Github Repository from @lastlegion. But I found a couple of deprecation issues that needed to be ironed out.

SSL Proxy as a Sidecar

The sidecar pattern is made up of 2 containers:

  • application container
  • sidecar container

Sidecar services augment other application and are scheduled on the Same Host using Nomad’s Group Stanza. The application build consists of the Hashicorp HTTP Echo Server - standing in as a template Webfrontend - and an NGINX Sidecar Proxy. The HTTP server is configured to return a simple Hello World string and listening on port 8080 - a port that is not accessible outside from localhost:

config {
        image = "hashicorp/http-echo:latest"
        ports = [
          "http",
        ]
        args  = [
          "-listen", ":${NOMAD_PORT_http}",
          "-text", "Hello World!",
        ]
      }

The NGINX reverse-proxy can be configured using Nomad's Template Stanza to listen on a different port that is accessible from the internet and forward all incoming traffic to our HTTP server:

template {
        data = <<EOF
          worker_processes  1;
          events {
              worker_connections  1024;
          }
          http {
              include       mime.types;
              default_type  application/octet-stream;
              sendfile        on;
              keepalive_timeout  65;
              upstream backend {
                {{ range service "app-server" }}
                server {{ .Address }}:{{ .Port }};
                {{ else }}server 127.0.0.1:65535; # force a 502
                {{ end }}
              }
              server {
                listen       8015 ssl;
                server_name  localhost;
                ssl_certificate      /secrets/certificate.crt;
                ssl_certificate_key  /secrets/certificate.key;
                location / {
                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                  proxy_set_header Host $http_host;
                  proxy_pass http://backend;
                }
              }
          }
        EOF
        destination = "config/nginx.conf"
      }
      template {
        # Warning: Fetch certificate from a secret store like vault in a production setting
        data = <<EOF
-----BEGIN CERTIFICATE-----
MIIDajCCAlKgAwIBAgITPAgZrdQO2jRhM/KkR8Czc1iC+jANBgkqhkiG9w0BAQsF
...
jlw8JveeVvRFTvdwW23+JHnTvqpremm5PhmvPPnRNOhR1WhKxvJWWdeYDBjBoaPo
vAIDwNat6MpbCNH6pSs=
-----END CERTIFICATE-----
        EOF
        destination = "secrets/certificate.crt"
      }
      template {
        # Warning: Fetch certificate from a secret store like vault in a production setting
        data = <<EOF
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDtReLnMCGfM6jI
...
EFG7bYgACbS289cpWA2nmcfhhGn7tV26xLuMFma0GQhr6jpQbuqONbrnlyRJGvMr
7iyk7nrNu9Ez3yM9zVMuvIJH
-----END PRIVATE KEY-----
        EOF
        destination = "secrets/certificate.key"
      }
    }
  }
}

Here we are using a set of self-signed TLS certificates to accept HTTPS traffic on port 8015 (I am using a none default port to de-conflict with another service running on this host), takes care of the TLS termination and forwards the request to our application container using a Nomad variables {{ range service "app-server" }}.

Run the Job

nomad plan tls_sidecar.tf
+ Job: "ssl-proxy-example"
+ Task Group: "ssl-proxy" (1 create)
  + Task: "app-server" (forces create)
  + Task: "ssl-proxy-sidecar" (forces create)

Scheduler dry-run:
- All tasks successfully allocated.

Job Modify Index: 0
To submit the job with version verification run:

nomad job run -check-index 0 tls_sidecar.tf

After running the job with the command above I can see the successful service registration in Consul. Here I can already see that the HTTP request results in the Hello World response - so this is the Consul client directly interacting with the app container through the port that is only accessible on localhost:

Hashicorp Nomad Sidecar Pattern

Switching to Nomad I can see that both services - the app and my proxy sidecar - was successfully deployed:

Hashicorp Nomad Sidecar Pattern

And finally testing the HTTPS connection - everything works as expected! My request is received by the NGINX proxy, TLS is terminated and the request then forwarded to the app. Sweet!

curl --insecure https://myserverip:8015
Hello World!

Complete Jobfile

job "ssl-proxy" {
  datacenters=["mydc"]
  group "ssl-proxy" {
        
    network {
        port "http" {
            static = 8080
        }
        port "https" {
            static = 8015
        }
    }

    service {
      name = "app-server"
      port = "http"

      check {
        type     = "http"
        port     = "http"
        path     = "/"
        interval = "2s"
        timeout  = "2s"
      }
    }

    task "app-server" {
      driver = "docker"

      config {
        image = "hashicorp/http-echo:latest"
        ports = [
          "http",
        ]
        args  = [
          "-listen", ":${NOMAD_PORT_http}",
          "-text", "Hello World!",
        ]
      }
    }

    task "ssl-proxy-sidecar" {
      driver = "docker"
      config {
        image = "nginx"
        ports = [
          "https",
        ]
        volumes = [
          "config/nginx.conf:/etc/nginx/nginx.conf",
          "secrets/certificate.crt:/secrets/certificate.crt",
          "secrets/certificate.key:/secrets/certificate.key",
        ]
      }
      
      template {
        data = <<EOF
          worker_processes  1;
          events {
              worker_connections  1024;
          }
          http {
              include       mime.types;
              default_type  application/octet-stream;
              sendfile        on;
              keepalive_timeout  65;
              upstream backend {
                {{ range service "app-server" }}
                server {{ .Address }}:{{ .Port }};
                {{ else }}server 127.0.0.1:65535; # force a 502
                {{ end }}
              }
              server {
                listen       8015 ssl;
                server_name  localhost;
                ssl_certificate      /secrets/certificate.crt;
                ssl_certificate_key  /secrets/certificate.key;
                location / {
                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                  proxy_set_header Host $http_host;
                  proxy_pass http://backend;
                }
              }
          }
        EOF
        destination = "config/nginx.conf"
      }
      template {
        # Warning: Fetch certificate from a secret store like vault in a production setting
        data = <<EOF
-----BEGIN CERTIFICATE-----
MIIDajCCAlKgAwIBAgITPAgZrdQO2jRhM/KkR8Czc1iC+jANBgkqhkiG9w0BAQsF
...
jlw8JveeVvRFTvdwW23+JHnTvqpremm5PhmvPPnRNOhR1WhKxvJWWdeYDBjBoaPo
vAIDwNat6MpbCNH6pSs=
-----END CERTIFICATE-----
        EOF
        destination = "secrets/certificate.crt"
      }
      template {
        # Warning: Fetch certificate from a secret store like vault in a production setting
        data = <<EOF
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDtReLnMCGfM6jI
...
EFG7bYgACbS289cpWA2nmcfhhGn7tV26xLuMFma0GQhr6jpQbuqONbrnlyRJGvMr
7iyk7nrNu9Ez3yM9zVMuvIJH
-----END PRIVATE KEY-----
        EOF
        destination = "secrets/certificate.key"
      }
    }
  }
}