Hashicorp Nomad Sidecar Pattern
Based on the Github Repository from @lastlegion. But I found a couple of deprecation issues that needed to be ironed out.
SSL Proxy as a Sidecar
The sidecar pattern is made up of 2 containers:
- application container
- sidecar container
Sidecar services augment other application and are scheduled on the Same Host using Nomad’s Group Stanza. The application build consists of the Hashicorp HTTP Echo Server - standing in as a template Webfrontend - and an NGINX Sidecar Proxy. The HTTP server is configured to return a simple Hello World string and listening on port 8080 - a port that is not accessible outside from localhost:
config {
image = "hashicorp/http-echo:latest"
ports = [
"http",
]
args = [
"-listen", ":${NOMAD_PORT_http}",
"-text", "Hello World!",
]
}
The NGINX reverse-proxy can be configured using Nomad's Template Stanza to listen on a different port that is accessible from the internet and forward all incoming traffic to our HTTP server:
template {
data = <<EOF
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
upstream backend {
{{ range service "app-server" }}
server {{ .Address }}:{{ .Port }};
{{ else }}server 127.0.0.1:65535; # force a 502
{{ end }}
}
server {
listen 8015 ssl;
server_name localhost;
ssl_certificate /secrets/certificate.crt;
ssl_certificate_key /secrets/certificate.key;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_pass http://backend;
}
}
}
EOF
destination = "config/nginx.conf"
}
template {
# Warning: Fetch certificate from a secret store like vault in a production setting
data = <<EOF
-----BEGIN CERTIFICATE-----
MIIDajCCAlKgAwIBAgITPAgZrdQO2jRhM/KkR8Czc1iC+jANBgkqhkiG9w0BAQsF
...
jlw8JveeVvRFTvdwW23+JHnTvqpremm5PhmvPPnRNOhR1WhKxvJWWdeYDBjBoaPo
vAIDwNat6MpbCNH6pSs=
-----END CERTIFICATE-----
EOF
destination = "secrets/certificate.crt"
}
template {
# Warning: Fetch certificate from a secret store like vault in a production setting
data = <<EOF
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDtReLnMCGfM6jI
...
EFG7bYgACbS289cpWA2nmcfhhGn7tV26xLuMFma0GQhr6jpQbuqONbrnlyRJGvMr
7iyk7nrNu9Ez3yM9zVMuvIJH
-----END PRIVATE KEY-----
EOF
destination = "secrets/certificate.key"
}
}
}
}
Here we are using a set of self-signed TLS certificates to accept HTTPS traffic on port 8015 (I am using a none default port to de-conflict with another service running on this host), takes care of the TLS termination and forwards the request to our application container using a Nomad variables {{ range service "app-server" }}.
Run the Job
nomad plan tls_sidecar.tf
+ Job: "ssl-proxy-example"
+ Task Group: "ssl-proxy" (1 create)
+ Task: "app-server" (forces create)
+ Task: "ssl-proxy-sidecar" (forces create)
Scheduler dry-run:
- All tasks successfully allocated.
Job Modify Index: 0
To submit the job with version verification run:
nomad job run -check-index 0 tls_sidecar.tf
After running the job with the command above I can see the successful service registration in Consul. Here I can already see that the HTTP request results in the Hello World response - so this is the Consul client directly interacting with the app container through the port that is only accessible on localhost:
Switching to Nomad I can see that both services - the app and my proxy sidecar - was successfully deployed:
And finally testing the HTTPS connection - everything works as expected! My request is received by the NGINX proxy, TLS is terminated and the request then forwarded to the app. Sweet!
curl --insecure https://myserverip:8015
Hello World!
Complete Jobfile
job "ssl-proxy" {
datacenters=["mydc"]
group "ssl-proxy" {
network {
port "http" {
static = 8080
}
port "https" {
static = 8015
}
}
service {
name = "app-server"
port = "http"
check {
type = "http"
port = "http"
path = "/"
interval = "2s"
timeout = "2s"
}
}
task "app-server" {
driver = "docker"
config {
image = "hashicorp/http-echo:latest"
ports = [
"http",
]
args = [
"-listen", ":${NOMAD_PORT_http}",
"-text", "Hello World!",
]
}
}
task "ssl-proxy-sidecar" {
driver = "docker"
config {
image = "nginx"
ports = [
"https",
]
volumes = [
"config/nginx.conf:/etc/nginx/nginx.conf",
"secrets/certificate.crt:/secrets/certificate.crt",
"secrets/certificate.key:/secrets/certificate.key",
]
}
template {
data = <<EOF
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
upstream backend {
{{ range service "app-server" }}
server {{ .Address }}:{{ .Port }};
{{ else }}server 127.0.0.1:65535; # force a 502
{{ end }}
}
server {
listen 8015 ssl;
server_name localhost;
ssl_certificate /secrets/certificate.crt;
ssl_certificate_key /secrets/certificate.key;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_pass http://backend;
}
}
}
EOF
destination = "config/nginx.conf"
}
template {
# Warning: Fetch certificate from a secret store like vault in a production setting
data = <<EOF
-----BEGIN CERTIFICATE-----
MIIDajCCAlKgAwIBAgITPAgZrdQO2jRhM/KkR8Czc1iC+jANBgkqhkiG9w0BAQsF
...
jlw8JveeVvRFTvdwW23+JHnTvqpremm5PhmvPPnRNOhR1WhKxvJWWdeYDBjBoaPo
vAIDwNat6MpbCNH6pSs=
-----END CERTIFICATE-----
EOF
destination = "secrets/certificate.crt"
}
template {
# Warning: Fetch certificate from a secret store like vault in a production setting
data = <<EOF
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDtReLnMCGfM6jI
...
EFG7bYgACbS289cpWA2nmcfhhGn7tV26xLuMFma0GQhr6jpQbuqONbrnlyRJGvMr
7iyk7nrNu9Ez3yM9zVMuvIJH
-----END PRIVATE KEY-----
EOF
destination = "secrets/certificate.key"
}
}
}
}