Hashicorp Nomad for NGINX Web Proxies
- Configure NGINX Reverse Proxy for the Nomad / Consul Web UI
- Create a Nomad Job to set up the NGINX Proxy
NGINX can be used to reverse proxy web services and balance load across multiple instances of the same service. A reverse proxy has the added benefits of enabling multiple web services to share a single, memorable domain and authentication to view internal systems.
Configure NGINX Reverse Proxy for the Nomad / Consul Web UI
To ensure every feature in the Nomad UI remains fully functional, you must properly configure your reverse proxy to meet Nomad's specific networking requirements.
Create a basic NGINX configuration file to reverse proxy the Web UI. It is important to name the NGINX configuration file /opt/ingress/nginx.conf otherwise the file will not bind correctly:
# /opt/ingress/nginx.conf
events {}
http {
# Since WebSockets are stateful connections but Nomad has multiple
# server nodes, an upstream with ip_hash declared is required to ensure
# that connections are always proxied to the same server node when possible.
upstream nomad {
ip_hash;
server localhost:4646;
}
upstream consul {
server localhost:8501;
}
server {
listen 8080;
server_name 0.0.0.0;
location / {
proxy_pass https://nomad;
proxy_ssl_server_name on;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# Nomad blocking queries will remain open for a default of 5 minutes.
# Increase the proxy timeout to accommodate this timeout with an
# additional grace period.
proxy_read_timeout 310s;
# Nomad log streaming uses streaming HTTP requests. In order to
# synchronously stream logs from Nomad to NGINX to the browser
# proxy buffering needs to be turned off.
proxy_buffering off;
# The Upgrade and Connection headers are used to establish
# a WebSockets connection.
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
# The default Origin header will be the proxy address, which
# will be rejected by Nomad. It must be rewritten to be the
# host address instead.
proxy_set_header Origin "${scheme}://${proxy_host}";
}
}
server {
listen 8081;
server_name 0.0.0.0;
location / {
proxy_pass https://consul;
proxy_ssl_server_name on;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_read_timeout 310s;
proxy_buffering off;
}
}
}
Note that this takes the HTTPS connection from the Nomad (and Consul) and forwards it using HTTP. You basically downgrade attacking yourself. TODO need to add TLS to NGINX!
Security
Restricting Access with HTTP Basic Authentication
You can restrict access to your website or some parts of it by implementing a username/password authentication. Usernames and passwords are taken from a file created and populated by a password file creation tool, for example such as apt install apache2-utils (Debian, Ubuntu) or yum install httpd-tools (RHEL/CentOS/Oracle Linux).
Run the htpasswd utility with the -c flag (to create a new file), the file pathname as the first argument, and the username as the second argument:
htpasswd -c /opt/ingress/.htpasswd myuser
We can limit access to the whole website with basic authentication by adding auth_basic to the server block but still make some website areas public by specifying auth_basic off; in specific location blocks. Or just add auth_basic to every location block you want to lock down:
# /opt/ingress/nginx.conf
events {}
http {
upstream nomad {
ip_hash;
server localhost:4646;
}
upstream consul {
server localhost:8501;
}
server {
listen 8080;
server_name 0.0.0.0;
auth_basic "Administrator’s Area";
auth_basic_user_file /etc/nginx/.htpasswd;
location / {
proxy_pass https://nomad;
proxy_ssl_server_name on;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_read_timeout 310s;
proxy_buffering off;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Origin "${scheme}://${proxy_host}";
}
}
server {
listen 8081;
server_name 0.0.0.0;
auth_basic "Administrator’s Area";
auth_basic_user_file /etc/nginx/.htpasswd;
location / {
proxy_pass https://consul;
proxy_ssl_server_name on;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_read_timeout 310s;
proxy_buffering off;
}
}
}
Test Run
Next in a new terminal session, start NGINX in Docker using this configuration file:
docker run --name ingress \
--rm \
--network host \
--mount type=bind,source=/opt/ingress/nginx.conf,target=/etc/nginx/nginx.conf \
--mount type=bind,source=/opt/ingress/.htpasswd,target=/etc/nginx/.htpasswd \
nginx:1.23.0-alpine
The Nomad UI is now being proxied on your server IP with port 8080 and Consul's UI on port 8081 - use your user login to access them:
Create a Nomad Job to set up the NGINX Proxy
I thought that I could now use Nomad to deploy the web proxy on the Nomad Master server. But apparently this is not the way... (why can't you reach out to master servers?). Anyway, as a proof of concept I will now spawn two web services on one of the minion servers on ports 44555 and 55444 and use Nomad to provide the web proxy for those services:
/etc/nomad.d/jobs/nginx_ingress.nomad
locals {
ports = [
{
port_label = "nomad"
port = 8080
},
{
port_label = "consul"
port = 8081
}
]
}
job "nginx" {
datacenters = ["dc1"]
group "nginx" {
count = 1
network {
mode = "host"
dynamic "port" {
for_each = local.ports
labels = [port.value.port_label]
content {
to = port.value.port
}
}
}
service {
name = "nginx"
}
task "nginx" {
driver = "docker"
config {
network_mode = "host"
image = "nginx:1.23.0-alpine"
ports = ["nomad","consul"]
volumes = [
"local/conf/nginx.conf:/etc/nginx/nginx.conf",
"local/conf/.htpasswd:/etc/nginx/conf.d/.htpasswd",
]
}
template {
data = <<EOF
events {}
http {
upstream nomad {
ip_hash;
server localhost:55444;
}
upstream consul {
server localhost:44555;
}
server {
listen 8080;
server_name 0.0.0.0;
auth_basic "Administrator’s Area";
auth_basic_user_file /etc/nginx/conf.d/.htpasswd;
location / {
proxy_pass http://nomad;
proxy_ssl_server_name on;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_read_timeout 310s;
proxy_buffering off;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Origin "${scheme}://${proxy_host}";
}
}
server {
listen 8081;
server_name 0.0.0.0;
auth_basic "Administrator’s Area";
auth_basic_user_file /etc/nginx/conf.d/.htpasswd;
location / {
proxy_pass http://consul;
proxy_ssl_server_name on;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_read_timeout 310s;
proxy_buffering off;
}
}
}
EOF
destination = "local/conf/nginx.conf"
change_mode = "signal"
change_signal = "SIGHUP"
}
template {
data = <<EOF
myuser:%dsFdsfg4a$#@cch#$%IvNykKW3d/
EOF
destination = "local/conf/.htpasswd"
change_mode = "signal"
change_signal = "SIGHUP"
}
}
}
}
nomad plan /etc/nomad.d/jobs/nginx_ingress.nomad
+ Job: "nginx"
+ Task Group: "nginx" (1 create)
+ Task: "nginx" (forces create)
Scheduler dry-run:
- All tasks successfully allocated.
Job Modify Index: 34024
To submit the job with version verification run:
nomad job run -check-index 34024 /etc/nomad.d/jobs/nginx_ingress.nomad
nomad status nginx
ID = nginx
Name = nginx
Submit Date = 2022-07-09T10:03:46+02:00
Allocations
ID Node ID Task Group Version Desired Status Created Modified
c92a039c 005f708b nginx 4 run running 20s ago 6s ago
nomad alloc logs c92a039c
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
I now have both my web services on port 55444 and 44555 proxied through ports 8080 and 8081 with the user login provided in .htpasswd