App Deployment with Hashicorp Nomad from Gitlab Part Deux

• Run a Docker Image with Gitlab Artifacts
  • Configuration Artifacts
  • Adding Volumes

Run a Docker Image with Gitlab Artifacts

I now want to run an ingress container for an existing web service. So far the NGINX ingress is configured to proxy-pass a couple of web frontend containers and direct traffic into that Docker cluster:

default.conf:

server {
    listen 80;
    listen [::]:80;

    server_name wiki.instar.com;

    return 301 https://$server_name$request_uri;
}


server {
    listen 443 ssl http2 default_server;
    listen [::]:443 ssl;
    ssl_certificate /opt/letsencrypt/live/wiki.instar.com/fullchain.pem;
    ssl_certificate_key /opt/letsencrypt/live/wiki.instar.com/privkey.pem;
    # include ssl/self-signed.conf;
    include ssl/ssl-params.conf;
    include /etc/nginx/conf.d/header.conf;

    server_name wiki.instar.com;

    location / {
        rewrite ^/(.*)$  /en/$1  permanent;
    }

    location /de/ {
      proxy_pass http://wiki_de:9999/;
    }

    location /en/ {
      proxy_pass http://wiki_en:8888/;
    }

    location /fr/ {
       proxy_pass http://wiki_fr:7777/;
    }
    
    location ~ /dl/.* {
        root /opt/wiki_downloads/;
        add_before_body /dl/theme/header.html;
        add_after_body /dl/theme/footer.html;
        autoindex on;
        autoindex_exact_size off;
        autoindex_format html;
        autoindex_localtime on;
    }


    error_page  404 /404.html;
    error_page  500 502 503 504 /50x.html;
    location = /50x.html {
    root   /usr/share/nginx/html;
  }
}

I was using a docker network to lock-down access to those containers - except traffic that was using the ingress. Which also allowed me to use the Docker DNS service to point NGINX to those front ends: proxy_pass http://wiki_de:9999/.

With Nomad I guess I now have to run those containers on the host network instead and proxy_pass from localhost instead. This is not an issue, though. As I can bind those ports to localhost and/or use the firewall to block all direct traffic.

I can spin up each of those containers using Nomad. Unlike before I am not using a dynamic port, but instead make it static for now (I am planing to add Consul to the mix later to handle the service discovery - but let's keep it simple'ish for now):

job "wiki_de" {
	datacenters = ["instaryun"]

	group "wiki_de" {
    count = 1
        
		network {
			mode = "host"
			port "http" {
				static = "9999"
			}
		}

		task "container" {
			driver = "docker"

			config {
				image = "mygitlab.mydomain.com:12345/wiki/wiki_de_mdx"
				ports = ["http"]

        auth {
          username = "mynomaduserongitlab"
          password = "acomplicatedpassword"
        }
			}
		}
	}
}

Configuration Artifacts

So there are three of those services and the last one is a file server - a download area. I can create this inside the Ingress job specification using the Artifact Stanza. This also takes care of the NGINX configuration files that I source control on Gitlab:

locals {
  ports = [
    {
      port_label = "http"
      port       = 80
    },
    {
      port_label = "https"
      port       = 443
    }
  ]
}

job "wiki_ingress" {
  datacenters = ["instaryun"]

  group "nginx" {
    count = 1

    network {
      mode = "host"
      dynamic "port" {
        for_each = local.ports
        labels   = [port.value.port_label]

        content {
          to = port.value.port
        }
      }
    }

    service {
      name = "nginx"
    }

    volume "letsencrypt" {
        type      = "host"
        read_only = true
        source    = "letsencrypt"
    }

    task "ingress_container" {
      driver = "docker"

      volume_mount {
            volume      = "letsencrypt"
            destination = "/opt/letsencrypt" #in the container
            read_only   = false
      }

      config {
        network_mode = "host"
        image = "nginx:1.23.0-alpine"
        ports = ["http","https"]
        volumes = [
          "local/nginx/configuration/conf.d:/etc/nginx/conf.d",
          "local/nginx/configuration/ssl:/etc/nginx/ssl",
          "local/nginx/configuration/nginx.conf:/etc/nginx/nginx.conf",
          "local/wiki_downloads:/opt/wiki_downloads",
        ]
      }

      artifact {
        source      = "git::git@my.gitlab.address.com/group/wiki_ingress.git"
        destination = "local/nginx"
        options {
          sshkey = "${base64encode(file(pathexpand("/etc/nomad.d/.ssh/id_rsa")))}"
          depth = 1
        }
      }

      artifact {
        source      = "git::git@my.gitlab.address.com/group/wiki_downloads.git"
        destination = "local/wiki_downloads"
        options {
          sshkey = "${base64encode(file(pathexpand("/etc/nomad.d/.ssh/id_rsa")))}"
          depth = 1
        }
      }      
    }
  }
}

Adding Volumes

You should first create the volume path before nomad is starting, then add the following configs in your client.hcl file [Plugin Stanza | Host Volume Stanza]:

nano /etc/nomad.d/client.hcl

client {
  enabled = true
  servers = ["myhost:port"]
  host_volume "letsencrypt" {
    path = "/etc/letsencrypt"
    read_only = true
  }
}

# Docker Configuration
plugin "docker" {
    volumes {
      enabled = true
    }
}

And then in the job specifications, inside the Group Stanza define the volume:

volume "letsencrypt" {
     type      = "host"
     read_only = true
     source    = "letsencrypt"
}

and then finally add following in the Task Stanza use the defined volume:

volume_mount {
      volume      = "letsencrypt"
      destination = "/opt/letsencrypt" #<-- in the container
      read_only   = false
}

That is a bit complicated for a simple volume mount....