Hashicorp Nomad Access Control

• Nomad ACL System
  • Core objects
• Enable ACLs on Nomad servers
  • Stage a Permissive Anonymous Policy
  • Run the Bootstrap Command
    • Using the Management Token
  • Deploy your anonymous policy
  • Create Management Tokens for Other Regions
  • Generate a Client Token
• Nomad ACL Policies
  • Agent rules
  • Operator rules
  • Quota rules
  • Host Volume rules
  • Plugin rules
• Generate User Token
  • Application Developer
  • Production Operations
• Generate Nomad Tokens with HashiCorp Vault (WIP)

Continuation of Hashicorp Nomad Dojo

Nomad ACL System

Core objects

• Tokens: Are either management or client types. The management tokens are effectively "root" in the system and can perform any operation (see Capabilities below). The client tokens are associated with one or more ACL policies which grant specific capabilities.
• Policies: Consist of a set of rules defining the capabilities or actions to be granted. The rules define the capabilities of a Nomad ACL token for accessing the objects in a Nomad cluster, objects like namespaces, node, agent, operator, quota, etc.
• Capabilities: Set of actions that can be performed including listing jobs, submitting jobs, querying nodes, etc.

Enable ACLs on Nomad servers

Set the enabled value of the acl stanza to true. The acl stanza is a top-level stanza:

acl {
  enabled = true
}

Add these lines to all servers and restart the Nomad service.

Stage a Permissive Anonymous Policy

Once the ACL system is enabled, you need to generate the initial token. This first management token is used to bootstrap the system. Care should be taken not to lose all of your management tokens. If you do, you will need to re-bootstrap the ACL subsystem.

Once the nomad acl bootstrap command is run, Nomad's default-deny policy will become enabled. You should have an acceptable anonymous policy prepared and ready to submit immediately after bootstrapping.

Create a file named anonymous.policy.hcl with this HCL content. You can use this as a transitional anonymous policy, which will minimize time in which requests can not be submitted to the cluster once you bootstrap:

namespace "*" {
  policy       = "write"
  capabilities = ["alloc-node-exec"]
}

agent {
  policy = "write"
}

operator {
  policy = "write"
}

quota {
  policy = "write"
}

node {
  policy = "write"
}

host_volume "*" {
  policy = "write"
}

Run the Bootstrap Command

Once the initial bootstrap is performed, it cannot be performed again unless the reset procedure is complete. Make sure to save this AccessorID and SecretID. The bootstrap token is a management type token, meaning it can perform any operation.

nomad acl bootstrap
Accessor ID  = 5b7fd453-d3f7-6814-81dc-fcfe6daedea5
Secret ID    = 9184ec35-65d4-9258-61e3-0c066d0a45c5
Name         = Bootstrap Token
Type         = management
Global       = true
Policies     = n/a
Create Time  = 2022-06-15 04:04:46.571360464 +0000 UTC
Create Index = 7
Modify Index = 7

• Accessor ID - The public identifier for a specific token. It can be used to look up information about a token or to revoke a token.
• Secret ID - Used to make requests to Nomad and should be kept private
• Name (optional) - A user-supplied identifier.
• Type - Shows the type of the token. Management tokens are used for administration and can be thought of as root-level tokens. Client tokens are used for applications and can have policies assigned to them at creation.
• Global - (bool) Indicates whether or not the token was created with the --global flag. Global tokens are replicated from the authoritative region to other connected regions in the cluster.
• Policies - (string) A list of the policies assigned to the token.
• Create Time - Wall clock time on the Nomad server leader when the token was generated.
• Create/Modification index - Used by Nomad internally to determine when a token was created and/or modified relative to other system events.

Using the Management Token

We are now locked out from using the CLI if we don't provide a management ACL Token with every request. To make this more comfortable we can add it to our shell config, e.g. ~/.bashrc or ./zshrc:

export NOMAD_TOKEN="9184ec35-65d4-9258-61e3-0c066d0a45c5"

Additionally, we are now also locked out from using the Nomad UI and have to provide the Secret ID:

Deploy your anonymous policy

Next, install the anonymous policy with the nomad acl policy apply command:

nomad acl policy apply -description "Anonymous policy (full-access)" anonymous anonymous.policy.hcl

You can verify that the policy is active by e.g. un-setting the token in your Nomad UI - you should now be able to access everything again without providing any tokens.

Create Management Tokens for Other Regions

Once you have bootstrapped ACLs on the servers of the authoritative region, you can create the replication tokens for all of the non-authoritative regions in a multi-region configuration. Create the replication token with the nomad acl token create command.

Use the -global flag on the nomad acl token create command or set the "Global" parameter to true in your Create Token API call to create global tokens:

nomad acl token create -type="management" -global=true -name="Cluster Replication Token"                            
Accessor ID  = ec175d30-26ea-4a54-4850-45f833acece5
Secret ID    = 9e2bb5ed-b3af-6bf3-5bbc-16dc684c5c31
Name         = Cluster Replication Token
Type         = management
Global       = true
Policies     = n/a
Create Time  = 2022-06-15 04:38:21.360199318 +0000 UTC
Create Index = 11462
Modify Index = 11462

Nomad has two token types: local and global. Local tokens are created and stored within the current Nomad region. However, for multi-region clusters, you can also create global tokens which are created in the authoritative region and replicated to the other regions.

Tokens can be deleted using the CLI:

nomad acl token delete 9e2bb5ed-b3af-6bf3-5bbc-16dc684c5c31

Generate a Client Token

Create a client token called "client1" that is associated with the policies "app1" and "app2". It does not matter that these policies do not yet exist. Nonexistent policies provide no capabilities to a token, but they will not cause an error:

nomad acl token create -name="client1" -policy="app1" -policy="app2"
Accessor ID  = a7bef400-a66c-d163-52c4-5ff4978f19b8
Secret ID    = 950e462b-b333-dd52-1e18-76837e9b97fa
Name         = client1
Type         = client
Global       = false
Policies     = [app1 app2]
Create Time  = 2020-01-23 20:10:11.600915 +0000 UTC
Create Index = 19
Modify Index = 19

Nomad ACL Policies

An ACL policy contains one or more rules. Rules contain coarse-grained policy dispositions. Rules typically have several policy dispositions:

• read: allow the resource to be read but not modified
• write: allow the resource to be read and modified
• deny: do not allow the resource to be read or modified. Deny takes precedence when multiple policies are associated with a token.
• list: allow the resource to be listed but not inspected in detail. Applies only to plugins.

Namespace rules are keyed by the namespace name they apply to. When no namespace is specified, the default namespace is the one used.

In addition to the coarse-grained policy disposition, the namespace stanza allows setting a more fine grained list of capabilities. This includes:

• deny - When multiple policies are associated with a token, deny will take precedence and prevent any capabilities.
• list-jobs - Allows listing the jobs and seeing coarse grain status.
• parse-job - Allows parsing a job from HCL to JSON.
• read-job - Allows inspecting a job and seeing fine grain status.
• submit-job - Allows jobs to be submitted or modified.
• dispatch-job - Allows jobs to be dispatched
• read-logs - Allows the logs associated with a job to be viewed.
• read-fs - Allows the filesystem of allocations associated to be viewed.
• alloc-exec - Allows an operator to connect and run commands in running allocations.
• alloc-node-exec - Allows an operator to connect and run commands in allocations running without filesystem isolation, for example, raw_exec jobs.
• alloc-lifecycle - Allows an operator to stop individual allocations manually.
• csi-register-plugin - Allows jobs to be submitted that register themselves as CSI plugins.
• csi-write-volume - Allows CSI volumes to be registered or deregistered.
• csi-read-volume - Allows inspecting a CSI volume and seeing fine grain status.
• csi-list-volume - Allows listing CSI volumes and seeing coarse grain status.
• csi-mount-volume - Allows jobs to be submitted that claim a CSI volume.
• list-scaling-policies - Allows listing scaling policies.
• read-scaling-policy - Allows inspecting a scaling policy.
• read-job-scaling - Allows inspecting the current scaling of a job.
• scale-job: Allows scaling a job up or down.
• sentinel-override - Allows soft mandatory policies to be overridden.

namespace "default" {
  policy = "write"
}

namespace "testing" {
  policy       = "read"
  capabilities = ["submit-job","list-jobs","read-job"]
}

namespace "sensitive" {
  policy = "read"
}

Agent rules

The agent rule controls access to the utility operations in the Agent API, such as join and leave. Agent rules are specified for all agents using the agent key:

agent {
  policy = "write"
}

Operator rules

The operator rule controls access to the Operator API. Operator rules look like:

operator {
  policy = "read"
}

Quota rules

The quota policy controls access to the quota specification operations in the Quota API, such as quota creation and deletion. Quota rules are specified for all quotas using the quota key:

quota {
  policy = "write"
}

Host Volume rules

The host_volume policy controls access to mounting and accessing host volumes.

host_volume "*" {
  policy = "write"
}

host_volume "prod-*" {
  policy = "deny"
}

host_volume "prod-ca-certificates" {
  policy = "read"
}

Plugin rules

The plugin policy controls access to CSI plugins, such as listing plugins or getting plugin status. Plugin rules are specified for all plugins using the plugin key:

plugin {
  policy = "read"
}

For more examples hit the docs.

Generate User Token

Application Developer

The application developer needs to be able to deploy an application into the Nomad cluster and control its lifecycle. Application developers are allowed to fetch logs from their running containers:

namespace "default" {
  policy = "read"
  capabilities = ["submit-job","dispatch-job","read-logs"]
}

nomad acl policy apply -description "Application Developer policy" app-dev app-dev.policy.hcl

nomad acl token create -name="App-dev token" -policy=app-dev -type=client | tee app-dev.token

Production Operations

The production operations team needs to be able to perform cluster maintenance and view the workload, including attached resources like volumes, in the running cluster. But should not be allowed to run or stop jobs in the cluster.

namespace "default" {
  policy = "read"
}

node {
  policy = "write"
}

agent {
  policy = "write"
}

operator {
  policy = "write"
}

plugin {
  policy = "list"
}

nomad acl policy apply -description "Production Operations policy" prod-ops prod-ops.policy.hcl

nomad acl token create -name="Test prod-ops token" -policy=prod-ops -type=client | tee prod-ops.token

Generate Nomad Tokens with HashiCorp Vault (WIP)

Generate Nomad Tokens with HashiCorp Vault