Salt Reactor and System Events
Salt Master Events
These events are fired on the Salt Master event bus.
- Salt Master Events
- Authentication events
- Start events
- Key events
- Job events
- Runner Events
- Presence Events
- Cloud Events
To view events run the following command on your master:
salt-run state.event pretty=true
Or this on your minion:
salt-call state.event pretty=true
If you now run a salt ubuntuAsus test.ping you will start to see an event log like:
20200807165432241482 {
"_stamp": "2020-08-07T16:54:32.241679",
"minions": [
"ubuntuAsus"
]
}
salt/job/20200807165432241482/new {
"_stamp": "2020-08-07T16:54:32.242042",
"arg": [],
"fun": "test.ping",
"jid": "20200807165432241482",
"minions": [
"ubuntuAsus"
],
"missing": [],
"tgt": "ubuntuAsus",
"tgt_type": "glob",
"user": "root"
}
salt/job/20200807165432241482/ret/ubuntuAsus {
"_stamp": "2020-08-07T16:54:32.295023",
"cmd": "_return",
"fun": "test.ping",
"fun_args": [],
"id": "ubuntuAsus",
"jid": "20200807165432241482",
"retcode": 0,
"return": true,
"success": true
}
You can see the timestamped event going out from the master and the return from your minion.
salt.runners.state
Watch Salt's event bus and block until the given tag is matched:
# Reboot a minion and run highstate when it comes back online
salt 'ubuntuAsus' system.reboot && \\
salt-run state.event 'salt/minion/ubuntuAsus/start' count=1 quiet=True && \\
salt 'ubuntuAsus' state.highstate
# Reboot multiple minions and run highstate when all are back online
salt -L 'ubuntuAsus,ubuntuMaster' system.reboot && \\
salt-run state.event 'salt/minion/*/start' count=3 quiet=True && \\
salt -L 'ubuntuAsus,ubuntuMaster' state.highstate
Custom Events
Use the Salt Event System to fire events from the master to the minion and vice-versa.
event.send
Start the event listener on your master salt-run state.event pretty=true and send a custom event from your minion:
salt-call event.send instar/custom/event
You will see the incoming event on your master:
salt/auth {
"_stamp": "2020-08-07T17:44:14.155523",
"act": "accept",
"id": "ubuntuAsus",
"pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w=AQAB\n-----END PUBLIC KEY-----",
"result": true
}
minion/refresh/ubuntuAsus {
"Minion data cache refresh": "ubuntuAsus",
"_stamp": "2020-08-07T17:44:14.242055"
}
instar/custom/event {
"_stamp": "2020-08-07T17:44:14.294823",
"cmd": "_minion_event",
"data": {
"__pub_fun": "event.send",
"__pub_jid": "20200807174414296024",
"__pub_pid": 85669,
"__pub_tgt": "salt-call"
},
"id": "ubuntuAsus",
"tag": "instar/custom/event"
}
salt/job/20200807174414299104/ret/ubuntuAsus {
"_stamp": "2020-08-07T17:44:14.299934",
"arg": [
"instar/custom/event"
],
"cmd": "_return",
"fun": "event.send",
"fun_args": [
"instar/custom/event"
],
"id": "ubuntuAsus",
"jid": "20200807174414299104",
"retcode": 0,
"return": true,
"tgt": "ubuntuAsus",
"tgt_type": "glob"
}
You can also send JSON formatted data with your event:
salt-call event.send instar/custom/event '{"what": "something happened", "is it serious": true}'
You will see the incoming event on your master:
salt/auth {
"_stamp": "2020-08-07T18:22:48.152369",
"act": "accept",
"id": "ubuntuAsus",
"pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w=AQAB\n-----END PUBLIC KEY-----",
"result": true
}
minion/refresh/ubuntuAsus {
"Minion data cache refresh": "ubuntuAsus",
"_stamp": "2020-08-07T18:22:48.242329"
}
instar/custom/event {
"_stamp": "2020-08-07T18:22:48.296609",
"cmd": "_minion_event",
"data": {
"__pub_fun": "event.send",
"__pub_jid": "20200807182248298801",
"__pub_pid": 85938,
"__pub_tgt": "salt-call",
"is it serious": true,
"what": "something happened"
},
"id": "ubuntuAsus",
"tag": "instar/custom/event"
}
salt/job/20200807182248300952/ret/ubuntuAsus {
"_stamp": "2020-08-07T18:22:48.302004",
"arg": [
"instar/custom/event",
"{\"what\": \"something happened\", \"is it serious\": true}"
],
"cmd": "_return",
"fun": "event.send",
"fun_args": [
"instar/custom/event",
"{\"what\": \"something happened\", \"is it serious\": true}"
],
"id": "ubuntuAsus",
"jid": "20200807182248300952",
"retcode": 0,
"return": true,
"tgt": "ubuntuAsus",
"tgt_type": "glob"
}
You can also add environment variables, grains or pillar data with your event:
salt-call event.send instar/custom/event with_grains=true //send all grains
salt-call event.send instar/custom/event with_grains='[os]'
You will see the incoming event on your master:
salt/auth {
"_stamp": "2020-08-07T18:35:38.536997",
"act": "accept",
"id": "ubuntuAsus",
"pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w=AQAB\n-----END PUBLIC KEY-----",
"result": true
}
minion/refresh/ubuntuAsus {
"Minion data cache refresh": "ubuntuAsus",
"_stamp": "2020-08-07T18:35:38.622552"
}
instar/custom/event {
"_stamp": "2020-08-07T18:35:38.669904",
"cmd": "_minion_event",
"data": {
"__pub_fun": "event.send",
"__pub_jid": "20200807183538672084",
"__pub_pid": 86062,
"__pub_tgt": "salt-call",
"grains": {
"os": "Ubuntu"
}
},
"id": "ubuntuAsus",
"tag": "instar/custom/event"
}
salt/job/20200807183538673638/ret/ubuntuAsus {
"_stamp": "2020-08-07T18:35:38.674478",
"arg": [
"instar/custom/event",
"with_grains=[os]"
],
"cmd": "_return",
"fun": "event.send",
"fun_args": [
"instar/custom/event",
"with_grains=[os]"
],
"id": "ubuntuAsus",
"jid": "20200807183538673638",
"retcode": 0,
"return": true,
"tgt": "ubuntuAsus",
"tgt_type": "glob"
}
Reactor
Invoking a Minion State by Event
The Salt Reactor can be configured to listen to salt.events and execute SLS files when triggered. We can configure Reactor by adding the following lines to our local Master config /etc/salt/master.d/local.conf :
reactor:
- 'instar/custom/*':
- /srv/reactor/highstate.sls
And create the nano /srv/reactor/highstate.sls file:
run_highstate:
cmd.state.highstate:
- tgt: 'ubuntuAsus'
(Reactor equivalent to salt ubuntuAsus state.highstate)
And restart the master:
systemctl restart salt-master
We can now trigger an event on our minion that matches instar/custom/* to see our Reactor spring into action:
salt-call event.send instar/custom/event
salt/auth {
"_stamp": "2020-08-07T19:33:47.014561",
"act": "accept",
"id": "ubuntuAsus",
"pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w=AQAB\n-----END PUBLIC KEY-----",
"result": true
}
minion/refresh/ubuntuAsus {
"Minion data cache refresh": "ubuntuAsus",
"_stamp": "2020-08-07T19:33:47.119011"
}
instar/custom/event {
"_stamp": "2020-08-07T19:33:47.221573",
"cmd": "_minion_event",
"data": {
"__pub_fun": "event.send",
"__pub_jid": "20200807193347220371",
"__pub_pid": 86296,
"__pub_tgt": "salt-call"
},
"id": "ubuntuAsus",
"tag": "instar/custom/event"
}
salt/job/20200807193347229445/ret/ubuntuAsus {
"_stamp": "2020-08-07T19:33:47.231956",
"arg": [
"instar/custom/event"
],
"cmd": "_return",
"fun": "event.send",
"fun_args": [
"instar/custom/event"
],
"id": "ubuntuAsus",
"jid": "20200807193347229445",
"retcode": 0,
"return": true,
"tgt": "ubuntuAsus",
"tgt_type": "glob"
}
20200807193349446458 {
"_stamp": "2020-08-07T19:33:49.447265",
"minions": [
"ubuntuAsus"
]
}
salt/job/20200807193349446458/new {
"_stamp": "2020-08-07T19:33:49.447548",
"arg": [],
"fun": "state.highstate",
"jid": "20200807193349446458",
"minions": [
"ubuntuAsus"
],
"missing": [],
"tgt": "ubuntuAsus",
"tgt_type": "glob",
"user": "root"
}
salt/auth {
"_stamp": "2020-08-07T19:33:49.464831",
"act": "accept",
"id": "ubuntuMaster",
"pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w=AQAB\n-----END PUBLIC KEY-----",
"result": true
}
salt/auth {
"_stamp": "2020-08-07T19:33:49.469238",
"act": "accept",
"id": "ubuntuAsus",
"pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w=AQAB\n-----END PUBLIC KEY-----",
"result": true
}
minion/refresh/ubuntuAsus {
"Minion data cache refresh": "ubuntuAsus",
"_stamp": "2020-08-07T19:33:49.625946"
}
salt/job/20200807193349446458/ret/ubuntuAsus {
"_stamp": "2020-08-07T19:33:49.773631",
"cmd": "_return",
"fun": "state.highstate",
"fun_args": [],
"id": "ubuntuAsus",
"jid": "20200807193349446458",
"out": "highstate",
"retcode": 2,
"return": {
"no_|-states_|-states_|-None": {
"__run_num__": 0,
"changes": {},
"comment": "No Top file or master_tops data matches found. Please see master log for details.",
"name": "No States",
"result": false
}
},
"success": false
}
no.None {
"__run_num__": 0,
"_stamp": "2020-08-07T19:33:49.773962",
"changes": {},
"comment": "No Top file or master_tops data matches found. Please see master log for details.",
"name": "No States",
"result": false,
"retcode": 2
}
The highstate event was successfully triggered by the Reactor - have to check why the top file for it was not found.
Forwarding Pillar Data
We previously created a file that downloaded source code from Github: nano /srv/salt/apptest.sls. Here we hardcoded the repository branch rev: master. We can use Reactor to forward this information for us:
{% from "apache/map.sls" import apache with context %}
{% set version = salt.pillar.get('version', 'master') %}
include:
- apache
apptest:
git.latest:
- name: https://github.com/mpolinowski/docker-elk.git
- rev: {{ version }}
- target: /opt/apptest
- watch_in:
- service: enable_apache
...
Query salt.pillar.get('version') to set Git branch or default to master. So we are now able to inject the branch by adding the respective pillar data to our state command:
salt '*' state.sls apptest pillar='{version: development}'
We can now add an event to our Reactor config that should trigger this state - nano /etc/salt/master.d/local.conf :
reactor:
- 'instar/custom/*':
- /srv/reactor/highstate.sls
- 'instar/deploy/testapp':
- /srv/reactor/deploy_testapp.sls
Now we can add the Reactor script nano /srv/reactor/deploy_testapp.sls:
deploy_testapp:
cmd.state.sls:
- tgt: {{ data.id }}
- kwarg:
mods: apptest
pillar:
version: {{ data.data.version }}
Which is the equivalent of running salt '*' state.sls mods=apptest manually - the version has to be injected by the event that triggers our Reactor script. We can now listen to the event bus on our master salt-run state.event pretty=true and manually trigger the instar/deploy/testapp event on our minion:
salt-call event.send instar/deploy/testapp version=master
The event will be registered on our master:
salt/auth {
"_stamp": "2020-08-08T10:58:42.732432",
"act": "accept",
"id": "ubuntuAsus",
"pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w=AQAB\n-----END PUBLIC KEY-----"
"result": true
}
minion/refresh/ubuntuAsus {
"Minion data cache refresh": "ubuntuAsus",
"_stamp": "2020-08-08T10:58:42.822297"
}
instar/deploy/testapp {
"_stamp": "2020-08-08T10:58:42.876885",
"cmd": "_minion_event",
"data": {
"__pub_fun": "event.send",
"__pub_jid": "20200808105842881554",
"__pub_pid": 3292,
"__pub_tgt": "salt-call",
"version": "master"
},
"id": "ubuntuAsus",
"tag": "instar/deploy/testapp"
}
salt/job/20200808105842880875/ret/ubuntuAsus {
"_stamp": "2020-08-08T10:58:42.881836",
"arg": [
"instar/deploy/testapp",
"version=master"
],
"cmd": "_return",
"fun": "event.send",
"fun_args": [
"instar/deploy/testapp",
"version=master"
],
"id": "ubuntuAsus",
"jid": "20200808105842880875",
"retcode": 0,
"return": true,
"tgt": "ubuntuAsus",
"tgt_type": "glob"
}
20200808105842909103 {
"_stamp": "2020-08-08T10:58:42.909751",
"minions": [
"ubuntuAsus"
]
}
salt/job/20200808105842909103/new {
"_stamp": "2020-08-08T10:58:42.909976",
"arg": [
{
"__kwarg__": true,
"mods": "apptest",
"pillar": {
"version": "master"
}
}
],
"fun": "state.sls",
"jid": "20200808105842909103",
"minions": [
"ubuntuAsus"
],
"missing": [],
"tgt": "ubuntuAsus",
"tgt_type": "glob",
"user": "root"
}
minion/refresh/ubuntuAsus {
"Minion data cache refresh": "ubuntuAsus",
"_stamp": "2020-08-08T10:58:43.041467"
}
salt/job/20200808105842909103/ret/ubuntuAsus {
"_stamp": "2020-08-08T10:58:49.734560",
"cmd": "_return",
"fun": "state.sls",
"fun_args": [
{
"mods": "apptest",
"pillar": {
"version": "master"
}
}
],
"id": "ubuntuAsus",
"jid": "20200808105842909103",
"out": "highstate",
"retcode": 0,
"return": {
"event_|-notify_of_fail_|-apptest/failed_|-send": {
"__run_num__": 4,
"__sls__": "apptest",
"__state_ran__": false,
"changes": {},
"comment": "State was not run because onfail req did not change",
"duration": 0.004,
"result": true,
"start_time": "10:58:49.740913"
},
"git_|-apptest_|-https://github.com/mpolinowski/docker-elk.git_|-latest": {
"__id__": "apptest",
"__run_num__": 2,
"__sls__": "apptest",
"changes": {},
"comment": "Repository /opt/apptest is up-to-date",
"duration": 3184.049,
"name": "https://github.com/mpolinowski/docker-elk.git",
"result": true,
"start_time": "10:58:46.510959"
},
"module_|-reload_apache_|-service.stop_|-run": {
"__run_num__": 1,
"__sls__": "apptest",
"changes": {},
"comment": "No changes detected",
"duration": 0.006,
"result": true,
"start_time": "10:58:46.510837"
},
"pkg_|-install_apache_|-apache2_|-installed": {
"__id__": "install_apache",
"__run_num__": 0,
"__sls__": "apache",
"changes": {},
"comment": "All specified packages are already installed",
"duration": 22.176,
"name": "apache2",
"result": true,
"start_time": "10:58:44.051565"
},
"service_|-enable_apache_|-apache2_|-running": {
"__id__": "enable_apache",
"__run_num__": 3,
"__sls__": "apache",
"changes": {},
"comment": "The service apache2 is already running",
"duration": 43.795,
"name": "apache2",
"result": true,
"start_time": "10:58:49.695390"
}
},
"success": true
}