Salt Beacons Module
Salt beacons are an event generation mechanism. Beacons leverage the Salt reactor system to make changes when beacon events occur.
Beacons let you use the Salt event system to monitor non-Salt processes. The beacon system allows the minion to hook into a variety of system processes and continually monitor these processes. When monitored activity occurs in a system process, an event is sent on the Salt event bus that can be used to trigger a reactor.
Salt beacons can currently monitor and send Salt events for many system activities, including:
-
file system changes
-
system load
-
service status
-
shell activity, such as user login
-
network and disk usage
See beacon modules for a current list.
Beacons are typically enabled by placing a beacons: top level block in /etc/salt/minion or any file in /etc/salt/minion.d/ such as /etc/salt/minion.d/beacons.conf or add it to pillars for that minion: They can be used with:
beacons.listbeacons.enablebeacons.disable
inotify
Create Watched File
We have created an SLS file that creates an Apache welcome page for us. We can now use a Beacon to keep an eye on that file and trigger our Reactor in case the file was changed:
/srv/salt/apache/welcome.sls
# Adding a blank front page
{% set name = salt.pillar.get('name') %}
check_pillar_values:
test.check_pillar:
- present:
- name
- failhard: True
welcome_page:
file.managed:
- name: /var/www/html/index.html
- contents: |
<!doctype html>
<body>
<h1>{{ name }}.</h1>
</body>
Add Beacon Configs to Minions
On the Salt minion, add the following configuration to /etc/salt/minion.d/beacons.conf:
beacons:
inotify:
- files:
/var/www/html/index.html:
mask:
- modify
- disable_during_state_run: True
Make sure that iNotify is installed on your minions:
salt debianMinions pkg.install python-pyinotify/salt redhatMinions pkg.install python-inotify! You can debug Pillars on your Minion withsalt-call -l debug saltutil.pillar_refreshandsalt-call -l debug pillar.items
This did not work for me - see how to install pyinotify on Ubuntu 20.04 and debug it below
Save the configuration file and restart the minion service. The beacon is now set up to notify salt upon modifications made to the file.
The Beacon can now be enabled by:
salt ubuntuAsus beacons.enable
ubuntuAsus:
----------
comment:
Enabled beacons on minion.
result:
True
salt ubuntuAsus beacons.list
ubuntuAsus:
beacons:
enabled: true
inotify:
- files:
/var/www/html/index.html:
mask:
- modify
- disable_during_state_run: true
You can test the beacon by running nano /var/www/html/index.html on your Minion and changing the file while having salt-run state.event pretty=true open on your master:
salt/beacon/ubuntuAsus/inotify//var/www/html/index.html {
"_stamp": "2020-08-08T18:00:42.793635",
"change": "IN_MODIFY",
"id": "ubuntuAsus",
"path": "/var/www/html/index.html"
}
Add Beacon Configs to Pillars
Alternatively, you can add the monitor task to nano /srv/pillar/monitor_welcome.sls:
beacons:
inotify:
- files:
/var/www/html/index.html:
mask:
- modify
- disable_during_state_run: True
Now share the Pillar function with affected Minions with the Top file nano /srv/pillar/top.sls:
base:
'*':
- name
- mine
ubuntuAsus:
- monitor_welcome
And push the update to your Minions with:
salt ubuntuAsus saltutil.refresh_pillar
salt ubuntuAsus pillar.get beacons
ubuntuAsus:
----------
inotify:
----------
/var/www/html/index.html:
----------
mask:
- close_write
disable_during_state_run:
True
The Beacon can now be enabled by:
salt ubuntuAsus beacons.enable
salt ubuntuAsus beacons.list
You can test the beacon by running nano /var/www/html/index.html on your Minion and changing the file while having salt-run state.event pretty=true open on your master:
salt/beacon/ubuntuAsus/inotify//var/www/html/index.html {
"_stamp": "2020-08-08T18:00:42.793635",
"change": "IN_MODIFY",
"id": "ubuntuAsus",
"path": "/var/www/html/index.html"
}
Notify Reactor
Now we need to react to the change notification by changing the file back to the default state:
nano /etc/salt/master.d/local.conf
Add the salt/beacon/*/inotify//var/www/html/index.html event to Reactor and have it start a script that fixes the issue:
reactor:
- 'instar/custom/*':
- /srv/reactor/highstate.sls
- 'instar/deploy/testapp':
- /srv/reactor/deploy_testapp.sls
- 'salt/beacon/*/inotify//var/www/html/index.html':
- /srv/reactor/fix_welcome.sls
Restart the master systemctl restart salt-master and create the nano /srv/reactor/fix_welcome.sls script:
fix_welcome:
cmd.state.sls:
- tgt: {{ data.id }}
- arg:
- apache.welcome
Now go and change the /var/www/html/index.html file again. Reactor should automatically fire and change it back to the default state:
salt-run state.event pretty=true
salt/auth {
"_stamp": "2020-08-08T18:23:40.820210",
"act": "accept",
"id": "ubuntuAsus",
"pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w=AQAB\n-----END PUBLIC KEY-----",
"result": true
}
salt/beacon/ubuntuAsus/inotify//var/www/html/index.html {
"_stamp": "2020-08-08T18:23:40.832944",
"change": "IN_MODIFY",
"id": "ubuntuAsus",
"path": "/var/www/html/index.html"
}
20200808182343071200 {
"_stamp": "2020-08-08T18:23:43.072071",
"minions": [
"ubuntuAsus"
]
}
salt/job/20200808182343071200/new {
"_stamp": "2020-08-08T18:23:43.072389",
"arg": [
"apache.welcome"
],
"fun": "state.sls",
"jid": "20200808182343071200",
"minions": [
"ubuntuAsus"
],
"missing": [],
"tgt": "ubuntuAsus",
"tgt_type": "glob",
"user": "root"
}
salt/auth {
"_stamp": "2020-08-08T18:23:43.090909",
"act": "accept",
"id": "ubuntuMaster",
"pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w=AQAB\n-----END PUBLIC KEY-----",
"result": true
}
salt/auth {
"_stamp": "2020-08-08T18:23:43.095612",
"act": "accept",
"id": "ubuntuAsus",
"pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w=AQAB\n-----END PUBLIC KEY-----",
"result": true
}
minion/refresh/ubuntuAsus {
"Minion data cache refresh": "ubuntuAsus",
"_stamp": "2020-08-08T18:23:43.217756"
}
salt/job/20200808182343071200/ret/ubuntuAsus {
"_stamp": "2020-08-08T18:23:43.466837",
"cmd": "_return",
"fun": "state.sls",
"fun_args": [
"apache.welcome"
],
"id": "ubuntuAsus",
"jid": "20200808182343071200",
"out": "highstate",
"retcode": 0,
"return": {
"file_|-welcome_page_|-/var/www/html/index.html_|-managed": {
"__id__": "welcome_page",
"__run_num__": 1,
"__sls__": "apache.welcome",
"changes": {
"diff": "--- \n+++ \n@@ -1,4 +1,4 @@\n <!doctype html>\n <body>\n- <h1>Centos Rocks!</h1>\n+ <h1>Ubuntu Rocks.</h1>\n </body>\n"
},
"comment": "File /var/www/html/index.html updated",
"duration": 15.031,
"name": "/var/www/html/index.html",
"result": true,
"start_time": "18:23:43.433832"
},
"test_|-check_pillar_values_|-check_pillar_values_|-check_pillar": {
"__id__": "check_pillar_values",
"__run_num__": 0,
"__sls__": "apache.welcome",
"changes": {},
"comment": "",
"duration": 1.504,
"name": "check_pillar_values",
"result": true,
"start_time": "18:23:43.429323"
}
},
"success": true
}
salt/beacon/ubuntuAsus/inotify//var/www/html/index.html {
"_stamp": "2020-08-08T18:23:43.797938",
"change": "IN_IGNORED",
"id": "ubuntuAsus",
"path": "/var/www/html/index.html"
}
20200808182343808522 {
"_stamp": "2020-08-08T18:23:43.808797",
"minions": [
"ubuntuAsus"
]
}
salt/job/20200808182343808522/new {
"_stamp": "2020-08-08T18:23:43.809341",
"arg": [
"apache.welcome"
],
"fun": "state.sls",
"jid": "20200808182343808522",
"minions": [
"ubuntuAsus"
],
"missing": [],
"tgt": "ubuntuAsus",
"tgt_type": "glob",
"user": "root"
}
minion/refresh/ubuntuAsus {
"Minion data cache refresh": "ubuntuAsus",
"_stamp": "2020-08-08T18:23:43.933767"
}
salt/job/20200808182343808522/ret/ubuntuAsus {
"_stamp": "2020-08-08T18:23:44.107148",
"cmd": "_return",
"fun": "state.sls",
"fun_args": [
"apache.welcome"
],
"id": "ubuntuAsus",
"jid": "20200808182343808522",
"out": "highstate",
"retcode": 0,
"return": {
"file_|-welcome_page_|-/var/www/html/index.html_|-managed": {
"__id__": "welcome_page",
"__run_num__": 1,
"__sls__": "apache.welcome",
"changes": {},
"comment": "File /var/www/html/index.html is in the correct state",
"duration": 12.866,
"name": "/var/www/html/index.html",
"result": true,
"start_time": "18:23:44.081579"
},
"test_|-check_pillar_values_|-check_pillar_values_|-check_pillar": {
"__id__": "check_pillar_values",
"__run_num__": 0,
"__sls__": "apache.welcome",
"changes": {},
"comment": "",
"duration": 1.119,
"name": "check_pillar_values",
"result": true,
"start_time": "18:23:44.078418"
}
},
"success": true
}
Debugging Pyinotify
File system monitoring through inotify can be interfaced through Python using pyinotify. This guide will demonstrate how to use a Python script to monitor a directory then explore practical uses by incorporating async modules or running additional threads.
Start by installing pip3 and use it to install pyinotify:
sudo apt-get install python3-pip
pip3 install pyinotify
Take this script and copy it to your home directory:
notify_me.py
import os
import pyinotify
class EventProcessor(pyinotify.ProcessEvent):
_methods = ["IN_CREATE",
"IN_OPEN",
"IN_ACCESS",
"IN_ATTRIB",
"IN_CLOSE_NOWRITE",
"IN_CLOSE_WRITE",
"IN_DELETE",
"IN_DELETE_SELF",
"IN_IGNORED",
"IN_MODIFY",
"IN_MOVE_SELF",
"IN_MOVED_FROM",
"IN_MOVED_TO",
"IN_Q_OVERFLOW",
"IN_UNMOUNT",
"default"]
def process_generator(cls, method):
def _method_name(self, event):
print("Method name: process_{}()\n"
"Path name: {}\n"
"Event Name: {}\n".format(method, event.pathname, event.maskname))
_method_name.__name__ = "process_{}".format(method)
setattr(cls, _method_name.__name__, _method_name)
for method in EventProcessor._methods:
process_generator(EventProcessor, method)
watch_manager = pyinotify.WatchManager()
event_notifier = pyinotify.Notifier(watch_manager, EventProcessor())
watch_this = os.path.abspath("notification_dir")
watch_manager.add_watch(watch_this, pyinotify.ALL_EVENTS)
event_notifier.loop()
Now Create a folder notification_dir inside your home directory and run the Python script:
python3 notify_me.py
Open another Terminal and create a file inside the folder:
touch ~/notification_dir/test
Switch back to your watch task and see if it recognized the change:
Method name: process_IN_CREATE()
Path name: /root/notification_dir/test
Event Name: IN_CREATE
Method name: process_IN_OPEN()
Path name: /root/notification_dir/test
Event Name: IN_OPEN
Method name: process_IN_ATTRIB()
Path name: /root/notification_dir/test
Event Name: IN_ATTRIB
Method name: process_IN_CLOSE_WRITE()
Path name: /root/notification_dir/test
Event Name: IN_CLOSE_WRITE
This worked! Pyinotify is now ready to go!
Debugging the Reactor
Run the Master in Debug mode and re-run everything to see what is happening:
systemctl stop salt-master
salt-master -l debug