Skip to main content

Salt Beacons Module

Mon Kok, Hong Kong

Salt beacons are an event generation mechanism. Beacons leverage the Salt reactor system to make changes when beacon events occur.

Beacons let you use the Salt event system to monitor non-Salt processes. The beacon system allows the minion to hook into a variety of system processes and continually monitor these processes. When monitored activity occurs in a system process, an event is sent on the Salt event bus that can be used to trigger a reactor.

Salt beacons can currently monitor and send Salt events for many system activities, including:

  • file system changes

  • system load

  • service status

  • shell activity, such as user login

  • network and disk usage

See beacon modules for a current list.

Beacons are typically enabled by placing a beacons: top level block in /etc/salt/minion or any file in /etc/salt/minion.d/ such as /etc/salt/minion.d/beacons.conf or add it to pillars for that minion: They can be used with:

  • beacons.list
  • beacons.enable
  • beacons.disable

inotify

Create Watched File

We have created an SLS file that creates an Apache welcome page for us. We can now use a Beacon to keep an eye on that file and trigger our Reactor in case the file was changed:

/srv/salt/apache/welcome.sls

# Adding a blank front page
{% set name = salt.pillar.get('name') %}

check_pillar_values:
  test.check_pillar:
    - present:
      - name
    - failhard: True

welcome_page:
  file.managed:
    - name: /var/www/html/index.html
    - contents: |
        <!doctype html>
        <body>
            <h1>{{ name }}.</h1>
        </body>

Add Beacon Configs to Minions

On the Salt minion, add the following configuration to /etc/salt/minion.d/beacons.conf:

beacons:
  inotify:
    - files:
        /var/www/html/index.html:
          mask:
            - modify     
    - disable_during_state_run: True

Make sure that iNotify is installed on your minions: salt debianMinions pkg.install python-pyinotify / salt redhatMinions pkg.install python-inotify! You can debug Pillars on your Minion with salt-call -l debug saltutil.pillar_refresh and salt-call -l debug pillar.items

This did not work for me - see how to install pyinotify on Ubuntu 20.04 and debug it below

Save the configuration file and restart the minion service. The beacon is now set up to notify salt upon modifications made to the file.

The Beacon can now be enabled by:

salt ubuntuAsus beacons.enable

ubuntuAsus:
    ----------
    comment:
        Enabled beacons on minion.
    result:
        True


salt ubuntuAsus beacons.list

ubuntuAsus:
    beacons:
      enabled: true
      inotify:
      - files:
          /var/www/html/index.html:
            mask:
            - modify
      - disable_during_state_run: true

You can test the beacon by running nano /var/www/html/index.html on your Minion and changing the file while having salt-run state.event pretty=true open on your master:

salt/beacon/ubuntuAsus/inotify//var/www/html/index.html {
    "_stamp": "2020-08-08T18:00:42.793635",
    "change": "IN_MODIFY",
    "id": "ubuntuAsus",
    "path": "/var/www/html/index.html"
}

Add Beacon Configs to Pillars

Alternatively, you can add the monitor task to nano /srv/pillar/monitor_welcome.sls:

beacons:
  inotify:
    - files:
        /var/www/html/index.html:
          mask:
            - modify     
    - disable_during_state_run: True

Now share the Pillar function with affected Minions with the Top file nano /srv/pillar/top.sls:

base:
  '*':
    - name
    - mine
  ubuntuAsus:
    - monitor_welcome

And push the update to your Minions with:

salt ubuntuAsus saltutil.refresh_pillar

salt ubuntuAsus pillar.get beacons

ubuntuAsus:
    ----------
    inotify:
        ----------
        /var/www/html/index.html:
            ----------
            mask:
                - close_write
        disable_during_state_run:
            True

The Beacon can now be enabled by:

salt ubuntuAsus beacons.enable

salt ubuntuAsus beacons.list

You can test the beacon by running nano /var/www/html/index.html on your Minion and changing the file while having salt-run state.event pretty=true open on your master:

salt/beacon/ubuntuAsus/inotify//var/www/html/index.html {
    "_stamp": "2020-08-08T18:00:42.793635",
    "change": "IN_MODIFY",
    "id": "ubuntuAsus",
    "path": "/var/www/html/index.html"
}

Notify Reactor

Now we need to react to the change notification by changing the file back to the default state:

nano /etc/salt/master.d/local.conf

Add the salt/beacon/*/inotify//var/www/html/index.html event to Reactor and have it start a script that fixes the issue:

reactor:
  - 'instar/custom/*':
    - /srv/reactor/highstate.sls
  - 'instar/deploy/testapp':
    - /srv/reactor/deploy_testapp.sls
  - 'salt/beacon/*/inotify//var/www/html/index.html':
    - /srv/reactor/fix_welcome.sls

Restart the master systemctl restart salt-master and create the nano /srv/reactor/fix_welcome.sls script:

fix_welcome:
  cmd.state.sls:
    - tgt: {{ data.id }}
    - arg:
      - apache.welcome

Now go and change the /var/www/html/index.html file again. Reactor should automatically fire and change it back to the default state:

salt-run state.event pretty=true                                                                                
salt/auth       {
    "_stamp": "2020-08-08T18:23:40.820210",
    "act": "accept",
    "id": "ubuntuAsus",
    "pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w=AQAB\n-----END PUBLIC KEY-----",
    "result": true
}
salt/beacon/ubuntuAsus/inotify//var/www/html/index.html {
    "_stamp": "2020-08-08T18:23:40.832944",
    "change": "IN_MODIFY",
    "id": "ubuntuAsus",
    "path": "/var/www/html/index.html"
}
20200808182343071200    {
    "_stamp": "2020-08-08T18:23:43.072071",
    "minions": [
        "ubuntuAsus"
    ]
}
salt/job/20200808182343071200/new       {
    "_stamp": "2020-08-08T18:23:43.072389",
    "arg": [
        "apache.welcome"
    ],
    "fun": "state.sls",
    "jid": "20200808182343071200",
    "minions": [
        "ubuntuAsus"
    ],
    "missing": [],
    "tgt": "ubuntuAsus",
    "tgt_type": "glob",
    "user": "root"
}
salt/auth       {
    "_stamp": "2020-08-08T18:23:43.090909",
    "act": "accept",
    "id": "ubuntuMaster",
    "pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w=AQAB\n-----END PUBLIC KEY-----",
    "result": true
}
salt/auth       {
    "_stamp": "2020-08-08T18:23:43.095612",
    "act": "accept",
    "id": "ubuntuAsus",
    "pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w=AQAB\n-----END PUBLIC KEY-----",
    "result": true
}
minion/refresh/ubuntuAsus       {
    "Minion data cache refresh": "ubuntuAsus",
    "_stamp": "2020-08-08T18:23:43.217756"
}
salt/job/20200808182343071200/ret/ubuntuAsus    {
    "_stamp": "2020-08-08T18:23:43.466837",
    "cmd": "_return",
    "fun": "state.sls",
    "fun_args": [
        "apache.welcome"
    ],
    "id": "ubuntuAsus",
    "jid": "20200808182343071200",
    "out": "highstate",
    "retcode": 0,
    "return": {
        "file_|-welcome_page_|-/var/www/html/index.html_|-managed": {
            "__id__": "welcome_page",
            "__run_num__": 1,
            "__sls__": "apache.welcome",
            "changes": {
                "diff": "--- \n+++ \n@@ -1,4 +1,4 @@\n <!doctype html>\n <body>\n-    <h1>Centos Rocks!</h1>\n+    <h1>Ubuntu Rocks.</h1>\n </body>\n"
            },
            "comment": "File /var/www/html/index.html updated",
            "duration": 15.031,
            "name": "/var/www/html/index.html",
            "result": true,
            "start_time": "18:23:43.433832"
        },
        "test_|-check_pillar_values_|-check_pillar_values_|-check_pillar": {
            "__id__": "check_pillar_values",
            "__run_num__": 0,
            "__sls__": "apache.welcome",
            "changes": {},
            "comment": "",
            "duration": 1.504,
            "name": "check_pillar_values",
            "result": true,
            "start_time": "18:23:43.429323"
        }
    },
    "success": true
}
salt/beacon/ubuntuAsus/inotify//var/www/html/index.html {
    "_stamp": "2020-08-08T18:23:43.797938",
    "change": "IN_IGNORED",
    "id": "ubuntuAsus",
    "path": "/var/www/html/index.html"
}
20200808182343808522    {
    "_stamp": "2020-08-08T18:23:43.808797",
    "minions": [
        "ubuntuAsus"
    ]
}
salt/job/20200808182343808522/new       {
    "_stamp": "2020-08-08T18:23:43.809341",
    "arg": [
        "apache.welcome"
    ],
    "fun": "state.sls",
    "jid": "20200808182343808522",
    "minions": [
        "ubuntuAsus"
    ],
    "missing": [],
    "tgt": "ubuntuAsus",
    "tgt_type": "glob",
    "user": "root"
}
minion/refresh/ubuntuAsus       {
    "Minion data cache refresh": "ubuntuAsus",
    "_stamp": "2020-08-08T18:23:43.933767"
}
salt/job/20200808182343808522/ret/ubuntuAsus    {
    "_stamp": "2020-08-08T18:23:44.107148",
    "cmd": "_return",
    "fun": "state.sls",
    "fun_args": [
        "apache.welcome"
    ],
    "id": "ubuntuAsus",
    "jid": "20200808182343808522",
    "out": "highstate",
    "retcode": 0,
    "return": {
        "file_|-welcome_page_|-/var/www/html/index.html_|-managed": {
            "__id__": "welcome_page",
            "__run_num__": 1,
            "__sls__": "apache.welcome",
            "changes": {},
            "comment": "File /var/www/html/index.html is in the correct state",
            "duration": 12.866,
            "name": "/var/www/html/index.html",
            "result": true,
            "start_time": "18:23:44.081579"
        },
        "test_|-check_pillar_values_|-check_pillar_values_|-check_pillar": {
            "__id__": "check_pillar_values",
            "__run_num__": 0,
            "__sls__": "apache.welcome",
            "changes": {},
            "comment": "",
            "duration": 1.119,
            "name": "check_pillar_values",
            "result": true,
            "start_time": "18:23:44.078418"
        }
    },
    "success": true
}

Debugging Pyinotify

File system monitoring through inotify can be interfaced through Python using pyinotify. This guide will demonstrate how to use a Python script to monitor a directory then explore practical uses by incorporating async modules or running additional threads.

Start by installing pip3 and use it to install pyinotify:

sudo apt-get install python3-pip

pip3 install pyinotify

Take this script and copy it to your home directory:

notify_me.py

import os
import pyinotify


class EventProcessor(pyinotify.ProcessEvent):
    _methods = ["IN_CREATE",
                "IN_OPEN",
                "IN_ACCESS",
                "IN_ATTRIB",
                "IN_CLOSE_NOWRITE",
                "IN_CLOSE_WRITE",
                "IN_DELETE",
                "IN_DELETE_SELF",
                "IN_IGNORED",
                "IN_MODIFY",
                "IN_MOVE_SELF",
                "IN_MOVED_FROM",
                "IN_MOVED_TO",
                "IN_Q_OVERFLOW",
                "IN_UNMOUNT",
                "default"]

def process_generator(cls, method):
    def _method_name(self, event):
        print("Method name: process_{}()\n"
               "Path name: {}\n"
               "Event Name: {}\n".format(method, event.pathname, event.maskname))
    _method_name.__name__ = "process_{}".format(method)
    setattr(cls, _method_name.__name__, _method_name)

for method in EventProcessor._methods:
    process_generator(EventProcessor, method)

watch_manager = pyinotify.WatchManager()
event_notifier = pyinotify.Notifier(watch_manager, EventProcessor())

watch_this = os.path.abspath("notification_dir")
watch_manager.add_watch(watch_this, pyinotify.ALL_EVENTS)
event_notifier.loop()

Now Create a folder notification_dir inside your home directory and run the Python script:

python3 notify_me.py

Open another Terminal and create a file inside the folder:

touch ~/notification_dir/test

Switch back to your watch task and see if it recognized the change:

Method name: process_IN_CREATE()
Path name: /root/notification_dir/test
Event Name: IN_CREATE

Method name: process_IN_OPEN()
Path name: /root/notification_dir/test
Event Name: IN_OPEN

Method name: process_IN_ATTRIB()
Path name: /root/notification_dir/test
Event Name: IN_ATTRIB

Method name: process_IN_CLOSE_WRITE()
Path name: /root/notification_dir/test
Event Name: IN_CLOSE_WRITE

This worked! Pyinotify is now ready to go!

Debugging the Reactor

Run the Master in Debug mode and re-run everything to see what is happening:

systemctl stop salt-master
salt-master -l debug