Elasticsearch v8, Filebeat (Docker) and Apache

• Setting up Filebeats
  • Run the Filebeat Setup
  • Configuration
  • Enable and configure Data Collection Modules
    • Prepare the Filebeat Container to Ingest Apache Logs
    • Check the Apache Module

Setting up Filebeats

Start by pulling the a fresh version of Filebeat:

docker pull elastic/filebeat:8.0.0

Run the Filebeat Setup

Running Filebeat with the setup command will create the index pattern and load visualizations , dashboards, and machine learning jobs.

I will create a folder:

mkdir -p /opt/beats/config/

and continue working from there.

Configuration

When running Filebeat in a container, you need to provide access to Docker’s unix socket in order for the add_docker_metadata processor to work. You can do this by mounting the socket inside the container. For example:

docker run -v /var/run/docker.sock:/var/run/docker.sock ...

To avoid privilege issues, you may also need to add --user=root to the docker run flags. Because the user must be part of the docker group in order to access /var/run/docker.sock, root access is required if Filebeat is running as non-root inside the container.

If Docker daemon is restarted the mounted socket will become invalid and metadata will stop working, in these situations there are two options:

• Restart Filebeat every time Docker is restarted
• Mount the entire /var/run directory (instead of just the socket)

I am just going to use CLI flags to mount the docker socket as volumes. This simplifies the configuration to:

nano /opt/beats/config/filebeat.yml

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml # enable all modules (nginx, kafka, redis, etc)
    reload.enabled: false

filebeat.autodiscover: # auto-discover tagged docker container
  providers:
    - type: docker
      hints.enabled: true

setup:
  kibana.host: "http://localhost:5601"
  dashboards.enable: true

output.elasticsearch:
  hosts: 'http://localhost:9200'
  username: 'elastic'
  password: 'changeme'

Note: If you set up Elasticsearch according to this guide, you will have a different elastic user password - e.g. ELASTIC_PASSWORD: 'a1hyme+ry1-AltBfpqxY'.

The beat configuration file must belong to the root user and all write permissions for other users must be revoked:

chown root:root /opt/beats/config/filebeat.yml
chmod go-w /opt/beats/config/filebeat.yml

Enable and configure Data Collection Modules

Prepare the Filebeat Container to Ingest Apache Logs

The Apache logs might be found in the /var/log/apache2 directory - depending on your Apache configuration:

-v /var/log/apache2:/var/log/apache2:ro

And secondly, we need to mount our module configuration file. The template configuration is located inside the Filebeat container under /usr/share/filebeat/modules.d/apache.yml.disabled:

nano /opt/beats/config/apache.yml

# Module: apache
# Docs: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-apache.html

- module: apache
  access:
    enabled: true
    var.paths: ["/var/log/apache2/access.log"]
  error:
    enabled: true
    var.paths: ["/var/log/apache2/error.log"]

save this file under apache.yml next to your filebeat.yml and mount it into the modules.d configuration folder - the complete docker command now looks like this:

docker run -d \
  --name filebeat \
  --restart unless-stopped \
  --user root \
  --net=host \
  -v /opt/beats/config/filebeat.yml:/usr/share/filebeat/filebeat.yml \
  -v /opt/beats/config/apache.yml:/usr/share/filebeat/modules.d/apache.yml \
  -v /var/lib/docker/containers:/var/lib/docker/containers:ro \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /var/log/apache2:/var/log/apache2:ro \
  elastic/filebeat:8.0.0

Restart the container and verify that the logs and module configuration was actually mounted:

docker exec -ti filebeat /bin/bash

/usr/share/filebeat# ls -la /var/log/apache2/
-rwxrwxrwx 1 root root  9634840 Feb 21 03:47 access.log
-rwxrwxrwx 1 root root    12225 Feb 21 03:47 error.log

ls -la /usr/share/filebeat/modules.d | grep apache
-rw-r--r-- 1 root root   613 Feb 22 05:54 apache.yml
-rw-r--r-- 1 root root   788 Feb  3 18:06 apache.yml.disabled

Check the Apache Module

1. Verify that the NGINX modules was actually enable. To see a list of available modules, run:

docker exec -ti filebeat /bin/bash


/usr/share/filebeat# ./filebeat modules list
Enabled:
apache

Disabled:
activemq apache auditd aws awsfargate azure barracuda bluecoat cef checkpoint cisco coredns crowdstrike cyberarkpas cylance elasticsearch envoyproxy f5 fortinet gcp google_workspace haproxy ibmmq icinga iis imperva infoblox iptables juniper kafka kibana logstash microsoft misp mongodb mssql mysql mysqlenterprise nats netflow netscout nginx o365 okta oracle osquery panw pensando postgresql proofpoint rabbitmq radware redis santa snort snyk sonicwall sophos squid suricata system threatintel tomcat traefik zeek zookeeper zoom zscaler

To manually activate or deactivate modules run:

./filebeat modules enable apache

To test your configuration file, change to the directory where the Filebeat binary is installed, and run Filebeat in the foreground with the following options specified:

./filebeat test config -e
Config OK

The documentation I found says that you now should run the setup command to load the available dashboards. I am not sure if this is still necessary since I already did this in the previous step. But running the command returns a Loaded Ingest pipelines - sounds good ~

./filebeat setup -e
Loaded Ingest pipelines