Search Engine Setup and Configuration

• 01 Search Engine Setup and Configuration
  • Installing Elasticsearch 6.x on CentOS
    • Import the Elasticsearch PGP Key
  • Installing from the RPM repository
    • Running Elasticsearch with systemd
    • Checking that Elasticsearch is running
    • Configuring Elasticsearch
  • Installing Kibana 6.x on CentOS
    • Running Kibana with systemd
  • Install X-Pack
    • Elasticsearch Security
    • Kibana Security
  • Enabling Anonymous Access

01 Search Engine Setup and Configuration

Installing Elasticsearch 6.x on CentOS

Elasticsearch is a distributed, JSON-based search and analytics engine designed for horizontal scalability, maximum reliability, and easy management.

Import the Elasticsearch PGP Key

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Installing from the RPM repository

Create a file called elasticsearch.repo in the /etc/yum.repos.d/ directory and add the following lines:

[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

And your repository is ready for use. You can now install Elasticsearch with one of the following commands:

sudo yum install elasticsearch

Running Elasticsearch with systemd

To configure Elasticsearch to start automatically when the system boots up, run the following commands:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Apparently there is no way to quietly reload the Elasticsearch service after changing the config file - you will be required to stop and restart instead:

sudo systemctl stop elasticsearch.service
sudo systemctl start elasticsearch.service

These commands provide no feedback as to whether Elasticsearch was started successfully or not. Instead, this information will be written in the log files located in /var/log/elasticsearch/.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:

curl -XGET 'localhost:9200/?pretty'

http://localhost:9200/_cat/indices?v&pretty

Configuring Elasticsearch

Elasticsearch loads its configuration from the /etc/elasticsearch/elasticsearch.yml file by default. Examples:

• cluster.name: e.g. instar-wiki
• node.name e.g. c21
• node.attr.rack: e.g r44
• path.data: /path/to/data
• path.logs: /path/to/logs
• network.host: localhost see config *
• http.port: 9200
• http.cors: enabled: true , allow-origin: /https?://localhost(:[0-9]+)?/, allow-origin: /https?://localhost(:[0-9][0-9][0-9][0-9])?/
• * e.g. network.host: 127.0.0.1, 192.168.1.200, 7.114.21.49

The RPM places config files, logs, and the data directory in the appropriate locations for an RPM-based system:

| | | | |---|---|---|---| | Type | Description | Default Location | Setting | | home | Elasticsearch home directory or $ES_HOME | /usr/share/elasticsearch | | | bin | Binary scripts including elasticsearch to start a node and elasticsearch-plugin to install plugins | /usr/share/elasticsearch/bin | | | conf | Configuration files including elasticsearch.yml | /etc/elasticsearch | ES_PATH_CONF | | conf | Environment variables including heap size, file descriptors. | /etc/sysconfig/elasticsearch | | | data | The location of the data files of each index / shard allocated on the node. Can hold multiple locations. | /var/lib/elasticsearch | path.data | | logs | Log files location. | /var/log/elasticsearch | path.logs | | plugins | Plugin files location. Each plugin will be contained in a subdirectory. | /usr/share/elasticsearch/plugins | |

Installing Kibana 6.x on CentOS

Kibana gives shape to your data and is the extensible user interface for configuring and managing all aspects of the Elastic Stack.

Create a file called kibana.repo in the /etc/yum.repos.d/ directory and add the following lines:

[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

And your repository is ready for use. You can now install Kibana with one of the following command:

sudo yum install kibana

Running Kibana with systemd

To configure Kibana to start automatically when the system boots up, run the following commands:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable kibana.service

Kibana can be started and stopped as follows:

sudo systemctl stop kibana.service
sudo systemctl start kibana.service

These commands provide no feedback as to whether Kibana was started successfully or not. Instead, this information will be written in the log files located in /var/log/kibana/. Kibana loads its configuration from the /etc/kibana/kibana.yml file by default. Examples:

• elasticsearch.url: Default: http://localhost:9200 The URL of the Elasticsearch instance to use for all your queries.
• server.port: Server port for the Kibana web UI - default 5601
• server.host: Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. The default is localhost, which usually means remote machines will not be able to connect. To allow connections from remote users, set this parameter to a non-loopback address.
• console.enabled: Default: true Set to false to disable Console.
• elasticsearch.username: s. below
• elasticsearch.password: If your Elasticsearch is protected with basic authentication, these settings provide the username and password that the Kibana server uses to perform maintenance on the Kibana index at startup. Your Kibana users still need to authenticate with Elasticsearch, which is proxied through the Kibana server. (see X-Pack below)
• server.ssl.enabled: Default: 'false' Enables SSL for outgoing requests from the Kibana server to the browser. When set to true, server.ssl.certificate and server.ssl.key are required
• server.ssl.certificate: s. below
• server.ssl.key: Paths to the PEM-format SSL certificate and SSL key files, respectively.
• server.ssl.certificateAuthorities: List of paths to PEM encoded certificate files that should be trusted.
• server.ssl.cipherSuites: Default: ECDHE-RSA-AES128-GCM-SHA256, ECDHE-ECDSA-AES128-GCM-SHA256, ECDHE-RSA-AES256-GCM-SHA384, ECDHE-ECDSA-AES256-GCM-SHA384, DHE-RSA-AES128-GCM-SHA256, ECDHE-RSA-AES128-SHA256, DHE-RSA-AES128-SHA256, ECDHE-RSA-AES256-SHA384, DHE-RSA-AES256-SHA384, ECDHE-RSA-AES256-SHA256, DHE-RSA-AES256-SHA256, HIGH,!aNULL, !eNULL, !EXPORT, !DES, !RC4, !MD5, !PSK, !SRP, !CAMELLIA. Details on the format, and the valid options, are available via the OpenSSL cipher list format documentation
• server.ssl.keyPassphrase: The passphrase that will be used to decrypt the private key. This value is optional as the key may not be encrypted.
• server.ssl.redirectHttpFromPort: Kibana will bind to this port and redirect all http requests to https over the port configured as server.port.
• server.ssl.supportedProtocols: Default: TLSv1, TLSv1.1, TLSv1.2 Supported protocols with versions. Valid protocols: TLSv1, TLSv1.1, TLSv1.2
• status.allowAnonymous: Default: false If authentication is enabled, setting this to true allows unauthenticated users to access the Kibana server status API and status page.

Type	Description	Default Location
home	Kibana home directory or $KIBANA_HOME	/usr/share/kibana
bin	Binary scripts including kibana to start the Kibana server and kibana-plugin to install plugins	/usr/share/kibana/bin
config	Configuration files including kibana.yml	/etc/kibana
data	The location of the data files written to disk by Kibana and its plugins	/var/lib/kibana
optimize	Transpiled source code. Certain administrative actions (e.g. plugin install) result in the source code being retranspiled on the fly.	/usr/share/kibana/optimize
plugins	Plugin files location. Each plugin will be contained in a subdirectory.	/usr/share/kibana/plugins

Install X-Pack

X-Pack is a single extension that integrates handy features — security, alerting, monitoring, reporting, graph exploration, and machine learning — you can trust across the Elastic Stack.

Elasticsearch Security

We need to add a user athentication to our Elasticsearch / Kibana setup. We will do this by installing X-Pack. To get started with installing the Elasticsearch plugin, go to /etc/elasticsearch/ and call the following function:

bin/elasticsearch-plugin install x-pack

Now restart Elasticsearch:

sudo systemctl stop elasticsearch.service
sudo systemctl start elasticsearch.service

You can either use the auto function to generate user passwords for Elasticsearch, Kibana (and the not yet installed Logstash):

bin/x-pack/setup-passwords auto

or swap the auto flag with interactive to use your own user logins. The auto output will look something like this:

Changed password for user kibana 
PASSWORD kibana = *&$*(80gfddzg

Changed password for user logstash_system
PASSWORD logstash_system = 58#$)Qljfksh

Changed password for user elastic
PASSWORD elastic = jgfisg)#*%&(@*#)

Now every interaction with Elasticsearch or Kibana will require you to authenticate with username: elastic and password: jgfisg)#%&(@#)

Kibana Security

Now we repeat these steps with Kibana. First navigate to /etc/kibana/ and call the following function:

bin/kibana-plugin install x-pack

And we have to add the login that Kibana has to use to access Elasticsearch (auto generated above) to the kibana.yml file in /etc/kibana/:

elasticsearch.username: 'kibana'
elasticsearch.password:  '*&$*(80gfddzg'

Now restart Kibana:

sudo systemctl stop kibana.service
sudo systemctl start kibana.service

Now navigate your browser http://localhost:5601/ and login with the 'elastic' user we generated above.

Enabling Anonymous Access

Incoming requests are considered to be anonymous if no authentication token can be extracted from the incoming request. By default, anonymous requests are rejected and an authentication error is returned (status code 401). To allow anonymous user to send search queries (Read access to specified indices), we need to add the following lines to the elasticsearch.yml file in /etc/elasticsearch/:

xpack.security.authc:
  anonymous:
    username: anonymous_user 
    roles: wiki_reader 
    authz_exception: true 

Now we have to switch to the Kibana webUI on http://localhost:5601/ and create the role: wiki_reader to allow read access to the wiki indices. First switch to the Management tab and click on user:

Then click on Add a User and add a user with the watcher_user role:

Switch back to the Management tab and click on role:

Click on Create Role and add the name wiki_reader that we choose for the role of the anonymous user inside the elasticsearch.yml file, assign the monitor_watcher privilege and choose the indices that you want the anonymous user to have READ access to:

Your configuration will be active after restarting Elasticsearch. Now you can use webservices to read from your ES database. But only the elastic user has the privileg to WRITE and to work in Kibana.