Performing an Elasticsearch v8 Upgrade
I want to upgrade a production instance of Elasticsearch / Kibana v7.16.3 to v8.0.0. I am going to use the docker-elk repository as a starting point and add a few little tweaks to it. I will commit the updated version to elk-server-compose.
Modifications
First, I am not going to use Logstash. The service is commented out. If you need to configure Logstash please refer to here.
Open Elasticsearch for a read-only, anonymous search agent and set CORS to allow everything and deactivate the paid XPACK features:
elasticsearch/config/elasticsearch.yml
# xpack.license.self_generated.type: trial
xpack.license.self_generated.type: basic
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true
xpack.security.authc:
anonymous:
username: anonymous_user
roles: search_agent
authz_exception: true
## CORS
http.cors.enabled : true
http.cors.allow-origin: "*"
http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-credentials: true
http.cors.allow-headers: X-Requested-With, X-Auth-Token, Content-Type, Content-Length, Authorization, Access-Control-Allow-Headers, Accept
Explanation: My Elasticsearch instance provides full-text search that is accessed from a variety of clients. I do not want those clients to have to use a user authentication to access the API. And since the request will not only come from a single URL I cannot use CORS to limit access - no matter where the request is coming from the API has to be available. To secure the database I will have to add the search_agent role in Kibana, once the instance is up and running. Users using this role will only have read access to selected indexes.
Blind Dry Run
Clone the docker-elk repository to a test server and change the ELK_VERSION=7.16.3 variable in .env to ELK_VERSION=8.0.0 and see what happens :)
docker-compose build
docker-compose up
The instance is coming up but Elasticsearch is exiting complaining that it needs to be updated to version 7.17.0 before you can go to version 8.0.0:
elasticsearch_1 | java.lang.IllegalStateException: cannot upgrade a node from version [7.16.3] directly to version [8.0.0], upgrade to version [7.17.0] first.
elasticsearch_1 | at org.elasticsearch.env.NodeMetadata.verifyUpgradeToCurrentVersion(NodeMetadata.java:122)
elasticsearch_1 | at org.elasticsearch.env.NodeEnvironment.checkForIndexCompatibility(NodeEnvironment.java:500)
elasticsearch_1 | at org.elasticsearch.env.NodeEnvironment.upgradeLegacyNodeFolders(NodeEnvironment.java:397)
elasticsearch_1 | at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:290)
elasticsearch_1 | at org.elasticsearch.node.Node.<init>(Node.java:388)
elasticsearch_1 | at org.elasticsearch.node.Node.<init>(Node.java:277)
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358)
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)
elasticsearch_1 | at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
elasticsearch_1 | at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
elasticsearch_1 | at org.elasticsearch.cli.Command.main(Command.java:77)
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
elasticsearch_1 | For complete error details, refer to the log at /usr/share/elasticsearch/logs/docker-cluster.lo
wiki_elk-master_elasticsearch_1 exited with code 1
Intermediate Version
So let's change the version to ELK_VERSION=7.17.0 and re-run:
docker-compose build
docker-compose up
Everything is working as expected. I am able to create an index. Now I want to see if I can bring this index into ESv8 without breaking anything. So let's change the version back to ELK_VERSION=8.0.0 and re-run:
docker-compose build
docker-compose up
The instance is coming up again and Elasticsearch is no longer complaining. But I have a new error from Kibana. With version 8 I am no longer allowed to use the default Elasticsearch admin account to connect Kibana to Elasticsearch:
kibana_1 | [2022-02-12T13:46:59.393+00:00][FATAL][root] Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead. Learn more: https://www.elastic.co/guide/en/elasticsearch/reference/8.0/service-accounts.html
kibana_1 | [2022-02-12T13:47:28.906+00:00][FATAL][root] Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead. Learn more: https://www.elastic.co/guide/en/elasticsearch/reference/8.0/service-accounts.html
kibana_1 | at ensureValidConfiguration (/usr/share/kibana/src/core/server/config/ensure_valid_configuration.js:25:11)
kibana_1 | at Server.preboot (/usr/share/kibana/src/core/server/server.js:160:5)
kibana_1 | at Root.preboot (/usr/share/kibana/src/core/server/root/index.js:48:14)
kibana_1 | at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:99:9)
kibana_1 | at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:216:5)
kibana_1 |
kibana_1 | FATAL Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead. Learn more: https://www.elastic.co/guide/en/elasticsearch/reference/8.0/service-accounts.html
wiki_elk-master_kibana_1 exited with code 78
Change the Kibana User
When we are starting the stack for the very first time, we now MUST initialize a password for the built-in kibana_system user to be able to start and access Kibana. First I need to change the user Kibana is configured to use:
kibana/config/kibana.yml
## X-Pack security credentials
#
elasticsearch.username: elastic
elasticsearch.password: 'changeme'
Change the elastic user to kibana_system:
## X-Pack security credentials
#
elasticsearch.username: kibana_system
elasticsearch.password: 'changeme'
Bringing the cluster back up with docker-compose up I can now see that the Error disappeared and was replaced by a warning that the Kibana instances tried to connect but failed:
elasticsearch | {"@timestamp":"2022-02-13T08:12:05.668Z", "log.level": "INFO", "message":"Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[63d89c8e892b][system_critical_read][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","elasticsearch.cluster.uuid":"6rdcLQTFRmWdJlOQasPOHA","elasticsearch.node.id":"2ybJ-D0PRvOHkklDputWqQ","elasticsearch.node.name":"63d89c8e892b","elasticsearch.cluster.name":"docker-cluster"}
Create Random User Logins
Now I can connect to the running Elasticsearch container and generate random passwords for both the elastic and kibana_system user - Copy those passwords and store them somewhere save!
docker-compose exec -T elasticsearch bin/elasticsearch-reset-password --batch --user elastic
WARNING: Owner of file [/usr/share/elasticsearch/config/users] used to be [root], but now is [elasticsearch]
WARNING: Owner of file [/usr/share/elasticsearch/config/users_roles] used to be [root], but now is [elasticsearch]
Password for the [elastic] user successfully reset.
New value: a1hyme+ry1-AltBfpqxY
docker-compose exec -T elasticsearch bin/elasticsearch-reset-password --batch --user kibana_system
Password for the [kibana_system] user successfully reset.
New value: Fug1bNAI3XJW7UWA5Ahm
Replace Kibana User Login
Replace the password of the kibana_system user inside the Kibana configuration file with the password generated in the previous step:
kibana/config/kibana.yml
elasticsearch.username: kibana_system
elasticsearch.password: 'Fug1bNAI3XJW7UWA5Ahm'
And change the ELASTIC_PASSWORD environment variable from the elasticsearch service inside the Compose file docker-compose.yml. But it is only used to initialize the keystore during the initial startup of Elasticsearch, and is ignored on subsequent runs:
ELASTIC_PASSWORD: 'a1hyme+ry1-AltBfpqxY'
And for another try:
docker-compose up
A long and exciting log-scroll later I finally read:
kibana | [2022-02-13T08:19:21.592+00:00][INFO ][plugins.fleet] Fleet setup completed
kibana | [2022-02-13T08:19:21.598+00:00][INFO ][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask
And I am able to access the Kibana web frontend on port 5601 - use the elastic user combined with the generated login. In my case this was a1hyme+ry1-AltBfpqxY:
And to my delight my index was successfully ported to version 8:
Creating an Anonymous User
Here I can add the user role search_agent user role that I added to the elasticsearch.yml and enable the read access to my index:
I can try accessing the cluster using CURL:
curl -XGET 'http://192.168.2.111:9200/wiki_index/_search?pretty=true&q=9020+Full+HD'
{
"took" : 34,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 398,
"relation" : "eq"
},
"max_score" : 12.786632,
"hits" : [ ... ]
Success!