Linux

How to Set Up an Elasticsearch cluster with Multiple Nodes

How to Set Up an Elasticsearch cluster with Multiple Nodes

How to set up an Elasticsearch cluster with Google Cloud Multiple Nodes.

Elasticsearch Clusters are groups of nodes with the same cluster names and additional Elasticsearch servers join and communicate with each other.

In this guide you will learn how to configure multiple virtual machines in Google Cloud with Elasticsearch and make it a cluster for communicating with each other.

This setup was tested on Google Cloud with several machines connected to each other as a cluster for Elasticsearch. So this setting will work on other cloud service providers like AWS or other Azure.

Precondition

After installing Elasticsearch, you can follow the steps below to configure.

You can also set Nginx-based settings on the main Elasticsearch server or instance.

Make sure you use the Internal IP in the hostname in the Elasticsearch configuration.

Configure Firewall

By default, the Elasticsearch service listens on port 9200 and the nodes in a cluster communicate on port 9300. So you need to allow ports 9200 and 9300 on your network.

If you use UFW, you need to allow this port.

sudo ufw allow 9200/tcp
sudo ufw allow 9300/tcp

On Google Cloud you must create firewall rules to allow this port.

Open VPC Network >> Firewall rules and click Create Firewall rules.

In Name, enter elasticsearch

In Target select All instances in the network

In the Source filter select IP range

In the IP Source range enter 0.0.0.0 / 0

In Protocols and ports, check TCP and enter 9200, 9300

Create Nodes for the Elasticsearch Cluster

To create additional nodes, you need to set up an additional server or instance and install Elasticsearch.

You need to install the version of Elasticsearch that is installed on the main server.

To check the version of Elasticsearch in the main instance, SSH to the instance and run the following command.

curl -XGET 'http://INTERNAL_IP:9200'

You will get an output that shows you what version is installed.

To install a specific version you can follow the steps below after installing and configuring Java.

sudo apt-get -y install apt-transport-https curl wget
sudo wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
sudo echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list
sudo apt-get update && apt-get install elasticsearch=7.1.1

After you install the correct version of Elasticsearch on all your instances / servers, you can continue to configure it to communicate with each other.

Change the Elasticsearch Configuration for Clusters

SSH to your main Elasticsearch server and make the following changes.

sudo nano /etc/elasticsearch/elasticsearch.yml

Add the following to your configuration.

cluster.name: CLUSTER_NAME
node.name: ${HOSTNAME}
node.master: true
node.data: true
discovery.zen.ping.unicast.hosts: ["INTERNAL_IP_SERVER_1", "INTERNAL_IP_SERVER_2"]
discovery.zen.minimum_master_nodes: 2

Change CLUSTER_NAME to the name you want.

If you want to name the node, you can also enter any unique name.

Include all internal IP addresses from all other servers at discovery.zen.ping.unicast.hosts

After you update the configuration, restart Elasticsearch.

sudo service elasticsearch restart

Now, SSH for additional clusters and edit the elasticsearch.yml file and make the same updates, and restart the service.

Check the State Cluster

If everything is configured correctly, your cluster must be active and running.

You can check the status using the following command.

{
   "cluster_name" : "cluster_name",
   "cluster_uuid" : "tDRxIHICTQewxqvRuN8QvQ",
   "version" : 1433,
   "state_uuid" : "hcFyhKijQtmkjSxKMe0AfQ",
   "master_node" : "Tg4JuAtLSQKnAZYVUxOoLg",
   "blocks" : { },
   "nodes" : {
     "Tg4JuAtLSQKnAZYVUxOoLg" : {
       "name" : "main-node",
       "ephemeral_id" : "x3HLaF-HSWWtljujhwXlxQ",
       "transport_address" : "INTERNAL_IP:9300",
       "attributes" : {
         "ml.machine_memory" : "15734808576",
         "xpack.installed" : "true",
         "ml.max_open_jobs" : "20",
         "ml.enabled" : "true"
       }
     },
     "s-EoZV4KQxO5YUhqYzv3yA" : {
       "name" : "node-1",
       "ephemeral_id" : "S3SGzhSiSJmJR-Z65A7e_Q",
       "transport_address" : "INTERNAL_IP:9300",
       "attributes" : {
         "ml.machine_memory" : "3862560768",
         "ml.max_open_jobs" : "20",
         "xpack.installed" : "true",
         "ml.enabled" : "true"
       }
     }
   },
   . . .

If you see output like this, your cluster is functioning properly.

Configure the Dedicated Master Node

Before configuring a specific master node, ensure that your cluster has at least 3 nodes that meet the master requirements. This is important to avoid split-brain situations, which can cause inconsistencies in your data if a network failure occurs.

sudo nano /etc/elasticsearch/elasticsearch.yml

Change the following lines to match the following.

node.master: true 
node.data: false

Restart Elasticsearch for the changes to take effect.

Configure Custom Data Nodes

To configure specific data nodes that do not meet the master requirements, edit the Elasticsearch configuration for each node and update the configuration to match the settings below.

node.master: false 
node.data: true

Restart Elasticsearch for the changes to take effect.

Configure the Minimum Master Node

It is important to specify the minimum number of masters that meet the requirements that need to be run so that the cluster is functioning normally, which is sometimes referred to as a quorum.

To calculate the minimum number of master nodes your cluster must have, count n / 2 + 1, where n is the total number of “master-eligible” nodes in your healthy cluster, then round the result to the nearest integer.

For example, for a 3-node cluster, the quorum is 2.

Now you have configured your Elasticsearch cluster with some basic optimizations and healthy running.

Conclusion

Now you have learned how to setup Elasticsearch and configure clusters in production-based settings.

Thank you for your time. If you encounter a problem or feedback, please leave a comment below.

Related posts

How to run Linux on an Android device

Howto

How to install Webmin and secure it with Let’s Encrypt SSL on Ubuntu 20.04 LTS

Linux

How Linux makes data recovery easy

Linux

How to Upgrade to Ubuntu 20.04

Linux

How to Install Apache on Ubuntu 20.04

Linux

How to Install PostgreSQL on Ubuntu 20.04

Linux

How to Install Nextcloud with Nginx and Let’s Encrypt SSL on Ubuntu 20.04 LTS

Linux

What is Fedora Linux?

InstallWorld

Linux Pinky Command Tutorial for Beginners (8 Examples)

Linux