Hey, this is Minh from WiseCode. I have been working with the Elastic Stack for around 1 year. And in the beginning, I find it very difficult to start with. That's why I create this series to help you guys get started with Elasticsearch the easier way.
In this series, you will learn how to install and configure Elastic Stack, how to manipulate data in Elasticsearch, how to use Kibana to visualize data and create dashboards, how to use Logstash to ingest logs, and more.
By the end of this series, I hope you will have basic hands-on experience in working with Elasticsearch and the Elastic Stack. So let's get started!
Table of Contents
What is the Elastic Stack?
The Elastic Stack comprises four main products: Elasticsearch, Kibana, Beats, and Logstash. Those are open-source products developed by the Elastic team, however, there are also enterprise-grade features you will have to pay. However, in the scope of this series, we won't have to spend a penny.
In short:
- Elasticsearch: a search engine
- Kibana: a data visualizer
- Logstash: a data processor
- Beats: a data shipper
What is Elasticsearch?
Just for a brief introduction, Elasticsearch is a super quick search engine based on Apache Lucene (you don't have to care about this for now). It provides a distributed and scalable full-text search engine with an HTTP web interface (which means you can interact with it just like REST API) and schema-free JSON documents.
Why I say it is super quick? Because it is built on top of Apache Lucene, which is a high-performance text search engine library. It is also distributed and scalable, which means you can easily scale it up to handle a very very large amount of data.
If you want to know more about Elasticsearch, you can check out the official website or their story.
I keep saying that Elasticsearch is a search engine, not a database. But to perform quick search queries, it has to store the data in its own database. Don't be mistaken that Elasticsearch is a database.
What is Kibana?
When you have the data, how do you want to see it?
Yes, that's the visualization part when Kibana comes in. Kibana is a data visualization tool that allows you to create dashboards, graphs, and charts to visualize your data. It also provides a web interface to interact with Elasticsearch.
In general, Kibana is the frontend of the Elastic Stack, in which you can use it to interact with Elasticsearch.
What is Logstash?
Logstash is a data processor that allows you to ingest data from multiple sources, transform it, and then send it to Elasticsearch.
In simple words, you can use Logstash to extract the information you need from the raw data.
What are Beats?
Beats is a data shipper that allows you to send data from hundreds or thousands of machines to Logstash or Elasticsearch.
It has many different types of Beats, such as Filebeat, Metricbeat, Packetbeat, etc. Each type will be used to ship a specific type of data.
That also means, it has many integrations with other services, such as databases (MySQL, MongoDB, PostgreSQL, etc.), web servers (Apache, Nginx, etc.), cloud services (AWS, GCP, etc.), and more.
Install the Elastic Stack
Let's go to the main part of this post, which is installing the Elastic Stack. However, in the scope of this series, we will only install Elasticsearch, Kibana and Logstash.
I am trying to cover the installation part as much as possible since tutorials on the Internet doesn't work for me, even the official docs from Elastic. So, trust me, I will make it work for you.
Also, there is another way to use Elastic Stack without installing that is to use the Elastic Cloud. However, it only offers a 14-day free trial, then you will have to pay for it. So, I will not cover that in this series.
Prerequisites
Before we start, you will need to have the following:
- A device with at least 8GB of RAM (my recommendation), I am using a 16GB RAM Debian-based laptop.
Install Elasticsearch
I highly recommend using the tar.gz package for Linux and MacOS, and the zip package for Windows. For those packages, Java is already included so you won't have to install it separately.
Follow these steps to install Elasticsearch:
- Go to the Elasticsearch download page and download the package for your OS. In my case, I choose the tar.gz package.
- For Linux and MacOS users, you will be redirected to this download page.
- Open Terminal, type the following command to download the package:
shell
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.11.1-linux-x86_64.tar.gz - After downloading, extract the package:
shell
tar -xzf elasticsearch-8.11.1-linux-x86_64.tar.gz - I usually move it to a folder for easier management:
shell
mv elasticsearch-8.11.1 ~/elastic
Now, Elasticsearch will be installed at ~/elastic/elasticsearch-8.11.1. But don't run it for now, I will show you how to set the heap size for Elasticsearch so that it won't eat up all your RAM.
Go to the ~/elastic/elasticsearch-8.11.1/config folder, open the jvm.options file, and change the following lines:
# ... -Xms2g -Xmx2g # ...
Remember to save the file.
All good, let's run Elasticsearch:
cd ~/elastic/elasticsearch-8.11.1 ./bin/elasticsearch
If you see something like the above image, then Elasticsearch is running successfully.
You will need to keep those information from the Terminal for further usage:
- Password for the elastic user.
- Enrollment token.
If you forgot to save those information, don't worry as you can reset or create new ones later.
Now, open your web browser and go to https://localhost:9200, where 9200 is the default port of Elasticsearch.
If it requires you to enter the username and password, put in elastic for the username, and the password from the Terminal for the password.
For example, in my case, the password is MnaHNcir2INs5BtwC-Nl. So I will put in elastic for the username, and MnaHNcir2INs5BtwC-Nl for the password.
If Elasticsearch is working properly, you will see the cluster information.
Ok cool, Elasticsearch is installed successfully. But you might ask, what's next, what will we do with the text in the image above?
Well, it's time for Kibana to come in. Let's install it.
I recommend to install Kibana immediately after you install Elasticsearch, as you won't have to deal with certificate expiration issues.
Install Kibana
Similarly, let's use the tar.gz package for Linux and MacOS, and the zip package for Windows.
- Go to the Kibana download page. I choose the tar.gz package.
- As I am using Linux, it will redirect me to this download page.
- Open Terminal, type the following command to download the package:
shell
wget https://artifacts.elastic.co/downloads/kibana/kibana-8.11.1-linux-x86_64.tar.gz - Extract the package:
shell
tar -xzf kibana-8.11.1-linux-x86_64.tar.gz - Move it to the same folder with Elasticsearch:
shell
mv kibana-8.11.1 ~/elastic
Cool, Kibana is installed at ~/elastic/kibana-8.11.1. Let's start it:
cd ~/elastic/kibana-8.11.1 ./bin/kibana
Copy the URL from the Terminal (http://localhost:5601/?code=732963), then open it in your web browser.
The first time you run Kibana, it will require you to configure Elastic.
Remember that you must run Elasticsearch from the previous section, if not, start running it now using the above instructions.
We need to connect Kibana to Elasticsearch using the enrollment token, and I already said that we will need to store the enrollment token for further usage.
If you didn't do that, don't worry, let's create a new one.
cd ~/elastic/elasticsearch-8.11.1 ./bin/elasticsearch-create-enrollment-token -s kibana
A new enrollment token is created.
Let's paste it into Kibana, then click Configure Elastic.
You will get a screen like this.
Wait a few seconds, then you will be redirected to the Kibana homepage.
If you are waiting for more than 5 minutes, please reload the page or go to
http://localhost:5601.
Cool, that's all for Kibana installation.
Install Logstash
For Logstash, you can download the tar.gz package directly without using commmand.
- Visit the Logstash download page.
- Choose your platform, I choose Linux x86_64 to get the tar.gz file.
- I assume the file is downloaded to
~/Downloads, so let's extract it:shellcd ~/Downloads tar -xzf logstash-8.11.1-linux-x86_64.tar.gz - Now move it to the same folder with Elasticsearch and Kibana:
shell
mv logstash-8.11.1 ~/elastic
Logstash is installed at ~/elastic/logstash-8.11.1. But as we don't have any data, so leave it for now.
Conclusion
Congratulations, you have installed the Elastic Stack successfully.
See you again in the next post.
