How to Use The Websocket Output Plugin For Elasticsearch
If you’re working with both Logstash and Elasticsearch, you may want to use a websocket in order to secure an uninterrupted connection between Logstash and your Elasticsearch cluster. The Websocket output plugin can allow for real-time communication between the server and client– in this case, between Elasticsearch and Logstash, through a websocket. In this article, we’ll explain how to use the Websocket output plugin for Elasticsearch.
Prerequisites for the Websocket Plugin
Before we can set up the Websocket plugin, we need to make sure we’ve covered all of the system requirements for this task. There are some important prerequisites to keep in mind:
- You must have Java and JDK 8 installed. You can confirm that Java is installed and determine the version with this command:
Elasticsearch must be installed or running from a source directory. You can make an HTTP request using cURL to see if the service is running:
curl -XGET "localhost:9200".
The Logstash service for Elasticsearch must also be installed and running in order for the plugin to work. Visit Elastic’s downloads page for Logstash to get started.
Install Logstash for the Elasticsearch cluster
The next step will be to install Logstash for the Elasticsearch cluster. If you already have Logstash installed, you can skip this section.
Homebrew installation for macOS
If you’re planning to use Homebrew to install Logstash, it’s best to update the package manager first. You can do this and then install Logstash for Elasticsearch with the following commands in a macOS terminal:
brew install logstash
Next, start the service using Homebrew:
brew services start logstash
You’ll see that the Logstash directory path for the Homebrew installation is in the
Installing Logstash from source
Another way to install Logstash is by downloading it straight from the source. Use the
wget command to download the package, or you can simply download it from Elastic’s site. To download Logstash 7.1 with
wget, use the following command in a Linux terminal:
NOTE: It’s also possible to install Wget on macOS using Homebrew (
brew update; brew install wget). If you get a response stating
Cannot write to .., try using the command with
sudo to avoid permissions-related issues.
Once you’ve downloaded Logstash, use the
unzip command to extract the Logstash directory from the archive:
sudo unzip logstash-7.1.0.zip
Installing Logstash on Linux
Next, we’ll install Logstash. For Debian-based Linux distributions, use the APT repository to install Logstash. To do this, first update the repository packages using the
sudo apt-get update
Then install the Logstash package with
sudo apt-get install logstash
For RPM-based distributions, use the
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
sudo yum install logstash
If you do a YUM installation, the configuration file should be found in the
Running Logstash on a
systemd Linux distribution
Once everything is installed correctly, it’s time to run Logstash. On a systemd distribution of Linux, you can use the
sudo systemctl start logstash.service
Running Logstash on a SysV distro of Linux
For SysV distributions, the following command is used instead:
sudo /etc/init.d/logstash start
Verify that the Logstash service is running
The default port for the Logstash service is
9600, so you can verify that Logstash is running by simply navigating to
http://localhost:9600 in a browser. Another option is to use the following cURL request to get a JSON response from the Elasticsearch cluster, which will also verify that Logstash is working:
curl -XGET localhost:9600?pretty=true
You should see the
"version" field returned as part of the JSON response– this is the version of Logstash running on the cluster. Be sure that the macro version of Logstash matches the one for Elasticsearch (i.e. you need to use version 7.x of Logstash with Elasticsearch 7.x) to avoid compatibility issues.
Install the Websocket Plugin using Logstash
If you’ve confirmed that Logstash is successfully installed, you can use this command to have the Logstash service install the Websocket plugin:
bin/elasticsearch-plugin install logstash-output-websocket
Note: The Logstash service doesn’t need to be running in order for the
logstash-plugin install command to work.
Configure Logstash to use the Websocket plugin
If Logstash is currently running, you’ll need to stop the service at this time. This will allow you to restart the service using a new configuration file for the plugin. The command or action you need to execute to stop the service will vary depending on the OS you’re using and how Logstash was installed.
If you’re running Logstash in the foreground from a terminal window, simply press CTRL+C to stop the service.
Create a configuration file for the Websocket pipeline
The next step in the process is to create or edit a configuration file for the Websocket plugin. You can edit the file using
sudo nano websocket-example.conf
NOTE: If you’re running Logstash from an unzipped source directory, you’ll need to create or move the
.conf file to the
Edit the Websocket configuration file settings
Once you’re in the configuration file, change the settings for the websocket. The basic structure of the JSON object in the configuration file looks like this:
Configure the Websocket options in the Logstash configuration file
There are several options to pass within the nested
websocket JSON field in the configuration file. Let’s look at some commonly-used options:
codec option allows you to use different encoding schemas for the websocket. The default value is
id option allows you to pass in a string to be used as the websocket’s plugin ID. If you omit this option, Elasticsearch will automatically generate an ID for the plugin.
Host and Port options
host option is a string representation of the host domain name or IP address for the websocket, while the
port option is an integer expressed without quotation marks (
"). The default values for these options are:
host => "0.0.0.0"
port => 3232
Restart the Logstash service
Now that you’ve created the Logstash configuration file and made the necessary changes, it’s time to restart the Logstash service. Use the
-f option with the
logstash command followed by the path to the configuration file to start the websocket:
bin/logstash -f config/websocket-example.conf
If you get an error stating
Logstash could not be started because there is already another instance, or if the port is in use by a running process, try using
lsof to locate the process. Then, you can use
sudo kill -9 to kill the process using its PID, as shown in the following example:
lsof -n -i4TCP:9600
sudo kill -9 12345
Be sure the path to the configuration file, as specified in the
bin/logstash command, is correct.
Using a websocket can help you maintain a persistent communication between Logstash and Elasticsearch. Data can flow into Elasticsearch from Logstash through a websocket– all you need is the Websocket output plugin for Elasticsearch to make it happen. With the instructions provided in this tutorial, you’ll be ready to install the plugin, make the configuration changes, and get started with an output websocket.
Pilot the ObjectRocket Platform Free!
Try Fully-Managed CockroachDB, Elasticsearch, MongoDB, PostgreSQL (Beta) or Redis.Get Started