How to Use The Websocket Output Plugin For Elasticsearch

Introduction

If you’re working with both Logstash and Elasticsearch, you may want to use a websocket in order to secure an uninterrupted connection between Logstash and your Elasticsearch cluster. The Websocket output plugin can allow for real-time communication between the server and client– in this case, between Elasticsearch and Logstash, through a websocket. In this article, we’ll explain how to use the Websocket output plugin for Elasticsearch.

Prerequisites for the Websocket Plugin

Before we can set up the Websocket plugin, we need to make sure we’ve covered all of the system requirements for this task. There are some important prerequisites to keep in mind:

• You must have Java and JDK 8 installed. You can confirm that Java is installed and determine the version with this command:

• Elasticsearch must be installed or running from a source directory. You can make an HTTP request using cURL to see if the service is running: curl -XGET "localhost:9200".

• The Logstash service for Elasticsearch must also be installed and running in order for the plugin to work. Visit Elastic’s downloads page for Logstash to get started.

Install Logstash for the Elasticsearch cluster

The next step will be to install Logstash for the Elasticsearch cluster. If you already have Logstash installed, you can skip this section.

Homebrew installation for macOS

If you’re planning to use Homebrew to install Logstash, it’s best to update the package manager first. You can do this and then install Logstash for Elasticsearch with the following commands in a macOS terminal:

Next, start the service using Homebrew:

You’ll see that the Logstash directory path for the Homebrew installation is in the Cellar directory:

Installing Logstash from source

Another way to install Logstash is by downloading it straight from the source. Use the wget command to download the package, or you can simply download it from Elastic’s site. To download Logstash 7.1 with wget, use the following command in a Linux terminal:

NOTE: It’s also possible to install Wget on macOS using Homebrew (brew update; brew install wget). If you get a response stating Cannot write to .., try using the command with sudo to avoid permissions-related issues.

Once you’ve downloaded Logstash, use the unzip command to extract the Logstash directory from the archive:

Installing Logstash on Linux

Next, we’ll install Logstash. For Debian-based Linux distributions, use the APT repository to install Logstash. To do this, first update the repository packages using the apt-get command:

Then install the Logstash package with apt-get:

For RPM-based distributions, use the yum command:

If you do a YUM installation, the configuration file should be found in the /etc/yum.repos.d/ directory.

Running Logstash on a systemd Linux distribution

Once everything is installed correctly, it’s time to run Logstash. On a systemd distribution of Linux, you can use the systemctl command:

Running Logstash on a SysV distro of Linux

For SysV distributions, the following command is used instead:

Verify that the Logstash service is running

The default port for the Logstash service is 9600, so you can verify that Logstash is running by simply navigating to http://localhost:9600 in a browser. Another option is to use the following cURL request to get a JSON response from the Elasticsearch cluster, which will also verify that Logstash is working:

You should see the "version" field returned as part of the JSON response– this is the version of Logstash running on the cluster. Be sure that the macro version of Logstash matches the one for Elasticsearch (i.e. you need to use version 7.x of Logstash with Elasticsearch 7.x) to avoid compatibility issues.

Install the Websocket Plugin using Logstash

If you’ve confirmed that Logstash is successfully installed, you can use this command to have the Logstash service install the Websocket plugin:

Note: The Logstash service doesn’t need to be running in order for the logstash-plugin install command to work.

Configure Logstash to use the Websocket plugin

If Logstash is currently running, you’ll need to stop the service at this time. This will allow you to restart the service using a new configuration file for the plugin. The command or action you need to execute to stop the service will vary depending on the OS you’re using and how Logstash was installed.

If you’re running Logstash in the foreground from a terminal window, simply press CTRL+C to stop the service.

Create a configuration file for the Websocket pipeline

The next step in the process is to create or edit a configuration file for the Websocket plugin. You can edit the file using nano:

NOTE: If you’re running Logstash from an unzipped source directory, you’ll need to create or move the .conf file to the logstash/config directory.

Edit the Websocket configuration file settings

Once you’re in the configuration file, change the settings for the websocket. The basic structure of the JSON object in the configuration file looks like this:

Configure the Websocket options in the Logstash configuration file

There are several options to pass within the nested websocket JSON field in the configuration file. Let’s look at some commonly-used options:

Codec option

The codec option allows you to use different encoding schemas for the websocket. The default value is plain.

ID option

The id option allows you to pass in a string to be used as the websocket’s plugin ID. If you omit this option, Elasticsearch will automatically generate an ID for the plugin.

Host and Port options

The host option is a string representation of the host domain name or IP address for the websocket, while the port option is an integer expressed without quotation marks ("). The default values for these options are:

Restart the Logstash service

Now that you’ve created the Logstash configuration file and made the necessary changes, it’s time to restart the Logstash service. Use the -f option with the logstash command followed by the path to the configuration file to start the websocket:

If you get an error stating Logstash could not be started because there is already another instance, or if the port is in use by a running process, try using lsof to locate the process. Then, you can use sudo kill -9 to kill the process using its PID, as shown in the following example:

Be sure the path to the configuration file, as specified in the bin/logstash command, is correct.

Conclusion

Using a websocket can help you maintain a persistent communication between Logstash and Elasticsearch. Data can flow into Elasticsearch from Logstash through a websocket– all you need is the Websocket output plugin for Elasticsearch to make it happen. With the instructions provided in this tutorial, you’ll be ready to install the plugin, make the configuration changes, and get started with an output websocket.