How To Use Python's JSON Library In Elasticsearch

Introduction

If you’re working with Python and Elasticsearch, it’s important to make sure you’ve lined up all the tools you need to get the job done efficiently. One key tool is Python’s JSON library. This built-in JSON library seamlessly converts Python dict (dictionary) type objects into JSON strings and vice versa, making it easy to pass data to Elasticsearch in the correct format. In this article, we’ll show you how to use the Python JSON library with Elasticsearch in your scripts. We’ll use Python to declare an Elasticsearch document as a Python dictionary. then convert that dictionary into a JSON string that can be passed to the Elasticsearch client’s API method calls.

Prerequisites

Let’s take a quick look at some key prerequisites that need to be taken care of before we can proceed with our task:

• Make sure Python 3 is installed and working. Although Python 2.7 may still work for the code example in this article, this version of Python is deprecated and will ultimately lose support.

• The PIP3 package manager for Python also needs to be installed on the Elasticsearch cluster that’s running the Python script. We’ll be using PIP3 to install the Elasticsearch client for Python:

• You’ll need to have some basic knowledge of Python and its syntax. We’ll cover some of Python’s keywords and functions, and the terminology and concepts may be confusing to a beginner Pythonista.

Create a Python script in the Elasticsearch project directory

Now that we’ve covered the prerequisites, let’s start working on our Python script. Navigate to the directory for your Elasticsearch project (use mkdir to create a new directory if needed):

Use the touch command to create a Python script:

The new Python file should now be in your project directory. You can use a terminal-based editor such as vim, nano, or gedit; however, it’s best to use an IDE that supports Python indentation and syntax locally. This will help you avoid frustrating syntax or indentation errors.

Import the Python package libraries for JSON and Elasticsearch

The first thing we’ll do in our script is import the necessary libraries to avoid receiving an ImportError when calling a library’s attribute or method. Here are the libraries we’ll need:

NOTE: The json library is native to Python 3; therefore, it doesn’t need to be installed with pip3. However, it still needs to be imported before you’re able to use it.

Get familiar with Python’s JSON library before using it with Elasticsearch

Before digging in to the code and trying to use the Python JSON library with Elasticsearch, it’s a good idea to play around with the JSON library and check out its attributes. You can type python3 in a terminal to use a Python interpreter or idle3 to use the IDLE Python environment. Two methods to take a closer look at are loads() and dumps():

Python’s dir() function can be very useful for getting more attribute information about a particular package, library or method in Python.

When you call the json.dumps() method, it returns a JSON string of whatever Python dictionary was passed to it, but it will throw a ValueError if the JSON object is not valid. This is a great way to catch potential JSON errors before the data gets passed to Elasticsearch. Conversely, the json.loads() method will take a valid JSON string and return a Python dictionary.

Catch JSON errors using a try-except error handling block in Python

You can use Python to catch errors using its try-except indentation block. Let’s check out an example of an invalid JSON string being passed to json.loads(). This will raise a ValueError inside of the try-except block:

Build an Elasticsearch JSON document with key-value pairs in a Python dictionary

Much like a JSON object, a Python dictionary is simply a set of key-value pairs. One dictionary can also be nested inside another dictionary value. Let’s look at an example of how we can build a nested Elasticsearch document in Python. In our example, we’ll build this nested document by passing the document’s _source data, in the form of a Python dict object, to the value of another dictionary’s object’s key. If this sounds a bit confusing, there’s no need to worry– it will become much clearer as we look at the code.

Declare a Python dictionary for the Elasticsearch document’s _source data

Notice that this example uses several different data types for its document fields and uses Python’s datetime library to create a valid timestamp:

NOTE: The datetime timestamp must be explicitly cast as a string; if not, the JSON library’s dumps() method will throw a TypeError:

Declare the Elasticsearch Python dictionary by nesting the _source data inside it

Create a JSON string of the Python dictionary using json.loads()

You can use the indent parameter to specify the number of spaces for each indentation in the JSON object. This serves to “prettify” the string, making it easier to read:

Build an Elasticsearch JSON document from a simple Python dictionary

At this point, we’re ready to prepare the JSON document that we’ll be passing to the Elasticsearch client. This document will be built from our Python dictionary object:

Compare the two JSON document dictionary objects in Python

Create a JSON string of the Elasticsearch document dictionary and make an API call

Here, we create our JSON string using json.dumps() and make sure it doesn’t raise any errors:

Finally, we make an API call to index this document in Elasticsearch:

Create a JSON Elasticsearch query using a Python dictionary

Pass the Python dictionary

Conclusion

It’s clear that using the Python JSON library with Elasticsearch can make your development tasks easier and more efficient. Now that we’ve finished creating our script, all we need to do is run it. You can run the Python script using the python3 command followed by the file name in a terminal or command prompt window. Be sure to specify the path or navigate to the location of the Python script when you execute the script.

At the end of your terminal output, you should see something like the following: