Elasticsearch and Scroll in Python

Introduction

Scrolling in Elasticsearch allows you retrieve a large number of documents, in steps or iterations, similar to pagination or a “cursor” in relational databases. In this article we’ll explore the Elasticsearch concept of scrolling, and how we can implement it in an application using the Python low-level client’s “helpers” library.

Let’s go over how to get documents from Elasticsearch with Scroll and Python.

Prerequisites

You’ll need to install the Elasticsearch service and start the cluster on your machine or server. Make sure that you have admin or sudo privileges so that you can install Python 3 or Elasticsearch if necessary.

Install Elasticsearch on Linux, Windows, and macOS

On a Linux distro that uses systemd you’ll have to download and install the archive and then use the systemctl utility to enable or start the service. Otherwise you can download the MSI installer for Windows, or use Homebrew’s brew command to tap and install the elastic repository on macOS.

Check that the Elasticsearch cluster is running

You should now be able to get a JSON response from your Elasticsearch cluster running on the default port of 9200. Navigate to localhost:9200 in a browser tab, if you’re developing locally, or use the following cURL request in a terminal or command prompt window to verify that the cluster is running:

The code used in this article has been designed and tested with Python version 3 in mind. The easiest way to install Python 3 modules is with the PIP package manager (pip3). Use the pip3 -V command to verify that the package manager is installed and working, and then use the following command to install the Elasticsearch client for Python:

Elasticsearch documents for the Scroll API

You’ll also need an index on your Elasticsearch cluster with a massive number of documents on it that you can use to test Elasticsearch’s Scroll API in Python. Check out our article about bulk indexing Elasticsearch documents in Python for more information.

Elasticsearch Scroll API

In a cURL or Kibana request you’d use an HTTP POST request to create a new index for that particular Elasticsearch scroll. The following is an example of such a request made in the Kibana Console UI and it should return the scroll’s "scroll_id" in the right panel:

NOTE: The 3m value in the above HTTP request is the time value that you’d like Elasticsearch to scroll for. You can use m for milliseconds and s for seconds, and, depending on the size of the documents and the overall index, a few milliseconds typically suffices.

Then all you have to do is make another HTTP request using the scroll ID, and this time you can use GET or POST, and you should omit the index name since it will be stored in the scroll index itself:

NOTE: The scroll ID will change if you make another scroll POST request with different parameters.

Scroll API in Python

There are three different ways to scroll Elasticsearch documents using the Python client library—using the client’s search() method, the helpers library’s scan() method, or the client’s scroll() method.

Scrolling Elasticsearch documents in Python

You can execute a regular search query that’s limited in size and has a scrolling time limit passed to it, or you can also use the client’s low-level scroll() method designed to work with Elastic’s Scroll API. The third option is to use the client’s helpers.scan() which works in a similar fashion to scroll().

Import the Python modules for Elasticsearch

The following Python code will import the Elasticsearch client and some of its libraries into the Python script:

Connect to Elasticsearch in Python

Once you’ve concatenated the string for the domain host you can pass it to the Elasticsearch() method and it should return a valid client instance connected to the cluster:

You can use the following code to validate the connection to the cluster:

If you’re host string is correct, and if the Elasticsearch cluster is running properly, then the above code should print the cluster information, otherwise it will set the client instance to None.

Scrolling Elasticsearch documents with search()

Now let’s learn about using the client’s search() method to scroll through Elasticsearch documents. The first step is to create a JSON object (using a dict object in Python) with the search size and query Elasticsearch fields for the dictionary keys: cURL

The above dictionary example will match all of the index’s documents to provide enough data for scrolling, and it will return just 42 documents.

Call the Elasticsearch client’s search() method

Now pass the search_body dictionary declared above to the client instance’s search() method and make sure to specify the index name as a parameter:

The above code will scroll for just 3 milliseconds. Make sure to increase that time for larger documents, or for a scroll procedure returning more documents.

We can now get the scroll ID from the response by accessing its _scroll_id key:

Scrolling Elasticsearch documecURL nts with scroll()

The second option is to use the client’s scroll() method, and now that we have a scroll ID we can pass it to its method call to continue the query:

The method’s parameters are body, rest_total_hits_as_int, scroll, and scroll_id. Unlike the helper library’s scan() method, scroll() does not accept a size parameter, but the optional scroll ID parameter should come in handy.

NOTE: The Boolean parameter rest_total_hits_as_int was introduced in version 7.0, and when set to True is will return the total number of document “hits” as an integer value.

Scrolling Elasticsearch documents with helpers.scan()

The last scan() method is a part of the client’s helpers library, and it’s basically a wrapper for the aforementioned scroll() method. The key difference us that helpers.scan() will return a generator instead of a JSON dictionary response. cURL One interesting feature of scan is that the index name is optional. The following is an example of how you can use it to scan for all the documents on the cluster:

As mentioned earlier, the method should return a generator object. The following code will explicitly cast the generator as a list in order to get its length with len() so that we can see how many documents were returned:

We can also enumerate the documents using the following example code:

The following is a model of all the possible parameters, and their respective default values, that you can pass to the scan() method:

Execute the Python script

Now, make sure to save the code in your Python script and use the python3 command in a terminal window to execute the script. Python should print some results that look something like the following:

Conclusion to the Scroll API

We’ve covered three different ways to scroll or scan through Elasticsearch documents using the Python low-level client library. The most common use case for scrolling documents is to reindex or copy an Elasticsearch index. Check out the example Python code in its entirety below.

Just the Code