How to use Pagination on a Query in Elasticsearch 6 using NodeJS

Introduction

It’s often that queries used in Elasticsearch return an extremely large number of results. This may be ok for a computer but too many results can be hard for us to process mentally. In these cases it’s useful to limit the number of results your query results through pagination. We’ll show you how use pagination step-by-step in Elasticsearch. If you’d just rather see the code, click here to jump to Just the Code.

Prerequisites

Before we show you how to compute a weighted average with Elasticsearch in Javascript, it’s important to make sure a few prerequisites are in place. There are only a few of system requirements for this task: NodeJS needs to be installed The elasticsearch npm module installed. A simple npm install elasticsearch should work in most cases. Elasticsearch also needs to be installed and running. * In our example, we have Elasticsearch installed locally using the default port of 9200. If your Elasticsearch installation is running on a different server, you’ll need to modify your javascript syntax accordingly.

Use pagination with the from and size parameters

We love to show by example. In our example we use an index representing a small grocery store called store. The store index contains the type products which lists all the stores products. To keep it simple our dataset only has a small number of products with just a few fields including the id, price, quantity, and department. Here is the json we used to create our dataset:

Here is the json we used to define the mapping:

First let’s show a query for all the products whose price is greater than $1.00. If this were a real grocery store database that would return an extremely large number of results.

File index.js: `js var elasticsearch = require(“elasticsearch”);

var client = new elasticsearch.Client({ hosts: [“http://localhost:9200”] });

/ Query Products Over $1.00 / client.search({ index: ‘store’, type: ‘products’, body: {

1
2
3
4
5
6
7
8
      "query": {

          "range": {

              "price": {

                  "gte": 1.00

              }

          }

      }

    }

}).then(function(resp) { console.log(“Successful query!”); console.log(resp); }, function(err) { console.trace(err.message); }); `

We run the application in index.js with NodeJS with this command:

Here is the output: `js Successful query! { took: 1, timed_out: false, _shards: { total: 5, successful: 5, skipped: 0, failed: 0 }, hits: { total: 13,

1
2
3
4
5
6
7
8
9
10
11
12
 max_score: 1,

 hits:

  [ [Object],

    [Object],

    [Object],

    [Object],

    [Object],

    [Object],

    [Object],

    [Object],

    [Object],

    [Object] ] } }

var elasticsearch = require(“elasticsearch”);

var client = new elasticsearch.Client({ hosts: [“http://localhost:9200”] });

/ Query Products Over $1.00 Limit to 2 Results with Pagination/ client.search({ from: 0, size: 2, index: ‘store’, type: ‘products’, body: {

1
2
3
4
5
6
7
8
      "query": {

          "range": {

              "price": {

                  "gte": 1.00

              }

          }

      }

    }

}).then(function(resp) { console.log(“Successful query!”); console.log(JSON.stringify(resp, null, 4)); }, function(err) { console.trace(err.message); }); `

The size is simple so let’s start there. It tells Elasticsearch how many results to return. The from parameter is a little trickier. It tells Elasticsearch how many to offset by for when you don’t want the first page of results.

Note It’s important to know that the from parameter is 0 based like in most programming languages. So if you want to return the results starting from the beginning use "from": 0. If you want to skip the first result and start at the second you’d use "from": 1. This is a little counterintuitive and takes getting used to.

Note Also the defaults are from is 0 and size is 10. So don’t freak out about getting thousands of results in the terminal because the default size will limit you to ten unless you change it.

Now that we’ve explained the query let’s see the results:

As you can see it returned only the two results like we specified in the size. If we wanted the next page of results we’d set "size": 2 and "from": 2. The size stays the same but since we’ve already retrieved the first page ( results 0 & 1) we set "from": 2. Let’s see that in action and make sure we get different results.

Output: `js { “took” : 2, “timed_out” : false, “_shards” : {

1
2
3
4
"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

}, “hits” : {

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
"total" : 13,

"max_score" : 1.0,

"hits" : [

  {

    "_index" : "store",

    "_type" : "products",

    "_id" : "8",

    "_score" : 1.0,

    "_source" : {

      "id" : "8",

      "name" : "Pepperoni Pizza",

      "price" : 2.99,

      "quantity" : 5,

      "department" : [

        "Frozen"

      ]

    }

  },

  {

    "_index" : "store",

    "_type" : "products",

    "_id" : "9",

    "_score" : 1.0,

    "_source" : {

      "id" : "9",

      "name" : "12 Pack Cola",

      "price" : 5.29,

      "quantity" : 6,

      "department" : [

        "Packaged Foods"

      ]

    }

  }

]

} } `

As expected we get the next page of results and you can see that the results are different. This is how you might page through results by incrementing the from parameter.

Note You’ll notice that the results did not come back in order of index. Look into Elasticsearch sorting to see how to do that.

Conclusion

In this tutorial we demonstrated how to use Elasticsearch pagination with the from and size parameters to limit the query results. This is a common use case. It’s very good for getting a bird’s eye view of your data. There are many other things you can do with aggregation and you can consult the Elasticsearch documentation to learn more about it. The documentation is also useful if you need help with syntax. We hope you found this tutorial on pagination helpful and can apply it to your specific application. If you have questions or this didn’t work for you please reach out to us so we can help. Thank you.

Just the Code

If you’re already comfortable with these concepts here’s all the code we used to demonstrate how to use pagination in Elasticsearch using NodeJS.

• Get the First Page of Results File index.js:

  1	var elasticsearch = require("elasticsearch");

var client = new elasticsearch.Client({ hosts: [“http://localhost:9200”] });

/ Query Products Over $1.00 Limit to 2 Results with Pagination/ client.search({ from: 0, size: 2, index: ‘store’, type: ‘products’, body: {

1
2
3
4
5
6
7
8
      "query": {

          "range": {

              "price": {

                  "gte": 1.00

              }

          }

      }

    }

}).then(function(resp) { console.log(“Successful query!”); console.log(JSON.stringify(resp, null, 4)); }, function(err) { console.trace(err.message); }); * Get the Second Page of Results
File `index.js`:js var elasticsearch = require(“elasticsearch”);

var client = new elasticsearch.Client({ hosts: [“http://localhost:9200”] });

/ Query Products Over $1.00 Limit to 2 Results with Pagination/ client.search({ from: 2, size: 2, index: ‘store’, type: ‘products’, body: {

1
2
3
4
5
6
7
8
      "query": {

          "range": {

              "price": {

                  "gte": 1.00

              }

          }

      }

    }

}).then(function(resp) { console.log(“Successful query!”); console.log(JSON.stringify(resp, null, 4)); }, function(err) { console.trace(err.message); }); `