How to use Pagination on a Query in Elasticsearch 6 using NodeJS

Have a Database Problem? Speak with an Expert for Free
Get Started >>

Introduction

It’s often that queries used in Elasticsearch return an extremely large number of results. This may be ok for a computer but too many results can be hard for us to process mentally. In these cases it’s useful to limit the number of results your query results through pagination. We’ll show you how use pagination step-by-step in Elasticsearch. If you’d just rather see the code, click here to jump to Just the Code.

Prerequisites

Before we show you how to compute a weighted average with Elasticsearch in Javascript, it’s important to make sure a few prerequisites are in place. There are only a few of system requirements for this task: NodeJS needs to be installed The elasticsearch npm module installed. A simple npm install elasticsearch should work in most cases. Elasticsearch also needs to be installed and running. * In our example, we have Elasticsearch installed locally using the default port of 9200. If your Elasticsearch installation is running on a different server, you’ll need to modify your javascript syntax accordingly.

Use pagination with the from and size parameters

We love to show by example. In our example we use an index representing a small grocery store called store. The store index contains the type products which lists all the stores products. To keep it simple our dataset only has a small number of products with just a few fields including the id, price, quantity, and department. Here is the json we used to create our dataset:

idnamepricequantitydepartment
1Multi-Grain Cereal4.994Packaged Foods
21lb Ground Beef3.9929Meat and Seafood
3Dozen Apples2.4912Produce
4Chocolate Bar1.292Packaged FoodsCheckout
51 Gallon Milk3.2916Dairy
60.5lb Jumbo Shrimp5.2912Meat and Seafood
7Wheat Bread1.295Bakery
8Pepperoni Pizza2.995Frozen
912 Pack Cola5.296Packaged Foods
10Lime Juice0.9920Produce
1112 Pack Cherry Cola5.5995Packaged Foods
121 Gallon Soy Milk3.3910Dairy
131 Gallon Vanilla SoyMilk3.499Dairy
141 Gallon Orange Juice3.294Juice

Here is the json we used to define the mapping:

1
2
3
4
5
6
7
8
9
10
11
12
{
    "mappings": {
        "products": {
            "properties" : {
                "name": { "type": "text"},
                "price": { "type": "double"},
                "quantity": { "type": "integer"},
                "department": { "type": "keyword"}
            }
        }
    }
}

First let’s show a query for all the products whose price is greater than $1.00. If this were a real grocery store database that would return an extremely large number of results.

File index.js: `js var elasticsearch = require(“elasticsearch”);

var client = new elasticsearch.Client({ hosts: [“http://localhost:9200”] });

/ Query Products Over $1.00 / client.search({ index: ‘store’, type: ‘products’, body: {


1
2
3
4
5
6
7
8
      "query": {
          "range": {
              "price": {
                  "gte": 1.00
              }
          }
      }
    }

}).then(function(resp) { console.log(“Successful query!”); console.log(resp); }, function(err) { console.trace(err.message); }); `

We run the application in index.js with NodeJS with this command:

1
$ node index.js

Here is the output: `js Successful query! { took: 1, timed_out: false, _shards: { total: 5, successful: 5, skipped: 0, failed: 0 }, hits: { total: 13,


1
2
3
4
5
6
7
8
9
10
11
12
 max_score: 1,
 hits:
  [ [Object],
    [Object],
    [Object],
    [Object],
    [Object],
    [Object],
    [Object],
    [Object],
    [Object],
    [Object] ] } }
1
2
3
4
5
As you can see with a real dataset this could easily get out of hand, so we'll use pagination to only look at the first 2 results.

To limit so that it only shows the first two results we use pagination. To use pagination all we need to do is set the `from` and `size` parameters when we search.  For this demonstration we'll show the code and then dissect it step by step afterwards:

File `index.js`:

var elasticsearch = require(“elasticsearch”);

var client = new elasticsearch.Client({ hosts: [“http://localhost:9200”] });

/ Query Products Over $1.00 Limit to 2 Results with Pagination/ client.search({ from: 0, size: 2, index: ‘store’, type: ‘products’, body: {


1
2
3
4
5
6
7
8
      "query": {
          "range": {
              "price": {
                  "gte": 1.00
              }
          }
      }
    }

}).then(function(resp) { console.log(“Successful query!”); console.log(JSON.stringify(resp, null, 4)); }, function(err) { console.trace(err.message); }); `

The size is simple so let’s start there. It tells Elasticsearch how many results to return. The from parameter is a little trickier. It tells Elasticsearch how many to offset by for when you don’t want the first page of results.

Note It’s important to know that the from parameter is 0 based like in most programming languages. So if you want to return the results starting from the beginning use "from": 0. If you want to skip the first result and start at the second you’d use "from": 1. This is a little counterintuitive and takes getting used to.

Note Also the defaults are from is 0 and size is 10. So don’t freak out about getting thousands of results in the terminal because the default size will limit you to ten unless you change it.

Now that we’ve explained the query let’s see the results:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
$ node index.js
Successful query!
{
    "took": 0,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 13,
        "max_score": 1,
        "hits": [
            {
                "_index": "store",
                "_type": "products",
                "_id": "14",
                "_score": 1,
                "_source": {
                    "id": "14",
                    "name": "1 Gallon Orange Juice",
                    "price": 3.29,
                    "quantity": 4,
                    "department": [
                        "Juice"
                    ]
                }
            },
            {
                "_index": "store",
                "_type": "products",
                "_id": "5",
                "_score": 1,
                "_source": {
                    "id": "5",
                    "name": "1 Gallon Milk",
                    "price": 3.29,
                    "quantity": 16,
                    "department": [
                        "Dairy"
                    ]
                }
            }
        ]
    }
}

As you can see it returned only the two results like we specified in the size. If we wanted the next page of results we’d set "size": 2 and "from": 2. The size stays the same but since we’ve already retrieved the first page ( results 0 & 1) we set "from": 2. Let’s see that in action and make sure we get different results.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
var elasticsearch = require("elasticsearch");

var client = new elasticsearch.Client({
  hosts: ["http://localhost:9200"]
});

/* Query Products Over $1.00 Limit to 2 Results with Pagination*/
client.search({
  from: 2,
  size: 2,
  index: 'store',
  type: 'products',
  body: {
          "query": {
              "range": {
                  "price": {
                      "gte": 1.00
                  }
              }
          }
        }
}).then(function(resp) {
  console.log("Successful query!");
  console.log(JSON.stringify(resp, null, 4));
}, function(err) {
  console.trace(err.message);
});

Output: `js { “took” : 2, “timed_out” : false, “_shards” : {


1
2
3
4
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0

}, “hits” : {


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
"total" : 13,
"max_score" : 1.0,
"hits" : [
  {
    "_index" : "store",
    "_type" : "products",
    "_id" : "8",
    "_score" : 1.0,
    "_source" : {
      "id" : "8",
      "name" : "Pepperoni Pizza",
      "price" : 2.99,
      "quantity" : 5,
      "department" : [
        "Frozen"
      ]
    }
  },
  {
    "_index" : "store",
    "_type" : "products",
    "_id" : "9",
    "_score" : 1.0,
    "_source" : {
      "id" : "9",
      "name" : "12 Pack Cola",
      "price" : 5.29,
      "quantity" : 6,
      "department" : [
        "Packaged Foods"
      ]
    }
  }
]

} } `

As expected we get the next page of results and you can see that the results are different. This is how you might page through results by incrementing the from parameter.

Note You’ll notice that the results did not come back in order of index. Look into Elasticsearch sorting to see how to do that.

Conclusion

In this tutorial we demonstrated how to use Elasticsearch pagination with the from and size parameters to limit the query results. This is a common use case. It’s very good for getting a bird’s eye view of your data. There are many other things you can do with aggregation and you can consult the Elasticsearch documentation to learn more about it. The documentation is also useful if you need help with syntax. We hope you found this tutorial on pagination helpful and can apply it to your specific application. If you have questions or this didn’t work for you please reach out to us so we can help. Thank you.

Just the Code

If you’re already comfortable with these concepts here’s all the code we used to demonstrate how to use pagination in Elasticsearch using NodeJS.

  • Get the First Page of Results File index.js:
    1
    var elasticsearch = require("elasticsearch");

var client = new elasticsearch.Client({ hosts: [“http://localhost:9200”] });

/ Query Products Over $1.00 Limit to 2 Results with Pagination/ client.search({ from: 0, size: 2, index: ‘store’, type: ‘products’, body: {


1
2
3
4
5
6
7
8
      "query": {
          "range": {
              "price": {
                  "gte": 1.00
              }
          }
      }
    }

}).then(function(resp) { console.log(“Successful query!”); console.log(JSON.stringify(resp, null, 4)); }, function(err) { console.trace(err.message); }); * Get the Second Page of Results
File `index.js`:
js var elasticsearch = require(“elasticsearch”);

var client = new elasticsearch.Client({ hosts: [“http://localhost:9200”] });

/ Query Products Over $1.00 Limit to 2 Results with Pagination/ client.search({ from: 2, size: 2, index: ‘store’, type: ‘products’, body: {


1
2
3
4
5
6
7
8
      "query": {
          "range": {
              "price": {
                  "gte": 1.00
              }
          }
      }
    }

}).then(function(resp) { console.log(“Successful query!”); console.log(JSON.stringify(resp, null, 4)); }, function(err) { console.trace(err.message); }); `

Pilot the ObjectRocket Platform Free!

Try Fully-Managed CockroachDB, Elasticsearch, MongoDB, PostgreSQL (Beta) or Redis.

Get Started

Keep in the know!

Subscribe to our emails and we’ll let you know what’s going on at ObjectRocket. We hate spam and make it easy to unsubscribe.