How to Use Percentile Aggregation in Elasticsearch with Curl

Introduction

An aggregation computation that comes up frequently when trying to figure out the distribution of your data is the percentile aggregation. Percentile aggregagtions are simple in Elasticsearch and this step-by-step example will show you an example of one that you can use as a basis for your own aggregation. In this step-by-step example we will interact with Elasticsearch using Curl commands to perform the percentile aggregation. If you’d just rather see the example code, click here to jump to Just the Code.

Note: The code will vary depending on all your system parameters but we hope to give you an idea of how this is done.

What is a Percentile Aggregation

Percentile aggregations are helpful to find outliers in data and figure out the distrubution of data. If you graduated in the top 10% of your class, that is a good example of percentile aggregation. Another way to look at it is to say that the 99th percentile is the value which is greater than 99% of all other values that appear in the dataset.

Prerequisites

Before we take a look at the how to perform a percentile aggregation, it’s important to mention a few prerequisites that need to be in place. For this task, the system requirements are minimal: Elasticsearch needs to be installed and running. Though it’s not required, it’s helpful to have some basic familiarity with the curl command. * In our example, we have Elasticsearch installed locally using the default port of 9200. If your Elasticsearch installation is running on a different server, you’ll need to modify your syntax accordingly.

Introduce our Example Data

Let’s look at an example that uses an index called store, which represents a small grocery store. This store index contains a type called products which lists the store’s products. To keep things simple, our example dataset will only contain a handful of products with just the following fields: id, price, quantity, and department. The code below shows the JSON used to create the dataset:

idnamepricequantitydepartment
1Multi-Grain Cereal4.994Packaged Foods
21lb Ground Beef3.9929Meat and Seafood
3Dozen Apples2.4912Produce
4Chocolate Bar1.292Packaged Foods, Checkout
51 Gallon Milk3.2916Dairy
60.5lb Jumbo Shrimp5.2912Meat and Seafood
7Wheat Bread1.295Bakery
8Pepperoni Pizza2.995Frozen
912 Pack Cola5.296Packaged Foods
10Lime Juice0.9920Produce
1112 Pack Cherry Cola5.595Packaged Foods
121 Gallon Soy Milk3.3910Dairy
131 Gallon Vanilla Soy Milk3.499Dairy
141 Gallon Orange Juice3.294Juice

Here is the json we used to define the mapping if our index:

{
    "mappings": {
        "products": {
            "properties" : {
                "name": { "type": "text"},
                "price": { "type": "double"},
                "quantity": { "type": "integer"},
                "department": { "type": "keyword"}
            }
        }
    }
}

Use the Percentile Aggregation

Now let’s say we want to get a sense of our pricing at our store. We could use a percentile aggregation to get a sense of the distribution of our prices. The code to sum the quantities of each product is below. Let’s take a look and then we’ll dissect the code afterwards.

$ curl -H "Content-Type: application/json" -XGET "127.0.0.1:9200/store/products/_search?size=0&pretty" -d '
> {
>     "aggs": {
>       "price_by_percentile": {
>         "percentiles": { "field": "price"}
>       }
>     }
> }
> '

{
  "took" : 44,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 14,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "price_by_percentile" : {
      "values" : {
        "1.0" : 0.99,
        "5.0" : 1.05,
        "25.0" : 2.49,
        "50.0" : 3.34,
        "75.0" : 4.99,
        "95.0" : 5.529999999999999,
        "99.0" : 5.59
      }
    }
  }
}

You can see what’s returned is an easy to read breakdown of our prices with our lowest price $0.99 at the low end (1st percentile) and $5.59 at the high end (99th percentile).

Now let’s go over the important parts of the code: We created an aggregator with "aggs" keyword. Then we gave it a name "price_by_percentile". This can be whatever name suits your application. Next we used "percentiles" to set the type of aggregator so Elasticsearch knows what calculation to perform with this aggregation. Lastly we specified which field we wanted to sum up ‘”price”`.

Note: In the curl url notice that we used size=0 because otherwise this query would return every product. With this parameter set, all we see is the aggregate information. We also specify the index and type in the url.

Conclusion

In this tutorial we demonstrated how to use Elasticsearch to use the percentile aggregation. There are many other operations you can perform with aggregations and you can consult the Elasticsearch documentation to learn more about it. The documentation is also useful if you need help with speciific syntax.

We hope you found tutorial helpful and you can apply it to your specific application. If you have questions or this didn’t work for you please reach out to us so we can help. Thank you.

Just the Code

If you’re already comfortable with curl and aggregations here’s all the code we used to demonstrate how to do a percentile aggregation with Elasticsearch and Curl.

$ curl -H "Content-Type: application/json" -XGET "127.0.0.1:9200/store/products/_search?size=0&pretty" -d '
> {
>     "aggs": {
>       "price_by_percentile": {
>         "percentiles": { "field": "price"}
>       }
>     }
> }
> '

Pilot the ObjectRocket Platform Free!

Try Fully-Managed Redis,
MongoDB & Elasticsearch

Get Started

OR

Try CockroachDB
in Beta

Get Started

Keep in the know!

Subscribe to our emails and we’ll let you know what’s going on at ObjectRocket. We hate spam and make it easy to unsubscribe.