How To Construct Elasticsearch Queries From A String Using Golang

Written by Data Pilot
July 10, 2019
  • Elasticsearch
  • Elasticsearch Driver
  • Golang
Have a Database Problem? Speak with an Expert for Free
Get Started >>

Introduction

If you need to get Elasticseearch documents via a Golang script, the queries you construct will be the most important part of your code. You’ll need to know how to construct and format these queries in order to search for documents with the go-elasticsearch driver. In this article, we’ll show how you can use Golang’s built-in packages and JSON encoding library to build valid JSON query strings that can then be passed to Elasticsearch.

Prerequisites for the Elasticsearch Golang driver’s API calls

Let’s review a few important prerequisites that need to be taken care of before proceeding with this tutorial:

  • First, Golang needs to be installed; in addition, both the $GOPATH and $GOROOT should be set in your bash profile. To check whether Golang is already installed and these paths are set, use the go version and go env commands in the terminal.

  • Elasticsearch also needs to be installed on the same node where your Golang scripts will be compiled and run. Use the Kibana UI or the cURL request below to verify that your Elasticsearch cluster is running:

1
curl -XGET "localhost:9200"
  • You’ll also need to confirm that the Golang driver for Elasticsearch (go-elasticsearch) has been installed in the server’s $GOPATH. To clone the library’s repository into your $GOPATH, use the git command shown below:
1
git clone --branch master https://github.com/elastic/go-elasticsearch.git $GOPATH/src/github.com/elastic/go-elasticsearch

Import the necessary packages for Elasticsearch and formatting JSON query strings

Now that we’ve made sure all our prerequisites are in place, let’s turn our attention to our Go script. We’ll start by importing some necessary packages, including the Golang packages and the package for the Elasticsearch Golang driver:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
package main


import (
"context"
"fmt"
"log"
"reflect"
"strings"
"bytes"
"strconv" // typecast "size" int as string
"encoding/json" // for encoding and validating JSON strings

// Import the Elasticsearch library packages
"github.com/elastic/go-elasticsearch/v8"
)

Make a function that can format and build Elasticsearch JSON query strings

In this article, our script will use an example function called constructQuery(). This function returns a valid *strings.Reader object from a string passed to it. We need this function to ensure that the strings being passed to the Elasticsearch API methods are valid JSON strings for querying Elasticsearch documents.

Declare the query function using Golang’s func keyword

We’ll use the func keyword to declare the new Golang function we described above, which will be used for formatting strings to valid Reader objects containing JSON queries for Elasticsearch. This particular example requires two parameters—- a string representing the JSON query, and an integer for the maximum number of Elasticsearch documents being returned by the API call:

1
func constructQuery(q string, size int) *strings.Reader {

Note that the function must return a *strings.Reader object, or Golang will throw an error during compiling.

Build a query string from the value passed to the q parameter

Next, let’s declare a new variable called query, which will contain the final concatenated result of the complete query string. Don’t forget to pass the document size integer to the strconv.Itoa() method to explicitly convert it to a string:

1
2
3
4
5
6
7
8
9
// Build a query string from string passed to function
var query = `{"query": {`

// Concatenate query string with string passed to method call
query = query + q

// Use the strconv.Itoa() method to convert int to string
query = query + `}, "size": ` + strconv.Itoa(size) + `}`
fmt.Println("\nquery:", query)

Let’s take a moment to look at the syntax needed for our query. Note that the first field in every Elasticsearch JSON query must be "query", and the complete query must be contained in brackets ({}) to be a valid JSON object.

Make sure the completed query string is a valid JSON object

Golang’s built-in json package has a method attribute called Valid() that returns a boolean when a string is passed to it. This boolean value signifies if the string is indeed a valid JSON object. In the following example, the query defaults to a simple ({}) match-all query if the original string concatenation is not a valid JSON object:

1
2
3
4
5
6
7
8
9
10
11
// Check for JSON errors
isValid := json.Valid([]byte(query)) // returns bool

// Default query is "{}" if JSON is invalid
if isValid == false {
fmt.Println("constructQuery() ERROR: query string not valid:", query)
fmt.Println("Using default match_all query")
query = "{}"
} else {
fmt.Println("constructQuery() valid JSON:", isValid)
}

Build the final query string and have the function return a *strings.Reader object

The last step is to pass the query string to the WriteString() method and instantiate a Reader object that can be passed to the Search() method:

1
2
3
4
5
6
7
8
9
10
// Build a new string from JSON query
var b strings.Builder
b.WriteString(query)

// Instantiate a *strings.Reader object from string
read := strings.NewReader(b.String())

// Return a *strings.Reader object
return read
}

Screenshot of the JSON Elasticsearch query string created by a function in Golang

Declare the main() function and connect to Elasticsearch

At this point, it’s time to declare the main() function and create a new client instance of the Golang driver:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
func main() {

// Allow for custom formatting of log output
log.SetFlags(0)

// Create a context object for the API calls
ctx := context.Background()

// Instantiate an Elasticsearch configuration
cfg := elasticsearch.Config{
Addresses: []string{
"http://localhost:9200",
},
Username: "user",
Password: "pass",
}

// Instantiate a new Elasticsearch client object instance
client, err := elasticsearch.NewClient(cfg)

// Check for connection errors to the Elasticsearch cluster
if err != nil {
fmt.Println("Elasticsearch connection error:", err)
}

Create a query string for the Elasticsearch API call

Now we’ll declare the Golang query string for the Search() API call:

1
2
3
4
5
6
7
8
9
// Create a new query string for the Elasticsearch method call
var query = `
"bool": {
"filter": {
"term": {
"SomeBool" : true
}
}
}`

Call the query builder function to create a Reader object for the Elasticsearch API call

In this section, we’ll call the function declared earlier, making sure to pass both the query string and an integer representing the "size" or number of documents returned by the API query:

1
2
3
// Pass the query string to the function and have it return a Reader object
read := constructQuery(query, 2)
fmt.Println("read:", read)

This returns a Reader object, which is what will allow us to use Elasticsearch and Golang to search for documents.

Declare an empty map interface for the returned documents

We’ll also need to create a new map interface{} object for the document strings returned by the API call:

1
2
// Instantiate a map interface object for storing returned documents
var mapResp map[string]interface{}

Create a bytes.Buffer object for the JSON query

The bytes.Buffer object is a tool that turns a byte slice into an io.Writer object. This allows the Elasticsearch JSON query string to be passed to an io.Reader or to a method like json.NewEncoder().

We’ll use a bytes.Buffer object in our script to check if the JSON object can be encoded:

1
2
3
4
5
var buf bytes.Buffer

// Attempt to encode the JSON query and look for errors
if err := json.NewEncoder(&buf).Encode(read); err != nil {
log.Fatalf("json.NewEncoder() ERROR:", err)

Pass the Reader object to the Golang driver’s Search() method

At this point, we’ve successfully constructed an Elasticsearch JSON query string in Golang. We’re ready to use the go-elasticsearch driver to query for documents. We call the client instance’s Search() method, pass the read object returned by the custom query function, and have it return a response:

1
2
3
4
5
6
7
8
9
10
11
// Query is a valid JSON object
} else {
fmt.Println("json.NewEncoder encoded query:", read, "\n")

// Pass the JSON query to the Golang client's Search() method
res, err := client.Search(
client.Search.WithContext(ctx),
client.Search.WithIndex("some_index"),
client.Search.WithBody(read),
client.Search.WithTrackTotalHits(true),
)

Check for API errors and parse the Elasticsearch response

Finally, we check for any API errors and use the json.NewDecoder() method to decode the API response from Elasticsearch:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// Check for any errors returned by API call to Elasticsearch
if err != nil {
log.Fatalf("Elasticsearch Search() API ERROR:", err)

// If no errors are returned, parse esapi.Response object
} else {
fmt.Println("res TYPE:", reflect.TypeOf(res))

// Close the result body when the function call is complete
defer res.Body.Close()

// Decode the JSON response and using a pointer
if err := json.NewDecoder(res.Body).Decode(&mapResp); err == nil {
fmt.Println(`&mapResp:`, &mapResp, "\n")
fmt.Println(`mapResp["hits"]:`, mapResp["hits"])
}
}
}
} // end of main() func

Conclusion

If you’re communicating with Elasticsearch using the Golang programming language, queries can be considered the heart and soul of your scripts. It’s important to know how to format your queries correctly so that they can be passed to the Golang driver’s Search() method. Fortunately, it’s easy to construct queries and then get Elasticsearch data using the go-elasticsearch driver. Using the code examples provided in this article, you’ll be prepared to write your own script to get Elasticsearch documents with Golang.

Just the Code

We’ve looked at the code one section at a time throughout this tutorial. Included below is the Golang script in its entirety:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
package main

import (
"bytes"
"context"
"encoding/json"
"fmt"
"log"
"reflect"
"strconv" // typecast "size" int as string
"strings"

// Import the Elasticsearch library packages
"github.com/elastic/go-elasticsearch/v8"
)

func constructQuery(q string, size int) *strings.Reader {

// Build a query string from string passed to function
var query = `{"query": {`

// Concatenate query string with string passed to method call
query = query + q

// Use the strconv.Itoa() method to convert int to string
query = query + `}, "size": ` + strconv.Itoa(size) + `}`
fmt.Println("\nquery:", query)

// Check for JSON errors
isValid := json.Valid([]byte(query)) // returns bool

// Default query is "{}" if JSON is invalid
if isValid == false {
fmt.Println("constructQuery() ERROR: query string not valid:", query)
fmt.Println("Using default match_all query")
query = "{}"
} else {
fmt.Println("constructQuery() valid JSON:", isValid)
}

// Build a new string from JSON query
var b strings.Builder
b.WriteString(query)

// Instantiate a *strings.Reader object from string
read := strings.NewReader(b.String())

// Return a *strings.Reader object
return read
}

func main() {

// Allow for custom formatting of log output
log.SetFlags(0)

// Create a context object for the API calls
ctx := context.Background()

// Instantiate an Elasticsearch configuration
cfg := elasticsearch.Config{
Addresses: []string{
"http://localhost:9200",
},
Username: "user",
Password: "pass",
}

// Instantiate a new Elasticsearch client object instance
client, err := elasticsearch.NewClient(cfg)

// Check for connection errors to the Elasticsearch cluster
if err != nil {
fmt.Println("Elasticsearch connection error:", err)
}

// Create a new query string for the Elasticsearch method call
var query = `
"bool": {
"filter": {
"term": {
"SomeBool" : true
}
}
}`

// Pass the query string to the function and have it return a Reader object
read := constructQuery(query, 2)

// Example of an invalid JSON string
//read = constructQuery("{bad json", 2)

fmt.Println("read:", read)

// Instantiate a map interface object for storing returned documents
var mapResp map[string]interface{}
var buf bytes.Buffer

// Attempt to encode the JSON query and look for errors
if err := json.NewEncoder(&buf).Encode(read); err != nil {
log.Fatalf("json.NewEncoder() ERROR:", err)

// Query is a valid JSON object
} else {
fmt.Println("json.NewEncoder encoded query:", read, "\n")

// Pass the JSON query to the Golang client's Search() method
res, err := client.Search(
client.Search.WithContext(ctx),
client.Search.WithIndex("some_index"),
client.Search.WithBody(read),
client.Search.WithTrackTotalHits(true),
)

// Check for any errors returned by API call to Elasticsearch
if err != nil {
log.Fatalf("Elasticsearch Search() API ERROR:", err)

// If no errors are returned, parse esapi.Response object
} else {
fmt.Println("res TYPE:", reflect.TypeOf(res))

// Close the result body when the function call is complete
defer res.Body.Close()

// Decode the JSON response and using a pointer
if err := json.NewDecoder(res.Body).Decode(&mapResp); err == nil {
fmt.Println(`&mapResp:`, &mapResp, "\n")
fmt.Println(`mapResp["hits"]:`, mapResp["hits"])
}
}
}
} // end of main() func
Rate 1 Star2 Stars3 Stars4 Stars5 Stars

Pilot the ObjectRocket Platform Free!

Try Fully-Managed CockroachDB, Elasticsearch, MongoDB, PostgreSQL (Beta) or Redis.

Get Started
Related Topics:
Additional Information:

Keep in the know!

Subscribe to our emails and we’ll let you know what’s going on at ObjectRocket. We hate spam and make it easy to unsubscribe.