How To Get Elasticsearch Documents Using Golang

Written by Data Pilot
July 10, 2019
  • Elasticsearch
  • Elasticsearch Driver
  • Golang
Have a Database Problem? Speak with an Expert for Free
Get Started >>

Introduction

If you’re a Golang developer working with Elasticsearch, you’ll probably want to query and access some of your Elasticsearch data via a Go script. Fortunately, the go-elasticsearch driver makes this task a quick and simple one– all it takes is some simple Golang code to execute a query and process the results. In this article, we’ll show you how to use the go-elasticsearch driver to query documents from a Golang script.

Prerequisites for querying Elasticsearch documents in Golang

Before we jump into the code used to get Elasticsearch documents in Golang, we’ll need to review the system requirements for this particular task. There are only a few prerequisites that need to be in place:

  • Golang must be installed, and the $GOPATH and $GOROOT must be exported in your bash profile. Use the go version and go env commands to see if Golang is installed and the appropriate paths are set.

  • Elasticsearch must be installed on the same node where you’ll be compiling and running your Golang scripts. You can use the Kibana UI or the cURL request shown below to check if the Elasticsearch cluster is running:

1
curl -XGET "localhost:9200"
  • The Golang driver for Elasticsearch (go-elasticsearch) must be installed in the server’s $GOPATH. You can use git to clone this library’s repository into your $GOPATH, using the command shown below:
1
git clone --branch master https://github.com/elastic/go-elasticsearch.git $GOPATH/src/github.com/elastic/go-elasticsearch
  • You’ll need to have an Elasticsearch index containing at least a few documents on your Elasticsearch cluster that you can use for querying and testing.

Create a Go script and import the necessary packages for the Elasticsearch client

One of the first things we’ll do in our Go script is import "bytes", the "encoding/json" package and a few other necessary packages for getting Elasticsearch documents:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
package main

import (
// Import packages for encoding strings to JSON
"bytes"
"encoding/json"

"context"
"fmt"
"log"
"reflect"
"strings"

// Import the Elasticsearch library packages
"github.com/elastic/go-elasticsearch/v8"
)

Declare the main() function and connect to Elasticsearch

Next, we’ll connect to Elasticsearch using the elasticsearch.NewClient() method. You can pass an optional configuration object instance to it as well, which we’ve done in the example below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
func main() {

// Allow for custom formatting of log output
log.SetFlags(0)

// Create a context object for the API calls
ctx := context.Background()

// Declare an Elasticsearch configuration
cfg := elasticsearch.Config{
Addresses: []string{
"http://localhost:9200",
},
Username: "user",
Password: "pass",
}

Check if the Elasticsearch cluster returned any API errors while connecting

It’s important to check for, and handle, any possible errors that might occur while connecting to Elasticsearch:

1
2
3
4
5
6
7
// Instantiate a new Elasticsearch client object instance
client, err := elasticsearch.NewClient(cfg)

// Exit the system if connection raises an error
if err != nil {
log.Fatalf("Elasticsearch connection error:", err)
}

Declare a mapping interface and query string for the Elasticsearch API call

We’ll need to instantiate an empty map[string]interface{} object. This will be used to temporarily store the Elasticsearch documents returned by the API call:

1
2
3
// Instantiate a mapping interface for API response
var mapResp map[string]interface{}
var buf bytes.Buffer

Declare an Elasticsearch JSON query string for the Golang driver

We’ll also need to declare an Elasticsearch query string that can be used to pass to the Golang method’s Search() method. Certain complex queries that use fields such as "bool" or "filter" may return a 400 HTTP response and error like the following:

1
[400 Bad Request] parsing_exception: Expected [START_OBJECT] but found [VALUE_STRING]

Make sure that your query string is a valid JSON object, as seen in the following example:

1
2
3
4
5
6
7
// Query for filtering a boolean value
var query = `{
"query": {
"term": {
"SomeStr" : "Some Value"
}
}, "size": 2}`

The query "size" is specified as the last field of the query. The default value of this parameter is 10, so if you omit this option, a maximum of 10 documents will be returned.

Create a query string to return all documents in an Elasticsearch index

Let’s look at a query string that, when passed to the Golang driver’s Search() method, will return all of an Elasticsearch index’s documents:

1
query = `{"query": {"match_all" : {}},"size": 2}`

Build and read the query string for the Elasticsearch Search() method

We can use the WriteString() method to build a strings.Builder object of the JSON query. This object can be passed to the strings.NewReader() method to get a *strings.Reader object that can be used to query Elasticsearch documents:

1
2
3
4
5
6
7
8
// Concatenate a string from query for reading
var b strings.Builder
b.WriteString(query)
read := strings.NewReader(b.String())

fmt.Println("read:", read)
fmt.Println("read TYPE:", reflect.TypeOf(read))
fmt.Println("JSON encoding:", json.NewEncoder(&buf).Encode(read))

Check if the JSON query string is valid

Be sure to use the json.NewEncoder() method to check if the JSON string is valid:

1
2
3
4
5
6
7
// Attempt to encode the JSON query and look for errors
if err := json.NewEncoder(&buf).Encode(read); err != nil {
log.Fatalf("Error encoding query: %s", err)

// Query is a valid JSON object
} else {
fmt.Println("json.NewEncoder encoded query:", read, "\n")

Call the Search() method passing the ‘read’ object containing the query body

Now that we have our encoded query, we can call the client instance’s client.Search() method. We’ll pass the *strings.Reader object, representing the JSON query, to the WithBody() method attribute of Search():

1
2
3
4
5
6
7
8
9
10
11
12
// Pass the JSON query to the Golang client's Search() method
res, err := client.Search(
client.Search.WithContext(ctx),
client.Search.WithIndex("some_index"),
client.Search.WithBody(read),
client.Search.WithTrackTotalHits(true),
client.Search.WithPretty(),
)

// Check for any errors returned by API call to Elasticsearch
if err != nil {
log.Fatalf("Elasticsearch Search() API ERROR:", err)

Check if the Search() API response returned any Elasticsearch client errors

Here’s another part of the code where it’s important to check for, and handle, any possible errors that were returned from the API call:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// If no errors are returned, parse esapi.Response object
} else {
fmt.Println("res TYPE:", reflect.TypeOf(res))

// Close the result body when the function call is complete
defer res.Body.Close()

// Decode the JSON response and using a pointer
if err := json.NewDecoder(res.Body).Decode(&mapResp); err != nil {
log.Fatalf("Error parsing the response body: %s", err)

// If no error, then convert response to a map[string]interface
} else {
fmt.Println("mapResp TYPE:", reflect.TypeOf(mapResp), "\n")

Iterate the documents returned by the Golang driver’s API call to Elasticsearch

Once we have our response object, we can use a for loop and iterate over the mapped response object’s "hits" attribute. As we iterate over this set, we’ll be printing out the id and the _source data for each returned document:

1
2
3
4
5
// Iterate the document "hits" returned by API call
for _, hit := range mapResp["hits"].(map[string]interface{})["hits"].([]interface{}) {

// Parse the attributes/fields of the document
doc := hit.(map[string]interface{})

Get an Elasticsearch document’s "_source" attribute in Golang

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// The "_source" data is another map interface nested inside of doc
source := doc["_source"]
fmt.Println("doc _source:", reflect.TypeOf(source))

// Get the document's _id and print it out along with _source data
docID := doc["_id"]
fmt.Println("docID:", docID)
fmt.Println("_source:", source, "\n")
} // end of response iteration

}
}
}
} // end of main() func

Conclusion

If you’re working with data in Elasticsearch, it’s important to be able to access the documents stored in your indexes. With the help of the go-elasticsearch driver, it’s easy to use the Search() method to get Elasticsearch documents in a Golang script. Just follow the examples and instructions provided in this tutorial, and you’ll be able to create your own Go script to query Elasticsearch and return documents.

Just the Code

Shown below is the Golang script to query Elasticsearch documents in its entirety:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
package main

import (
"bytes"
"context"
"encoding/json"
"fmt"
"log"
"reflect"
"strings"

// Import the Elasticsearch library packages
"github.com/elastic/go-elasticsearch/v8"
)

func main() {

// Allow for custom formatting of log output
log.SetFlags(0)

// Create a context object for the API calls
ctx := context.Background()

// Declare an Elasticsearch configuration
cfg := elasticsearch.Config{
Addresses: []string{
"http://localhost:9200",
},
Username: "user",
Password: "pass",
}

// Instantiate a new Elasticsearch client object instance
client, err := elasticsearch.NewClient(cfg)

// Exit the system if connection raises an error
if err != nil {
log.Fatalf("Elasticsearch connection error:", err)
}

// Instantiate a mapping interface for API response
var mapResp map[string]interface{}
var buf bytes.Buffer

// Query for filtering a boolean value
var query = `{
"query": {
"term": {
"SomeStr" : "Some Value"
}
}, "size": 2}`

// More example query strings to read and pass to Search()
//query = `{"query": {"match_all" : {}},"size": 2}`
//query = `{"query": {"term" : {"SomeBool": true}},"size": 3}`
query = `{"query": {"match" : {"SomeStr": "Value"}},"size": 3}`

/*
// The following query returns:
// [400 Bad Request] parsing_exception: Expected [START_OBJECT] but found [VALUE_STRING]

var query = `"query": {
"bool": { // omit this field to avoid 400 HTTP response
"filter": { // omit this field to avoid 400 HTTP response
"term": {
"SomeBool" : true
}
}
},
"size": 3}`
*/

// Concatenate a string from query for reading
var b strings.Builder
b.WriteString(query)
read := strings.NewReader(b.String())

fmt.Println("read:", read)
fmt.Println("read TYPE:", reflect.TypeOf(read))
fmt.Println("JSON encoding:", json.NewEncoder(&buf).Encode(read))

// Attempt to encode the JSON query and look for errors
if err := json.NewEncoder(&buf).Encode(read); err != nil {
log.Fatalf("Error encoding query: %s", err)

// Query is a valid JSON object
} else {
fmt.Println("json.NewEncoder encoded query:", read, "\n")

// Pass the JSON query to the Golang client's Search() method
res, err := client.Search(
client.Search.WithContext(ctx),
client.Search.WithIndex("some_index"),
client.Search.WithBody(read),
client.Search.WithTrackTotalHits(true),
client.Search.WithPretty(),
)

// Check for any errors returned by API call to Elasticsearch
if err != nil {
log.Fatalf("Elasticsearch Search() API ERROR:", err)

// If no errors are returned, parse esapi.Response object
} else {
fmt.Println("res TYPE:", reflect.TypeOf(res))

// Close the result body when the function call is complete
defer res.Body.Close()

// Decode the JSON response and using a pointer
if err := json.NewDecoder(res.Body).Decode(&mapResp); err != nil {
log.Fatalf("Error parsing the response body: %s", err)

// If no error, then convert response to a map[string]interface
} else {
fmt.Println("mapResp TYPE:", reflect.TypeOf(mapResp), "\n")

// Iterate the document "hits" returned by API call
for _, hit := range mapResp["hits"].(map[string]interface{})["hits"].([]interface{}) {

// Parse the attributes/fields of the document
doc := hit.(map[string]interface{})

// The "_source" data is another map interface nested inside of doc
source := doc["_source"]
fmt.Println("doc _source:", reflect.TypeOf(source))

// Get the document's _id and print it out along with _source data
docID := doc["_id"]
fmt.Println("docID:", docID)
fmt.Println("_source:", source, "\n")
} // end of response iteration

}
}
}
} // end of main() func
Rate 1 Star2 Stars3 Stars4 Stars5 Stars

Pilot the ObjectRocket Platform Free!

Try Fully-Managed CockroachDB, Elasticsearch, MongoDB, PostgreSQL (Beta) or Redis.

Get Started
Related Topics:
Additional Information:

Keep in the know!

Subscribe to our emails and we’ll let you know what’s going on at ObjectRocket. We hate spam and make it easy to unsubscribe.