How to Make An HTTP Request In Golang To Index Elasticsearch Documents

Written by Data Pilot
July 22, 2019
  • Elasticsearch
  • Golang
Have a Database Problem? Speak with an Expert for Free
Get Started >>

Introduction

This tutorial will demonstrate how to make an HTTP request in Golang to index Elasticsearch documents. This tutorial will provide examples of how to use Golang’s built-in packages to index Elasticsearch documents Golang HTTP client, how to index Elasticsearch documents with Golang using just the built-in "net/http" HTTP client package and how to index documents with HTTP in Golang without the need for third-party applications.

Prerequisites for using Golang’s HTTP client to index Elasticsearch documents

  • Golang must be running and the $GOPATH and $GOROOT must be exported in the bash profile. Execute the go version and go env commands to confirm Golang is properly installed and the paths properly set.

  • Elasticsearch must be installed on the same node compiling the Golang scripts. Use the Kibana UI to verify the Elasticsearch cluster is running or execute the following cURL request:

1
curl -XGET "localhost:9200"

How to Import the Needed Golang Packages for Indexing Elasticsearch Documents

The following script contains all of the native packages needed to index Elasticsearch documents using just Golang’s http package without a third-party driver:

1
2
3
4
5
6
7
8
9
10
11
12
package main

import (
"bytes" // marshal JSON string to bytes
"fmt" // format and print out data
"io/ioutil" // for reading API response
"log" // log errors to server
"strconv" // Convert the Elasticsearch port integer to a string
"net/http" // to make HTTP requests to Elasticsearch
"reflect" // get object type
"strings" // to eval string objects
)

How to Declare the Main() Function and JSON Strings for Elasticsearch Documents

For thoroughness, this tutorial will explain how to index a single Elasticsearch document and how to make a call to the _bulk API using an HTTP request.

Declare the main() function

Execute the following command to declare the main() function:

1
2
3
4
func main() {

// Allow for custom formatting of log output
log.SetFlags(0)

How to instantiate the JSON strings for the HTTP request to Elasticsearch

First, create a valid JSON string that will represent the Elasticsearch documents being indexed. Make certain to include an index row if the string contains multiple documents for the _bulk API request. Also create a new line character ('n') at the end of the bulk JSON string to avoid getting a 400 HTTP response. An example of this function follows:

1
2
3
4
5
6
7
8
9
10
11
// Use backticks (`) for multiline JSON strings
bulkRequest := `{"index": {"_index": "some_index", "_id": 42}}
{"str field": "Original Value 1", "int field": 1234, "bool field": true}
{"index": {"_index": "some_index", "_id": 123}}
{"str field": "Original Value 2", "int field": 5678, "bool field": false}`

// HTTP 400: "The bulk request must be terminated by a newline [\n]"
bulkRequest = bulkRequest + "n"

// JSON string for indexing a single Elasticsearch document
indexOne := `{"str field": "Original Value 3", "int field": 9012, "bool field": true}`

How to place the Elasticsearch documents strings into a string slice

As shown in the following script, declare a string-typed array slice and append the JSON strings containing the Elasticsearch documents to it using Golang’s append() function:

1
2
3
4
5
6
// Declare an empty string array to store JSON document strings
var docStrings []string

// Append the strings to the slice array
docStrings = append(docStrings, bulkRequest)
docStrings = append(docStrings, indexOne)

How to Iterate Over the Elasticsearch Document String Slice and Make an HTTP Request for Each String

This section will show how to iterate over the strings and how each type of index API request has to be handled differently. The bulk document request must have _bulk in the HTTP request header and the index name in the JSON document’s index row. However, the single document API request must have the index name and document type specified in the header itself.

How to iterate over the string slice and create a bytes buffer slice from the JSON document

Execute the following command:

1
2
3
4
5
6
7
8
9
10
// Iterate over the string array and make API calls
for i, str := range docStrings {

// Print the document's index number in the array
fmt.Printf("index:", i)

// Convert Elasticsearch document string to bytes buffer object for API call
byteSlice := []byte(str)
buf := bytes.NewBuffer(byteSlice)
fmt.Println("bytes buffer for JSON string:", reflect.TypeOf(buf))

How to set up the HTTP request’s header string to match the Elasticsearch JSON document type

The header for the request is passed to the http package as a string. The following script shows how the header can be customized differently, depending on the request:

1
2
3
4
5
6
7
8
9
10
11
12
// Default HTTP verb is "POST"
verb := "POST"

// Default domain and port for Elasticsearch cluster
domain := "http://localhost"
port := 9200

// Convert the port int to a string and concatenate with domain
domain = domain + ":" + strconv.Itoa(port) + "/"

// Elasticsearch index the documents are created in
ind := "some_index"

How to check if the API request is a _bulk request by evaluating the request’s string

If the document string contains an index row (_index), then insert the _bulk parameter into the request string using the following script:

1
2
3
4
5
6
7
8
9
10
11
// Check if doc string contains "_index" for _bulk API requests
isBulk := strings.Contains(str, `"_index"`)

// Use the _bulk API if JSON doc string contains multiple docs
if isBulk == true {
requestStr = requestStr + "_bulk?pretty=true"
verb = "PUT"
} else {
// The Elasticsearch index must be explicitly declared in header
requestStr = requestStr + ind + "/_doc?pretty=true"
}

After the request string is complete it can be passed to the NewRequest() method.

How to make an HTTP request to Elasticsearch using the http.NewRequest method

Execute the following script to make a request using the http.NewRequest method and have it return both a request and error object:

1
2
// Call the HTTP package's NewRequest() method
req, err := http.NewRequest(verb, domain, buf)

How to set the request header’s Content-Type so Elasticsearch doesn’t return a 406 HTTP error

Elasticsearch versions 6.0 and later require that all requests with a JSON body have the header’s "Content-Type" specified. Failuer to do this will result in a response with a "Content-Type header [] is not supported","status":406} error as shown here:

1
2
3
4
5
6
7
8
// ES 6.0> requires Content-Type header to avoid 406 HTTP error:
req.Header.Set("Content-Type", "application/json")

// Print out the HTTP request and check for errors
fmt.Println("nHTTP Request:", req)
if err != nil {
log.Fatalf("http.NewRequest ERROR:", err)
}

Now print out the request and any errors it may have.

How to Make an API Request Using an HTTP Client Instance for Each Document Iteration

Execute the following script to instantiate a new client instance that will be used to contain the Elasticsearch response body:

1
2
3
4
5
6
7
8
9
10
11
// Instantiate a new client object
client := &http.Client{}

// Pass HTTP request to Elasticsearch client and check for errors
resp, err := client.Do(req)
if err != nil {
log.Fatalf("client.Do ERROR:", err)
}

// Close the response body after operations are complete
defer resp.Body.Close()

How to parse the resp.Body object returned by the HTTP client instance

The following commands uses the ioutil package’s ReadAll() method to parse the response body returned by Elasticsearch:

1
2
3
4
5
// Parse out the response body and check for errors
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalf("client.Do ERROR:", err)
}

How to print out the Elasticsearch JSON response to the Golang HTTP request

As shown on the following script, specifically identify the ioutil response body object as a string and then print out the results:

1
2
3
4
5
6
7
// Convert the bytes object []uint8 of the JSON response to a string
strBody := string(body)

// Print out the response body
fmt.Println("nresp.Body:", strBody, "nn")
}
}

Conclusion

This tutorial explained how to make an HTTP request in Golang to index an Elasticsearch document without the need of any third-party packages or drivers. The tutorial covered how to import the needed Golang packages to index Elasticsearch documents with Golang, how to declare the main() function and JSON strings for Elasticsearch documents, how to instantiate the JSON strings to index documents with HTTP in Golang, how to place the Elasticsearch documents strings into a string slice, how to iterate over the Elasticsearch document string slice, how to index Elasticsearch documents with Golang HTTP client request for each string and how to print out the Elasticsearch JSON response to the Golang HTTP request. Remember that to use Golang’s HTTP client to index documents in Elasticsearch the program must be installed on the same node compiling the Golang scripts.

Just the Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
package main

import (
"bytes" // marshal JSON string to bytes
"fmt" // format and print out data
"io/ioutil" // for reading API response
"log" // log errors to server
"net/http" // to make HTTP requests to Elasticsearch
"reflect" // get object type
"strconv" // Convert the Elasticsearch port integer to a string
"strings" // to eval string objects
)

func main() {

// Allow for custom formatting of log output
log.SetFlags(0)

// Use backticks (`) for multiline JSON strings
bulkRequest := `{"index": {"_index": "some_index", "_id": 42}}
{"str field": "Original Value 1", "int field": 1234, "bool field": true}
{"index": {"_index": "some_index", "_id": 123}}
{"str field": "Original Value 2", "int field": 5678, "bool field": false}`

// HTTP 400: "The bulk request must be terminated by a newline [\n]"
bulkRequest = bulkRequest + "n"

// JSON string for indexing a single Elasticsearch document
indexOne := `{"str field": "Original Value 3", "int field": 9012, "bool field": true}`

// Declare an empty string array to store JSON document strings
var docStrings []string

// Append the strings to the array
docStrings = append(docStrings, bulkRequest)
docStrings = append(docStrings, indexOne)

// Iterate over the string array and make API calls
for i, str := range docStrings {

// Print the document's index number in the array
fmt.Printf("index:", i)

// Convert Elasticsearch document string to bytes buffer object for API call
byteSlice := []byte(str)
buf := bytes.NewBuffer(byteSlice)
fmt.Println("bytes buffer for JSON string:", reflect.TypeOf(buf))

// Default HTTP verb is "POST"
verb := "POST"

// Default domain and port for Elasticsearch cluster
requestStr := "http://localhost"
port := 9200

// Convert the port int to a string and concatenate with domain
requestStr = requestStr + ":" + strconv.Itoa(port) + "/"

// Elasticsearch index the documents are created in
ind := "some_index"

// Check if doc string contains "_index" for _bulk API requests
isBulk := strings.Contains(str, `"_index"`)

// Use the _bulk API if JSON doc string contains multiple docs
if isBulk == true {
requestStr = requestStr + "_bulk?pretty=true"
verb = "PUT"
} else {
// The Elasticsearch index must be explicitly declared in header
requestStr = requestStr + ind + "/_doc?pretty=true"
}

// Call the HTTP package's NewRequest() method
req, err := http.NewRequest(verb, requestStr, buf)

// ES 6.0> requires Content-Type header to avoid 406 HTTP error:
// "error":"Content-Type header [] is not supported","status":406}
req.Header.Set("Content-Type", "application/json")

// Print out the HTTP request and check for errors
fmt.Println("nHTTP Request:", req)
if err != nil {
log.Fatalf("http.NewRequest ERROR:", err)
}

// Instantiate a new client object
client := &http.Client{}

// Pass HTTP request to Elasticsearch client and check for errors
resp, err := client.Do(req)
if err != nil {
log.Fatalf("client.Do ERROR:", err)
}

// Close the response body after operations are complete
defer resp.Body.Close()

// Parse out the response body and check for errors
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalf("client.Do ERROR:", err)
}

// Convert the bytes object []uint8 of the JSON response to a string
strBody := string(body)

// Print out the response body
fmt.Println("nresp.Body:", strBody, "nn")

}
}
Rate 1 Star2 Stars3 Stars4 Stars5 Stars

Pilot the ObjectRocket Platform Free!

Try Fully-Managed CockroachDB, Elasticsearch, MongoDB, PostgreSQL (Beta) or Redis.

Get Started
Related Topics:
Additional Information:

Keep in the know!

Subscribe to our emails and we’ll let you know what’s going on at ObjectRocket. We hate spam and make it easy to unsubscribe.