Index Multiple Elasticsearch Documents In A JSON File Using Golang

Written by Data Pilot
July 12, 2019
  • Elasticsearch
  • Golang
  • Golang HTTP Client
Have a Database Problem? Speak with an Expert for Free
Get Started >>

Introduction

This tutorial will explain how to index JSON documents with Elasticsearch using Golang’s built-in packages. The JSON encoding library can be used to index Elasticsearch documents with the Golang HTTP client and upload JSON Elasticsearch documents with Golang to index multiple documents. Indexing Elasticsearch documents with Golang can be done with only the built-in Golang "net/http" HTTP client package. No third-party drivers or package libraries are needed to index JSON Elasticsearch documents with Golang.

Prerequisites for using Golang’s HTTP client to index Elasticsearch documents

  • Golang must be installed and the $GOPATH and $GOROOT must be exported in the bash profile. Confirm Golang is installed and the paths are set with the go version and go env commands.

  • Elasticsearch must be installed on the node running and compiling the Golang scripts. Verify the Elasticsearch cluster is running with the Kibana UI or by executing the following cURL request:

1
curl -XGET "localhost:9200"
  • Make sure to have a JSON file with some Elasticsearch documents in the same directory as the Golang script making the API calls to the Elasticsearch cluster. Confirm the project directory is in the $GOPATH directory.

How to Import the Required Golang Packages for Indexing Elasticsearch Documents

Declare package main at the top of the script and then import all of the necessary Golang packages for indexing Elasticsearch documents from a JSON file with the following commands:

1
2
3
4
5
6
7
8
9
10
11
package main

import (
"bytes" // for converting JSON to bytes array
"fmt" // for printing to console
"io/ioutil" // for reading IO of JSON file
"log" // for logging errors
"net/http" // for making HTTP requests
"os" // for opening JSON file
"reflect" // get object type
)

How to Create a JSON File with Elasticsearch Documents in Valid JSON Format

Make sure there is a JSON file in the same directory as the $GOPATH, the Go project and this Go script. The _bulk API requires the file to have a .json file extension and the documents be in a valid JSON format.

As shown below, each document must have an "index" line before each document body that specifies the index name and _id:

1
{"index": {"_index": "my_index", "_id": 1}}

NOTE: Omitting the "_id" field will result in Elasticsearch dynamically generating an alpha-numeric _id for the document at the time of indexing.

Sample Elasticsearch documents used for testing purposes.

The sample Elasticsearch documents used for this tutorial are as follows:

1
2
3
4
5
6
{"index": {"_index": "some_index", "_id": 42}}
{"str field": "Original Value 1", "int field": 1234, "bool field": true}
{"index": {"_index": "some_index", "_id": 123}}
{"str field": "Original Value 2", "int field": 5678, "bool field": false}
{"index": {"_index": "some_index", "_id": 777}}
{"str field": "Original Value 3", "int field": 9012, "bool field": true}

How to Open a JSON File at the Beginning of the Main Function

Use the following command to declare the main() function and use the os package’s Open() method to have Golang return the raw string data from the JSON file:

1
2
3
4
5
6
7
8
9
10
11
12
func main() {

// Allow for custom formatting of log output
log.SetFlags(0)

// Use the OS package to load the JSON file
file, err := os.Open("docs.json")
if err != nil {
log.Fatalf("os.Open() ERROR:", err)
}
// Close the file AFTER operations are complete
defer file.Close()

Make sure to use the defer statement and call the JSON file’s Close() method after all of the operations are complete to free up the memory used by the JSON file.

How to use Golang’s ReadAll() Method to Create a Byte Slice from the JSON String

Pass the raw JSON string data to the ioutil.ReadAll() method and have it return a byte slice []uint8 of the JSON document data that will be used in executing the HTTP request as follows:

1
2
3
4
5
6
7
8
// Call ioutil.ReadAll() to create a bytes array from file's JSON data
byteSlice, err := ioutil.ReadAll(file)

// Check for IO errors
if err != nil {
log.Fatalf("ioutil.ReadAll() ERROR:", err)
}
fmt.Println("bytesStr TYPE:", reflect.TypeOf(byteSlice), "n")

How to Make an HTTP Request in Golang to Index the JSON Documents to Elasticsearch

Make a new HTTP request using Golang’s http package and pass the JSON data byte slice to the bytes.NewBuffer() method parameter with the following script:

1
2
3
4
5
6
7
8
9
10
11
12
13
// Make HTTP request using "PUT" or "POST" verb
req, err := http.NewRequest("PUT", "http://localhost:9200/_bulk?pretty=true", bytes.NewBuffer(byteSlice))

// ES 6.0> requires Content-Type header to avoid 406 HTTP error:
// "error":"Content-Type header [] is not supported","status":406}
req.Header.Set("Content-Type", "application/x-ndjson")

// Print out the HTTP request and check for errors
if err != nil {
log.Fatalf("http.NewRequest ERROR:", err)
} else {
fmt.Println("HTTP Request:", req)
}

NOTE: If using Elasticsearch 6.0 or later, then the request’s header’s Content-Type (using the request object’s Header.Set() method) must be set or else Elasticsearch will return a 406 HTTP response.

How to Declare a Golang HTTP Client Instance for the Elasticsearch Response

Execute the following command to instantiate a client instance of the Golang HTTP package library and obtain the Elasticsearch HTTP response by passing the request to the Do() method:

1
2
3
4
5
6
7
8
9
10
11
// Instantiate a new client object
client := &http.Client{}

// Pass HTTP request to Elasticsearch client and check for errors
resp, err := client.Do(req)
if err != nil {
log.Fatalf("client.Do() ERROR:", err)
}

// Close the response body after operations are complete
defer resp.Body.Close()

The ioutil.ReadAll() method call will return a byte slice of Elasticsearch’s response to the index API request. The slice can be explicitly typecast as a string by passing it to string() with the following script:

1
2
3
4
5
6
7
8
9
10
11
12
// Parse out the response body and check for errors
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalf("client.Do ERROR:", err)
}

// Convert the bytes object []uint8 of the JSON response to a string
strBody := string(body)

// Print out the response body
fmt.Println("nresp.Body:", strBody)
}

Finally, make sure to print out the response. If the index call was a success Elasticsearch should return a successful JSON response with "result" : "created" nested in the JSON object, as show here:

Screenshot of a JSON response from Elasticsearch after an index API call using Golang

Conclusion

This tutorial covered how to index Elasticsearch documents with the Golang HTTP client package. Examples were provided that showed how to index Elasticsearch documents with Golang, upload JSON Elasticsearch documents with Golang and index JSON documents with Elasticsearch. When indexing JSON Elasticsearch documents with Golang using Elasticsearch version 6.0 or later it is important to remember that the request’s header’s Content-Type must be set or Elasticsearch will return a 406 HTTP response.

Just the Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
package main

import (
"bytes" // for converting JSON to bytes array
"fmt" // for printing to console
"io/ioutil" // for reading IO of JSON file
"log" // for logging errors
"net/http" // for making HTTP requests
"os" // for opening JSON file
"reflect" // get object type
)

func main() {

// Allow for custom formatting of log output
log.SetFlags(0)

// Use the OS package to load the JSON file
file, err := os.Open("docs.json")
if err != nil {
log.Fatalf("os.Open() ERROR:", err)
}
// Close the file AFTER operations are complete
defer file.Close()

// Call ioutil.ReadAll() to create a bytes array from file's JSON data
byteSlice, err := ioutil.ReadAll(file)

/*
CONTENTS OF JSON FILE:
{"index": {"_index": "some_index", "_id": 42}}
{"str field": "Original Value 1", "int field": 1234, "bool field": true}
{"index": {"_index": "some_index", "_id": 123}}
{"str field": "Original Value 2", "int field": 5678, "bool field": false}
{"index": {"_index": "some_index", "_id": 777}}
{"str field": "Original Value 3", "int field": 9012, "bool field": true}
*/

// Check for IO errors
if err != nil {
log.Fatalf("ioutil.ReadAll() ERROR:", err)
}
fmt.Println("bytesStr TYPE:", reflect.TypeOf(byteSlice), "n")

// Make HTTP request using "PUT" or "POST" verb
req, err := http.NewRequest("PUT", "http://localhost:9200/_bulk?pretty=true", bytes.NewBuffer(byteSlice))

// ES 6.0> requires Content-Type header to avoid 406 HTTP error:
// "error":"Content-Type header [] is not supported","status":406}
req.Header.Set("Content-Type", "application/x-ndjson")

// Print out the HTTP request and check for errors
if err != nil {
log.Fatalf("http.NewRequest ERROR:", err)
} else {
fmt.Println("HTTP Request:", req)
}

// Instantiate a new client object
client := &http.Client{}

// Pass HTTP request to Elasticsearch client and check for errors
resp, err := client.Do(req)
if err != nil {
log.Fatalf("client.Do
ERROR:", err)
}

// Close the response body after operations are complete
defer resp.Body.Close()

// Parse out the response body and check for errors
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalf("client.Do ERROR:", err)
}

// Convert the bytes object []uint8 of the JSON response to a string
strBody := string(body)

// Print out the response body
fmt.Println("nresp.Body:", strBody)
}
Rate 1 Star2 Stars3 Stars4 Stars5 Stars

Pilot the ObjectRocket Platform Free!

Try Fully-Managed CockroachDB, Elasticsearch, MongoDB, PostgreSQL (Beta) or Redis.

Get Started
Related Topics:
Additional Information:

Keep in the know!

Subscribe to our emails and we’ll let you know what’s going on at ObjectRocket. We hate spam and make it easy to unsubscribe.