How To Insert Elasticsearch Documents Into An Index Using Golang

Written by Data Pilot
July 08, 2019
  • Elasticsearch
  • Elasticsearch Driver
  • Golang
Have a Database Problem? Speak with an Expert for Free
Get Started >>

Introduction

If you’re storing data in Elasticsearch, you may want to use a Golang script to perform operations such as inserting, updating and deleting documents. Fortunately, it’s easy to execute Elasticsearch operations from Golang with the help of the go-elasticsearch Golang driver. In this article, we’ll provide step-by-step instructions on using the go-elasticsearch driver to index documents in Elasticsearch.

Prerequisites for indexing Elasticsearch documents in Golang

Before we start looking at the code we’ll use to create Elasticsearch documents in Golang, we need to go over a few important system requirements. For this task, the key prerequisites include:

  • The Golang language needs to be installed, and both $GOPATH and $GOROOT need to be exported in your bash profile. You can use the go version and go env commands to confirm that Golang is installed and the correct paths are set.

  • Elasticsearch needs to be installed on the node that is running and compiling the Golang scripts. You can use the Kibana UI or the following cURL request to verify that the Elasticsearch cluster is running:

1
curl -XGET "localhost:9200"
  • The Golang driver for Elasticsearch (go-elasticsearch) must be installed in the server’s $GOPATH. Use git to clone the library’s repository into your $GOPATH, as shown in the example below:
1
git clone --branch master https://github.com/elastic/go-elasticsearch.git $GOPATH/src/github.com/elastic/go-elasticsearch

Create a Go script and import packages

Now that we’ve made sure that everything we need is installed and set up correctly, we can start working on our Go script. Create a new file (with the .go file extension) and put package main at the top. Be sure to import all of the necessary packages and libraries, as seen in the example below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
package main

import (
"context"
"encoding/json"
"fmt"
"log"
"reflect"
"strconv"
"strings"

// Import the Elasticsearch library packages
"github.com/elastic/go-elasticsearch/v8"
"github.com/elastic/go-elasticsearch/v8/esapi"
)

Create a struct data type for the Elasticsearch documents’ fields

We’ll use the Golang struct data type to create a framework for the Elasticsearch documents being indexed and the index’s respective fields:

1
2
3
4
5
6
// Declare a struct for Elasticsearch fields
type ElasticDocs struct {
SomeStr string
SomeInt int
SomeBool bool
}

Declare a function that marshals Elasticsearch struct data into a JSON string

Next, let’s look at a simple function that will convert the Elasticsearch struct document instances into a JSON string. The code shown below may look a bit complicated, but what’s actually happening is pretty simple– all the function does is convert the struct into a string literal, and then that string gets passed to Golang’s json.Marshal() method to have it return a JSON encoding of the string:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// A function for marshaling structs to JSON string
func jsonStruct(doc ElasticDocs) string {

// Create struct instance of the Elasticsearch fields struct object
docStruct := &ElasticDocs{
SomeStr: doc.SomeStr,
SomeInt: doc.SomeInt,
SomeBool: doc.SomeBool,
}

fmt.Println("\ndocStruct:", docStruct)
fmt.Println("docStruct TYPE:", reflect.TypeOf(docStruct))

// Marshal the struct to JSON and check for errors
b, err := json.Marshal(docStruct)
if err != nil {
fmt.Println("json.Marshal ERROR:", err)
return string(err.Error())
}
return string(b)
}

Declare the main() function and create a new Elasticsearch Golang client instance

In our Go script, all of the API method calls need to be inside the main() function or called inside of it from another function. Let’s create a new context object for the API calls and create a map object for the Elasticsearch documents:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
func main() {

// Allow for custom formatting of log output
log.SetFlags(0)

// Create a context object for the API calls
ctx := context.Background()

// Create a mapping for the Elasticsearch documents
var (
docMap map[string]interface{}
)
fmt.Println("docMap:", docMap)
fmt.Println("docMap TYPE:", reflect.TypeOf(docMap))

Instantiate a Elasticsearch client configuration and Golang client instance

In this step, we’ll instantiate a new Elasticsearch configuration object. Make sure you pass the correct host and port information to its Addresses attribute, as well as any username or password:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// Declare an Elasticsearch configuration
cfg := elasticsearch.Config{
Addresses: []string{
"http://localhost:9200",
},
Username: "user",
Password: "pass",
}

// Instantiate a new Elasticsearch client object instance
client, err := elasticsearch.NewClient(cfg)

if err != nil {
fmt.Println("Elasticsearch connection error:", err)
}

Check if the Golang client for Elasticsearch returned any errors when connecting to the cluster

Next, we’ll check to see if the connection to Elasticsearch was successful or if any errors were returned:

1
2
3
4
5
6
7
8
9
// Have the client instance return a response
res, err := client.Info()

// Deserialize the response into a map.
if err != nil {
log.Fatalf("client.Info() ERROR:", err)
} else {
log.Printf("client response:", res)
}

Create Elasticsearch struct documents and put them into an array

We’ll declare an empty string array to store the Elasticsearch documents that we’re currently representing as JSON strings. The code below shows a few Elasticsearch document examples that will be used for indexing. To set their fields’ values, all you need to do is modify the struct instance’s attributes:

1
2
3
4
5
6
7
8
9
10
11
12
13
// Declare empty array for the document strings
var docs []string

// Declare documents to be indexed using struct
doc1 := ElasticDocs{}
doc1.SomeStr = "Some Value"
doc1.SomeInt = 123456
doc1.SomeBool = true

doc2 := ElasticDocs{}
doc2.SomeStr = "Another Value"
doc2.SomeInt = 42
doc2.SomeBool = false

We’ll pass these document instances to the jsonStruct() function we declared earlier, and we’ll have them return JSON strings that represent each of the documents. Then, we’ll use Golang’s append() function to add the JSON strings to the string array:

1
2
3
4
5
6
7
// Marshal Elasticsearch document struct objects to JSON string
docStr1 := jsonStruct(doc1)
docStr2 := jsonStruct(doc2)

// Append the doc strings to an array
docs = append(docs, docStr1)
docs = append(docs, docStr2)

Iterate the Elasticsearch document array and call the Golang client’s IndexRequest() method

Now that we have a document array set up, we’ll iterate over it, making API requests to the Elasticsearch cluster as we go. These API calls will index documents by calling the Golang driver’s esapi.IndexRequest() method:

1
2
3
4
5
6
7
8
9
10
11
12
13
// Iterate the array of string documents
for i, bod := range docs {
fmt.Println("\nDOC _id:", i+1)
fmt.Println(bod)

// Instantiate a request object
req := esapi.IndexRequest{
Index: "some_index",
DocumentID: strconv.Itoa(i + 1),
Body: strings.NewReader(bod),
Refresh: "true",
}
fmt.Println(reflect.TypeOf(req))

Check if the IndexRequest() API method call returned any errors

The last step in our iteration over the document array is to get a response back from the API call and check it for possible errors:

1
2
3
4
5
6
// Return an API response object from request
res, err := req.Do(ctx, client)
if err != nil {
log.Fatalf("IndexRequest ERROR: %s", err)
}
defer res.Body.Close()

In the code shown below, we parse the result object returned by the API response if no errors were returned:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
if res.IsError() {
log.Printf("%s ERROR indexing document ID=%d", res.Status(), i+1)
} else {

// Deserialize the response into a map.
var resMap map[string]interface{}
if err := json.NewDecoder(res.Body).Decode(&resMap); err != nil {
log.Printf("Error parsing the response body: %s", err)
} else {
log.Printf("\nIndexRequest() RESPONSE:")
// Print the response status and indexed document version.
fmt.Println("Status:", res.Status())
fmt.Println("Result:", resMap["result"])
fmt.Println("Version:", int(resMap["_version"].(float64)))
fmt.Println("resMap:", resMap)
fmt.Println("\n")
}
}
}
}

Each document iteration should print out a map[string]interface{} object response that looks like this:

1
resMap: map[_id:4 _index:some_index _primary_term:1 _seq_no:3 _shards:map[failed:0 successful:1 total:2] _type:_doc _version:1 forced_refresh:true result:created]

Conclusion

If you’re a Go developer, you’ll probably want to interact with your Elasticsearch indices using the Golang programming language. With the help of the official Golang driver go-elasticsearch, you can use Elasticsearch and Golang to index documents with a simple script. Using the instructions provided in this tutorial, you’ll be ready to write your own script to create Elasticsearch documents with Golang.

Just the Code

In this article, we’ve reviewed our Golang script one section at a time. The code shown below represents the entire script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
package main

import (
    "context"
    "encoding/json"
    "fmt"
    "log"
    "reflect"
    "strconv"
    "strings"

    // Import the Elasticsearch library packages
    "github.com/elastic/go-elasticsearch/v8"
    "github.com/elastic/go-elasticsearch/v8/esapi"
)

// Declare a struct for Elasticsearch fields
type ElasticDocs struct {
    SomeStr  string
    SomeInt  int
    SomeBool bool
}

// A function for marshaling structs to JSON string
func jsonStruct(doc ElasticDocs) string {

    // Create struct instance of the Elasticsearch fields struct object
    docStruct := &ElasticDocs{
        SomeStr: doc.SomeStr,
        SomeInt: doc.SomeInt,
        SomeBool: doc.SomeBool,
    }

    fmt.Println("\ndocStruct:", docStruct)
    fmt.Println("docStruct TYPE:", reflect.TypeOf(docStruct))

    // Marshal the struct to JSON and check for errors
    b, err := json.Marshal(docStruct)
    if err != nil {
        fmt.Println("json.Marshal ERROR:", err)
        return string(err.Error())
    }
    return string(b)
}

func main() {

    // Allow for custom formatting of log output
    log.SetFlags(0)

    // Create a context object for the API calls
    ctx := context.Background()

    // Create a mapping for the Elasticsearch documents
    var (
        docMap map[string]interface{}
    )
    fmt.Println("docMap:", docMap)
    fmt.Println("docMap TYPE:", reflect.TypeOf(docMap))

    // Declare an Elasticsearch configuration
    cfg := elasticsearch.Config{
        Addresses: []string{
            "http://localhost:9200",
        },
        Username: "user",
        Password: "pass",
    }

    // Instantiate a new Elasticsearch client object instance
    client, err := elasticsearch.NewClient(cfg)

    if err != nil {
        fmt.Println("Elasticsearch connection error:", err)
    }

    // Have the client instance return a response
    res, err := client.Info()

    // Deserialize the response into a map.
    if err != nil {
        log.Fatalf("client.Info() ERROR:", err)
    } else {
        log.Printf("client response:", res)
    }

    // Declare empty array for the document strings
    var docs []string

    // Declare documents to be indexed using struct
    doc1 := ElasticDocs{}
    doc1.SomeStr = "Some Value"
    doc1.SomeInt = 123456
    doc1.SomeBool = true

    doc2 := ElasticDocs{}
    doc2.SomeStr = "Another Value"
    doc2.SomeInt = 42
    doc2.SomeBool = false

    // Marshal Elasticsearch document struct objects to JSON string
    docStr1 := jsonStruct(doc1)
    docStr2 := jsonStruct(doc2)

    // Append the doc strings to an array
    docs = append(docs, docStr1)
    docs = append(docs, docStr2)

    // Iterate the array of string documents
    for i, bod := range docs {
        fmt.Println("\nDOC _id:", i+1)
        fmt.Println(bod)

        // Instantiate a request object
        req := esapi.IndexRequest{
            Index:      "some_index",
            DocumentID: strconv.Itoa(i + 1),
            Body:       strings.NewReader(bod),
            Refresh:    "true",
        }
        fmt.Println(reflect.TypeOf(req))

        // Return an API response object from request
        res, err := req.Do(ctx, client)
        if err != nil {
            log.Fatalf("IndexRequest ERROR: %s", err)
        }
        defer res.Body.Close()

        if res.IsError() {
            log.Printf("%s ERROR indexing document ID=%d", res.Status(), i+1)
        } else {

            // Deserialize the response into a map.
            var resMap map[string]interface{}
            if err := json.NewDecoder(res.Body).Decode(&resMap); err != nil {
                log.Printf("Error parsing the response body: %s", err)
            } else {
                log.Printf("\nIndexRequest() RESPONSE:")
                // Print the response status and indexed document version.
                fmt.Println("Status:", res.Status())
                fmt.Println("Result:", resMap["result"])
                fmt.Println("Version:", int(resMap["_version"].(float64)))
                fmt.Println("resMap:", resMap)
                fmt.Println("\n")
            }
        }

    }
}
Rate 1 Star2 Stars3 Stars4 Stars5 Stars

Pilot the ObjectRocket Platform Free!

Try Fully-Managed CockroachDB, Elasticsearch, MongoDB, PostgreSQL (Beta) or Redis.

Get Started
Related Topics:
Additional Information:

Keep in the know!

Subscribe to our emails and we’ll let you know what’s going on at ObjectRocket. We hate spam and make it easy to unsubscribe.