How To Index Elasticsearch Documents Using The Olivere Golang Driver

Written by Data Pilot
July 22, 2019
  • Elasticsearch
  • Golang
Have a Database Problem? Speak with an Expert for Free
Get Started >>

Introduction

This tutorial will cover how to index Elasticsearch documents using Olivere. The Golang driver for Elasticsearch will support multiple versions of the Elasticsearch program. The default port for Elasticsearch is 9200, but the port can be reconfigured in the Elasticsearch cluster’s elasticsearch.yml file. Be sure to understand the prerequisites for indexing documents in Elasticsearch with the Golang Olivere driver before attempting the exercises in this tutorial.

Prerequisites for using the Olivere Driver for Elasticsearch

  • An Elasticsearch cluster must be successfully running on the server.

  • Make sure the index specified in the code exists on the Elasticsearch cluster of the system operating the Go package.

  • Have sudo access to a mac operating system or Linux server.

NOTE: A private key authorizing sudo access is required to remotely access the server through SSH.

  • Golang must also be installed on the same machine as the Elasticsearch cluster. The $GOROOT and $GOPATH paths must be exported after installation. Type the command go version into a terminal window to determine the currently-installed version of Go.

How to Install the Olivere Golang Package Driver for Elasticsearch

The following command will install the Olivere Go driver for Elasticsearch 7.x:

1
go get gopkg.in/olivere/elastic.v7

If using older versions of the driver, change the 7 at the end of the above command to same number as the major version of the Elasticsearch cluster. If in doubt, execute the following command to obtain the current version of the cluster:

1
curl -XGET localhost:9200

Terminal screenshot using cURL to get Elasticsearch version and go get to install the Olivere driver for Golang

How to Declare an Index Name and Create a Go Struct for the Elasticsearch Documents

As shown in the following script, the const keyword is used to produce a global variable for the name of the Elasticsearch index:

1
2
3
const (
index = "some_index"
)

How to Declare a struct for the Elasticsearch index’s fields

When declaring a struct for the Elasticsearch index’s fields, the JSON string tags used to map to the literal representation of the index’s field names must be included in the command. Execute the following script to declare a Golang struct type for the Elasticsearch index fields and field types:

1
2
3
4
5
6
// Film represents a movie with some properties.
type ElasticIndex struct {
FieldStr string `json:"field str"`
FieldInt int `json:"field int"`
FieldBool bool `json:"field bool"`
}

How to use the Olivere Golang Driver to Connect to the Elasticsearch Cluster

First, create a new client instance of the Olivere package driver’s Client class at the beginning of the main() function with the following script:

1
2
3
4
5
6
7
func main() {
// Instantiate a client instance of the elastic library
client, err := elastic.NewClient(
elastic.SetSniff(true),
elastic.SetURL("http://localhost:9200"),
elastic.SetHealthcheckInterval(5*time.Second), // quit trying after 5 seconds
)

Next, change the URL domain string passed to the SetURL() method parameter to correspond to the same settings of the server.

How to check if the Olivere client returned any errors while connecting to Elasticsearch

Execute the following command:

1
2
3
4
5
6
7
8
9
// Check and see if olivere's NewClient() method returned an error
if err != nil {
fmt.Println("elastic.NewClient() ERROR: %v", err)
log.Fatalf("quiting connection..")
} else {
// Print client information
fmt.Println("client:", client)
fmt.Println("client TYPE:", reflect.TypeOf(client))
}

How to Declare a Context object with a timeout for the Elasticsearch API calls

The following commands will instantiate a context object having a 3 second timeout, causing the object to abort the attempt after three seconds:

1
2
3
4
5
6
7
// Declare a timeout context for the API calls
ctx, stop := context.WithTimeout(context.Background(), 3*time.Second)
defer stop()

// Print client information
fmt.Println("client:", client)
fmt.Println("client TYPE:", reflect.TypeOf(client), "\n")

Alternatively, a simple context object without a timeout can be created with the following declaration:

1
ctx := context.Background()

How to Confirm an Elasticsearch Index Exists

Execute the following script to pass the index string name to the client instance’s IndexExists() method to check if the index already exists on the Elasticsearch cluster:

1
2
3
4
// Check if the Elasticsearch index already exists
exist, err := client.IndexExists(index).Do(ctx)
if err != nil {
log.Fatalf("IndexExists() ERROR: %v", err)

In the event of errors, the system will immediately exit the script after logging the errors.

How to use the Golang Struct for the Elasticsearch Index to Setup Documents

If the index does exist, instantiate a slice for the ElasticIndex struct documents and then add a few documents, for later indexing, by executing the following commands:

1
2
3
4
5
6
7
8
9
// Index some documents if the index exists
} else if exist {

// Instantiate new Elasticsearch documents from the ElasticIndex struct
newDocs := []ElasticIndex{
{FieldStr: "ä½ å¥½ä¸–ç•Œ", FieldInt: 42, FieldBool: true},
{FieldStr: "hello world", FieldInt: 777, FieldBool: false},
{FieldStr: "bonjour monde", FieldInt: 1234, FieldBool: true},
}

How to Index the Elasticsearch Documents with the Olivere Index() Method

Have Elasticsearch dynamically generate an alpha-numeric ID by calling the client object’s Index() method and then declaring an integer variable for the document’s _id field with the following script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// Declare an integer for the doc _id
var id int = 1 // Omit this if you want dynamically generated _id

// Iterate over the docs and index them one-by-one
for _, doc := range newDocs {
_, err = client.Index().
Index(index).
Type("_doc"). // unique doctype now deprecated
BodyJson(doc). // pass struct instance to BodyJson

// Omit this if you want dynamically generated _id
Id(strconv.Itoa(id)). // Convert int to string

Do(ctx) // Initiate API call with context object

The ctx context instance must be passed to the Index() method’s nested Do() method and the doc struct instance should be passed to the BodyJson() method. Now remove the Id() method call to have Elasticsearch generate the document’s _id.

How to check for Index() errors and increment the _id counter

Execute the following script to see if the Index() method call returned any errors and then increment the _id counter with each iteration with id += 1 or id++:

1
2
3
4
5
6
7
8
9
10
11
12
// Check for errors in each iteration
if err != nil {
log.Fatalf("client.Index() ERROR: %v", err)
} else {
fmt.Println("\nElasticsearch document indexed:", doc)
fmt.Println("doc object TYPE:", reflect.TypeOf(doc))
}

// Increment _id counter
id += 1 // Omit this if you want dynamically generated _id

} // end of doc iterator

How to Flush() the Elasticsearch Index and Print the API Response

First, use the client’s Flush() method to free up the server’s memory of the Elasticsearch index. Now pair an else{} statement with the } else if exist { statement block, explained earlier, to check if the index doesn’t exist with the following script:

1
2
3
4
5
_, err = client.Flush(index).WaitIfOngoing(true).Do(ctx)
} else {
fmt.Println("client.Index() ERROR: the index", index, "does not exist")
}
}

When navigating to the project folder in terminal window to index document an Elasticsearch with Golang script, be sure to use the go run command to execute the command. The final output of the index API calls to Elasticsearch should resemble the following:

1
2
3
4
5
6
7
8
Elasticsearch document indexed: {ä½ å¥½ä¸–ç•Œ 42 true}
doc object TYPE: main.ElasticIndex

Elasticsearch document indexed: {hello world 777 false}
doc object TYPE: main.ElasticIndex

Elasticsearch document indexed: {bonjour monde 1234 true}
doc object TYPE: main.ElasticIndex

Verifying the indexed Elasticsearch documents in Kibana

If the Kibana service is running, on default port is 5601, then navigate to the Kibana Console UI under Dev Tools and use the following HTTP request to inspect the indexed documents:

1
GET some_index/_search

Screenshot of Kibana getting Elasticsearch documents indexed in Golang

Conclusion

This tutorial explained how to index an Elasticsearch document using Golang programming language and the Elasticsearch Olivere driver. The article specifically covered how to install the Golang driver for Elasticsearch and how to connect to Elasticsearch with Golang and the Olivere driver. The tutorial also covered how to confirm an Elasticsearch index exists, set up documents, check for index errors, flush the Elasticsearch index and print the API response.

Just the Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
package main

import (
"context" // Context object for Do() methodsd
"fmt" // Format and print cluster data
"log" // Log errors and quit
"reflect" // Get object methods and attributes
"strconv" // Convert _id int to string
"time" // Set a timeout for the connection

// Import the Olivere Golang driver for Elasticsearch
"github.com/olivere/elastic"
)

const (
index = "some_index"
)

// Film represents a movie with some properties.
type ElasticIndex struct {
FieldStr string `json:"field str"`
FieldInt int `json:"field int"`
FieldBool bool `json:"field bool"`
}

func main() {

// Instantiate a client instance of the elastic library
client, err := elastic.NewClient(
elastic.SetSniff(true),
elastic.SetURL("http://localhost:9200"),
elastic.SetHealthcheckInterval(5*time.Second), // quit trying after 5 seconds
)

// Check and see if olivere's NewClient() method returned an error
if err != nil {
fmt.Println("elastic.NewClient() ERROR: %v", err)
log.Fatalf("quiting connection..")
} else {
// Print client information
fmt.Println("client:", client)
fmt.Println("client TYPE:", reflect.TypeOf(client))
}

// Declare a timeout context for the API calls
ctx, stop := context.WithTimeout(context.Background(), 3*time.Second)
defer stop()

// Check if the Elasticsearch index already exists
exist, err := client.IndexExists(index).Do(ctx)
if err != nil {
log.Fatalf("IndexExists() ERROR: %v", err)

// Index some documents if the index exists
} else if exist {

// Instantiate new Elasticsearch documents from the ElasticIndex struct
newDocs := []ElasticIndex{
{FieldStr: "ä½ å¥½ä¸–ç•Œ", FieldInt: 42, FieldBool: true},
{FieldStr: "hello world", FieldInt: 777, FieldBool: false},
{FieldStr: "bonjour monde", FieldInt: 1234, FieldBool: true},
}

// Declare an integer for the doc _id
var id int = 1 // Omit this if you want dynamically generated _id

// Iterate over the docs and index them one-by-one
for _, doc := range newDocs {
_, err = client.Index().
Index(index).
Type("_doc"). // unique doctype now deprecated
BodyJson(doc). // pass struct instance to BodyJson

// Omit this if you want dynamically generated _id
Id(strconv.Itoa(id)). // Convert int to string

Do(ctx) // Initiate API call with context object

// Check for errors in each iteration
if err != nil {
log.Fatalf("client.Index() ERROR: %v", err)
} else {
fmt.Println("\nElasticsearch document indexed:", doc)
fmt.Println("doc object TYPE:", reflect.TypeOf(doc))
}

// Increment _id counter
id += 1 // Omit this if you want dynamically generated _id

} // end of doc iterator

_, err = client.Flush(index).WaitIfOngoing(true).Do(ctx)
} else {
fmt.Println("client.Index() ERROR: the index", index, "does not exist")
}
}
Rate 1 Star2 Stars3 Stars4 Stars5 Stars

Pilot the ObjectRocket Platform Free!

Try Fully-Managed CockroachDB, Elasticsearch, MongoDB, PostgreSQL (Beta) or Redis.

Get Started
Related Topics:
Additional Information:

Keep in the know!

Subscribe to our emails and we’ll let you know what’s going on at ObjectRocket. We hate spam and make it easy to unsubscribe.