How To Index Elasticsearch Documents Using The Olivere Golang Driver
Introduction
This tutorial will cover how to index Elasticsearch documents using Olivere. The Golang driver for Elasticsearch will support multiple versions of the Elasticsearch program. The default port for Elasticsearch is 9200
, but the port can be reconfigured in the Elasticsearch cluster’s elasticsearch.yml
file. Be sure to understand the prerequisites for indexing documents in Elasticsearch with the Golang Olivere driver before attempting the exercises in this tutorial.
Prerequisites for using the Olivere Driver for Elasticsearch
An Elasticsearch cluster must be successfully running on the server.
Make sure the index specified in the code exists on the Elasticsearch cluster of the system operating the Go package.
Have
sudo
access to a mac operating system or Linux server.
NOTE: A private key authorizing sudo
access is required to remotely access the server through SSH.
- Golang must also be installed on the same machine as the Elasticsearch cluster. The
$GOROOT
and$GOPATH
paths must be exported after installation. Type the commandgo version
into a terminal window to determine the currently-installed version of Go.
How to Install the Olivere Golang Package Driver for Elasticsearch
The following command will install the Olivere Go driver for Elasticsearch 7.x:
1 | go get gopkg.in/olivere/elastic.v7 |
If using older versions of the driver, change the 7
at the end of the above command to same number as the major version of the Elasticsearch cluster. If in doubt, execute the following command to obtain the current version of the cluster:
1 | curl -XGET localhost:9200 |
How to Declare an Index Name and Create a Go Struct for the Elasticsearch Documents
As shown in the following script, the const
keyword is used to produce a global variable for the name of the Elasticsearch index:
1 2 3 | const ( index = "some_index" ) |
How to Declare a struct for the Elasticsearch index’s fields
When declaring a struct for the Elasticsearch index’s fields, the JSON string tags used to map to the literal representation of the index’s field names must be included in the command. Execute the following script to declare a Golang struct
type for the Elasticsearch index fields and field types:
1 2 3 4 5 6 | // Film represents a movie with some properties. type ElasticIndex struct { FieldStr string `json:"field str"` FieldInt int `json:"field int"` FieldBool bool `json:"field bool"` } |
How to use the Olivere Golang Driver to Connect to the Elasticsearch Cluster
First, create a new client instance of the Olivere package driver’s Client class at the beginning of the main()
function with the following script:
1 2 3 4 5 6 7 | func main() { // Instantiate a client instance of the elastic library client, err := elastic.NewClient( elastic.SetSniff(true), elastic.SetURL("http://localhost:9200"), elastic.SetHealthcheckInterval(5*time.Second), // quit trying after 5 seconds ) |
Next, change the URL domain string passed to the SetURL()
method parameter to correspond to the same settings of the server.
How to check if the Olivere client returned any errors while connecting to Elasticsearch
Execute the following command:
1 2 3 4 5 6 7 8 9 | // Check and see if olivere's NewClient() method returned an error if err != nil { fmt.Println("elastic.NewClient() ERROR: %v", err) log.Fatalf("quiting connection..") } else { // Print client information fmt.Println("client:", client) fmt.Println("client TYPE:", reflect.TypeOf(client)) } |
How to Declare a Context object with a timeout for the Elasticsearch API calls
The following commands will instantiate a context
object having a 3
second timeout, causing the object to abort the attempt after three seconds:
1 2 3 4 5 6 7 | // Declare a timeout context for the API calls ctx, stop := context.WithTimeout(context.Background(), 3*time.Second) defer stop() // Print client information fmt.Println("client:", client) fmt.Println("client TYPE:", reflect.TypeOf(client), "\n") |
Alternatively, a simple context object without a timeout can be created with the following declaration:
1 | ctx := context.Background() |
How to Confirm an Elasticsearch Index Exists
Execute the following script to pass the index string name to the client instance’s IndexExists()
method to check if the index already exists on the Elasticsearch cluster:
1 2 3 4 | // Check if the Elasticsearch index already exists exist, err := client.IndexExists(index).Do(ctx) if err != nil { log.Fatalf("IndexExists() ERROR: %v", err) |
In the event of errors, the system will immediately exit the script after logging the errors.
How to use the Golang Struct for the Elasticsearch Index to Setup Documents
If the index does exist, instantiate a slice for the ElasticIndex
struct documents and then add a few documents, for later indexing, by executing the following commands:
1 2 3 4 5 6 7 8 9 | // Index some documents if the index exists } else if exist { // Instantiate new Elasticsearch documents from the ElasticIndex struct newDocs := []ElasticIndex{ {FieldStr: "ä½ å¥½ä¸–ç•Œ", FieldInt: 42, FieldBool: true}, {FieldStr: "hello world", FieldInt: 777, FieldBool: false}, {FieldStr: "bonjour monde", FieldInt: 1234, FieldBool: true}, } |
How to Index the Elasticsearch Documents with the Olivere Index() Method
Have Elasticsearch dynamically generate an alpha-numeric ID by calling the client object’s Index()
method and then declaring an integer variable for the document’s _id
field with the following script:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | // Declare an integer for the doc _id var id int = 1 // Omit this if you want dynamically generated _id // Iterate over the docs and index them one-by-one for _, doc := range newDocs { _, err = client.Index(). Index(index). Type("_doc"). // unique doctype now deprecated BodyJson(doc). // pass struct instance to BodyJson // Omit this if you want dynamically generated _id Id(strconv.Itoa(id)). // Convert int to string Do(ctx) // Initiate API call with context object |
The ctx
context instance must be passed to the Index()
method’s nested Do()
method and the doc
struct instance should be passed to the BodyJson()
method. Now remove the Id()
method call to have Elasticsearch generate the document’s _id
.
How to check for Index() errors and increment the _id counter
Execute the following script to see if the Index()
method call returned any errors and then increment the _id
counter with each iteration with id += 1
or id++
:
1 2 3 4 5 6 7 8 9 10 11 12 | // Check for errors in each iteration if err != nil { log.Fatalf("client.Index() ERROR: %v", err) } else { fmt.Println("\nElasticsearch document indexed:", doc) fmt.Println("doc object TYPE:", reflect.TypeOf(doc)) } // Increment _id counter id += 1 // Omit this if you want dynamically generated _id } // end of doc iterator |
How to Flush() the Elasticsearch Index and Print the API Response
First, use the client’s Flush()
method to free up the server’s memory of the Elasticsearch index. Now pair an else{}
statement with the } else if exist {
statement block, explained earlier, to check if the index doesn’t exist with the following script:
1 2 3 4 5 | _, err = client.Flush(index).WaitIfOngoing(true).Do(ctx) } else { fmt.Println("client.Index() ERROR: the index", index, "does not exist") } } |
When navigating to the project folder in terminal window to index document an Elasticsearch with Golang script, be sure to use the go run
command to execute the command. The final output of the index API calls to Elasticsearch should resemble the following:
1 2 3 4 5 6 7 8 | Elasticsearch document indexed: {ä½ å¥½ä¸–ç•Œ 42 true} doc object TYPE: main.ElasticIndex Elasticsearch document indexed: {hello world 777 false} doc object TYPE: main.ElasticIndex Elasticsearch document indexed: {bonjour monde 1234 true} doc object TYPE: main.ElasticIndex |
Verifying the indexed Elasticsearch documents in Kibana
If the Kibana service is running, on default port is 5601
, then navigate to the Kibana Console UI under Dev Tools and use the following HTTP request to inspect the indexed documents:
1 | GET some_index/_search |
Conclusion
This tutorial explained how to index an Elasticsearch document using Golang programming language and the Elasticsearch Olivere driver. The article specifically covered how to install the Golang driver for Elasticsearch and how to connect to Elasticsearch with Golang and the Olivere driver. The tutorial also covered how to confirm an Elasticsearch index exists, set up documents, check for index errors, flush the Elasticsearch index and print the API response.
Just the Code
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | package main import ( "context" // Context object for Do() methodsd "fmt" // Format and print cluster data "log" // Log errors and quit "reflect" // Get object methods and attributes "strconv" // Convert _id int to string "time" // Set a timeout for the connection // Import the Olivere Golang driver for Elasticsearch "github.com/olivere/elastic" ) const ( index = "some_index" ) // Film represents a movie with some properties. type ElasticIndex struct { FieldStr string `json:"field str"` FieldInt int `json:"field int"` FieldBool bool `json:"field bool"` } func main() { // Instantiate a client instance of the elastic library client, err := elastic.NewClient( elastic.SetSniff(true), elastic.SetURL("http://localhost:9200"), elastic.SetHealthcheckInterval(5*time.Second), // quit trying after 5 seconds ) // Check and see if olivere's NewClient() method returned an error if err != nil { fmt.Println("elastic.NewClient() ERROR: %v", err) log.Fatalf("quiting connection..") } else { // Print client information fmt.Println("client:", client) fmt.Println("client TYPE:", reflect.TypeOf(client)) } // Declare a timeout context for the API calls ctx, stop := context.WithTimeout(context.Background(), 3*time.Second) defer stop() // Check if the Elasticsearch index already exists exist, err := client.IndexExists(index).Do(ctx) if err != nil { log.Fatalf("IndexExists() ERROR: %v", err) // Index some documents if the index exists } else if exist { // Instantiate new Elasticsearch documents from the ElasticIndex struct newDocs := []ElasticIndex{ {FieldStr: "ä½ å¥½ä¸–ç•Œ", FieldInt: 42, FieldBool: true}, {FieldStr: "hello world", FieldInt: 777, FieldBool: false}, {FieldStr: "bonjour monde", FieldInt: 1234, FieldBool: true}, } // Declare an integer for the doc _id var id int = 1 // Omit this if you want dynamically generated _id // Iterate over the docs and index them one-by-one for _, doc := range newDocs { _, err = client.Index(). Index(index). Type("_doc"). // unique doctype now deprecated BodyJson(doc). // pass struct instance to BodyJson // Omit this if you want dynamically generated _id Id(strconv.Itoa(id)). // Convert int to string Do(ctx) // Initiate API call with context object // Check for errors in each iteration if err != nil { log.Fatalf("client.Index() ERROR: %v", err) } else { fmt.Println("\nElasticsearch document indexed:", doc) fmt.Println("doc object TYPE:", reflect.TypeOf(doc)) } // Increment _id counter id += 1 // Omit this if you want dynamically generated _id } // end of doc iterator _, err = client.Flush(index).WaitIfOngoing(true).Do(ctx) } else { fmt.Println("client.Index() ERROR: the index", index, "does not exist") } } |
Pilot the ObjectRocket Platform Free!
Try Fully-Managed CockroachDB, Elasticsearch, MongoDB, PostgreSQL (Beta) or Redis.
Get Started