How To Bulk Insert Multiple Documents To A MongoDB Collection Using Python

Introduction to insertMany() in PyMongo

If you’re using MongoDB to store data, there will be times when you want to insert multiple documents into a collection at once. It’s easy to accomplish this task with Python when you use the collection.insert_many() method. This useful method is PyMongo’s version of the insertMany() method; both methods can be used to insert more than one document at a time. In this article, we’ll show how to use the PyMongo bulk insert method to index MongoDB documents in Python.

Prerequisites

Before we look at any Python code, it’s important to review the prerequisites for the task. There are a few key system requirements to consider:

• It’s helpful to have a working knowledge of Python and its syntax. The examples in this article will use containers such as iterators and generators, so you’ll need to be comfortable with these concepts.

• You’ll need to have Python installed. Python 3 is recommended since Python 2 is now deprecated.

• You’ll also need to install the PyMongo low-level Python driver for MongoDB using the PIP package manager:

• If you haven’t done so already, you’ll need to install the MongoDB server application. You can check to see what version of MongoDB is running by using the mongo --version command in a terminal; you could also enter the Mongo shell in a terminal by typing mongo and pressing Return.

• You will need to have a MongoDB collection you can use to test your PyMongo API calls. Since you’ll be bulk inserting data, it’s best to stick to a test collection where there are no concerns about possible data loss or corruption.

Import Modules and Connect to the MongoDB Server using the MongoClient class

Once you’ve made sure that all the system requirements are in place, it’s time to dive into the Python code. The first step is to import the MongoClient class, and create a new client instance of the driver:

We’ll be using the mongo_client object to make method calls to the MongoDB database and its collections

At this point, you can also import any other Python libraries or modules that may be necessary; for example, we’ll import the JSON library for Python that can be used to create JSON strings or to parse JSON strings using its json.loads() method:

All of the libraries shown above come included with the Python installation, so there’s no need to install any of them with PIP.

Using Python dictionaries in PyMongo

In Python, a dictionary object acts as the equivalent of a JSON or BSON MongoDB document. This basic data structure can easily be created using curly brackets {}, as in the following example:

A Python dictionary is made up of key-value pairs of data. To access a specific key in a Python dictionary, you simply call the object name followed by square brackets [] containing the key’s name in string form. In this example, our dictionary is named "simple_dict", and the specific key we’re accessing is named "key". We store the value of that key in "some_value":

Constructing a MongoDB document using a Python dictionary

Every key-value pair in a Python dictionary object can translate to a MongoDB field-value pair when it’s used to index documents. Much like a JSON object, the Python dictionary uses commas (,) to separate the key-value pairs.

Dictionaries can be constructed on a single line, like in the following example:

If the dictionary is going to be quite long, or if it contains lengthy strings, it’s helpful to use multiple lines when declaring the key-value pairs:

Adding the MongoDB documents to a list to be inserted later

The PyMongo insert_one() method only needs a single dict type object be passed to it during the API call, but the insert_many() method requires that the dictionary objects be nested inside a Python list ([]) for the method call to work. Just passing a single dictionary to the method will result in a TypeError exception. Here’s an example of what not to do:

PyMongo raises a TypeError exception if a Python list isn’t passed to the insert_many() method

Put the MongoDB document dictionary objects into a Python list

There are different ways to append, index, or put items into a Python dictionary. Our next examples will illustrate these different options.

Use the += operator to add an item to the MongoDB document list

You can use the += operator to append an element to a list object. Be sure to enclose the MongoDB document object in square brackets, as seen in the example below:

Use the Python List methods append() and insert() to add MongoDB documents

The append() method for List-type objects can also be used to add an element to the end of a Python list:

The insert() method (which is not to be confused with the MongoDB Collection’s insert() method), however, is a bit different from the two previous methods we saw. With insert(), you can specify the position in the list where you want to insert the item. This method requires two arguments to be passed—- the first must be an integer representing the index location where the object will be inserted, and the second is the object itself:

If the integer value passed to the insert() method’s index parameter exceeds the number of elements in the list, then the object will simply be indexed as the last element in the list.

Declare the MongoDB documents and the list at the same time

In addition to all of the options we’ve just reviewed, another option is to declare the MongoDB document dictionaries and the list at the same time. To accomplish this, we just pass it all into the method call at once, as shown below:

The list object is now ready to be passed to the PyMongo insert many method.

Use an iterator to parse multiple MongoDB documents and append them to a list

Now that we’ve seen how to add a MongoDB document to a list, let’s look at an example of how to iterate over an existing list of values in order to create another Python list comprised of MongoDB documents.

We’ll assume that there was a list of various fruits in the form of a Python list, and these fruits needed to be inserted into a MongoDB collection as documents:

Iterate over the Python list to create dictionary objects for the MongoDB documents

First, we’ll need to create another empty list. This will serve both as the container for the documents, and as the parameter to be passed in the method call:

Next, we’ll use that list of fruits to build a new list of MongoDB dictionary objects by creating a new dictionary object with each iteration. We could just use a simple for loop, but the enumerate() function is more efficient:

Pass objects, like datetime.now(), to the MongoDB dictionary document

You can add more key-value pairs to a Python dictionary to give the MongoDB documents more content. In the following example, we’ll create a random integer (using the random.randint() method call) to generate some additional data for the fruit. We’ll use a random choice of either 0 or 1 to determine if each fruit is labeled as “good” or “bad, then we’ll use the value of datetime.now() to create a timestamp for the fruit. Finally, we’ll create a UUID for each fruit:

The final step is to put the dictionary object into the mongo_docs list that we created earlier, and then pass that list to the insert_many() method once we finish iterating over the fruits list:

Iterate over a list of items, call the PyMongo bulk index method insert_many(), and verify the inserted documents in MongoDB Compass

Create a MongoDB document list from a JSON string

If your script is running on the server side of a website, a more common way to build a list of MongoDB dictionary objects would be by parsing an HTTP message string. The Python JSON library makes this task simple, as we’ll see in the next example.

Inserting MongoDB documents in the form of an HTTP Message JSON string from a POST request

For this example, we’ll assume the client made an HTTP request to the MongoDB server and the message request came in the form of a JSON string, which will be indexed as MongoDB documents:

At first, this string looks a bit difficult to parse. Fortunately, the json.loads() method call can convert that JSON string into a Python dictionary that can then be inserted into a MongoDB collection:

If the method call to json.loads() doesn’t raise any exceptions, it means that the returned object is a Python dictionary. This dictionary can then be parsed and finally passed to the PyMongo index_many() method:

Using a JSON string and PyMongo to insert_many() documents, and then verify the insertion with MongoDB Compass

Have the PyMongo API call return a “results” object for the inserted documents

The PyMongo API call can also return a pymongo.results.InsertManyResult object. This helps you verify that the insertion was successful, and gives you more information on what changes were made to the MongoDB collection.

To get this information, have the method call return an object when calling it:

Then, take the returned result object and access its various attributes. One attribute is the inserted_ids list, which lists all the IDs that were created for the documents upon insertion. Use Python’s len() function to see the total number of documents inserted:

Conclusion

When you need to insert a batch of documents into a MongoDB collection, creating a Python script is a good way to get the job done. The PyMongo driver makes it easy to bulk insert MongoDB documents with Python. With the step-by-step instructions provided in this article, you’ll have no trouble performing a MongoDB bulk insert in a Python script.

Throughout this article, we examined the code one section at a time. The following Python script contains all of the examples in this article and can be used to insert_many() MongoDB documents into a collection: