Data Processing using MongoDB MapReduce

Introduction

There has been so much talk lately about MapReduce and how it can help developers process large data sets into useful aggregated results. In this tutorial, we are going to show you some examples using the MapReduce() function on a MongoDB collection. MapReduce is broken down into two phases:

• The map phase filters out the data and transforms it so that it can be processed.
• The reduce phase performs the aggregations, or analysis over that data.

All mapReduce() functions in MongoDB take the documents within a single collection and performs a query of the data. During implementation the map phase is always executed before the reduce phase. In the map phase, data is collected and transformed into key/value pairs. Finally, during the reduce phase, that data will be collected from the generated map, then condensed/combined (or reduced) to an aggregated set of data that will later be stored in a collection. Let’s see how it works!

Prerequisites

For the following examples, we will be using MongoDB shell to execute commands.

• You should have MongoDB installed
• It’s not required but it’s recommended that you have some previous shell experience with MongoDB.
• You should have access to the MongoDB shell ( Execute mongo )
• Start the Mongo Daemon (in another terminal window execute mongod)

MapReduce Syntax

This syntax is common across MongoDB examples, with the exception of the optional arguments. Let’s look at what’s available and what they mean:

1. The out refers to the location of the results. You can output to a collection or inline.
2. The query refers to the selection criteria that will be used for query operators to determine the document input.
3. The sort is the sort criteria to sort the input documents.
4. The limit sets the maximum number of documents to be used as input into the map function.

While there are more options, these are the most commonly used.

Executing the MapReduce Function

Now that we understand how mapReduce works and have some basic knowledge of the syntax, let’s see it in action! Make sure that you have the Mongo Daemon running in another terminal and set up your database. Here we are using a small student db:

Define the map() function

In our example, the map function should produce key/value pairs, where the key will be the student’s name and the value is the grade. Once this is complete, the function will emit a key/value pair. Enter this in the Mongo shell:

Define the reduce() function

The reduce function will group all the key/value pairs and return the sum of all the values. In this specific example, it will find the names of each student and sum the grades from all their classes. Enter this in the Mongo shell:

Note: You won’t see any validation responses after entering these two functions in the shell.

Execute the mapReduce() function

Once we have defined both our map and reduce functions, we can run the query and declare the collection where it will store the aggregated data.

The mapReduce query was successful and our output shows that a collection was created. When we view the contents of the collection, we see that the reduce() function calculated the sum of all the grades for each student.

Conclusion

The mapReduce function is commonly used for aggregating large sets of data. This example was a brief introduction into setting up both the map and reduce functions, in order to perform a simple mapReduce() query. The map function generated key/value pairs from the original data, then the reduce function performed the calculation using the key/value data. Once the mapReduce function was executed, it stored that calculated data into a collection. Now that you are familiar with how it works, it’s time to try it on a larger cluster of data!