How to Create a Web Scraper with Mongoose, NodeJS, Axios, and Cheerio - Part 3

Introduction

In this third part of the tutorial on how to build a webscraper from scratch we will concentrate on how to save the data we’ve scraped into a Mongo Database. We will be utilizing the npm packages Mongoose which is a library that takes a model based approach to creating data.

If you’re just joining us this is multi-part series we’re in the midst of creating a web scraper from scratch. It will store the data it has scraped using MongoDB. The backend will be written in Javascript running on NodeJS. There are several libraries we will use on the backend to make our lives easier including Express, Axios, and Cheerio. On the front-end we will use Bootstrap for styling and JQuery to send requests to the backend. Finally we will deploy the site using Heroku.

In the first part of the series we got a basic app that served our homepage running using nodejs, express, and bootstrap. In the second part, we built a function that will do the webscraping using axios and cheerio. Now now we’ll show you how to store the data you’ve scraped in a Mongo Database.

Install and require Mongoose

You’ll need a driver to interact with Mongo from NodeJS and there are a few out there including mongodb and mongojs but we’ll be using one called Mongoose that takes a schema/model based approach that we feel is pretty intuitive. It takes a little more work to get Mongoose working but we feel it pays off in the end.

First let’s install the library with npm:

Next we’ll require it in our app.js:

Connect to Mongo

Then we’ll use this code to connect to a Mongo with a new database called idioms_db:

Note: Don’t get thrown by the process.env code, this will only come into play when we deploy our app.

Run your app again and make sure you don’t get any errors.

Create a Schema

Here comes the hard part about Mongoose. When we want to save data through Mongoose we’ll have to do so through a Model which is created from a Schema. A schema just describes what kind of data we are storing. Let’s create a models folder to put our schema in:

Now let’s create a schema for the idioms called idioms.js:

In idioms.js we’ll first need to require Mongoose. Then we need to think about what data we want to store as an idiom. We’ve decided that we just want a string of the idiom and a link to it’s page. The string should be required but the link should not be. Here’s how we create that schema in Mongoose:

File: webscraperDemo/models/idioms.js

Notice at the end of the code how we create a Model from our schema and then export that Model so we can use it in app.js.

Create a route to save idioms

Now to prove that we are actually interacting with the database we will create a route that will scrape for idioms by a term and save them to our database. This is a long chunk of code so we’ll post it and then dissect it afterwards:

File: /webscraperDemo/app.js `js // Route to go out and scrape for idioms that have an entered string app.post(“/idioms/scrape/:searchTerm”, function(req, res) {

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
scrape(req.params.searchTerm)

.then(function(foundIdioms) {

    console.log("scraped:");

    console.log(foundIdioms);

    // Save scraped Idioms

    foundIdioms.forEach(function(eachIdiom) {

        Idiom.create(eachIdiom)

        .then(function(savedIdiom) {

            // If saved successfully, print the new Idiom document to the console

            console.log(savedIdiom);

        })

        .catch(function(err) {

            // If an error occurs, log the error message

            console.log(err.message);

        });

    });

    



    res.json(foundIdioms);

})

.catch(function(err) {

    res.json(err);

});

}); `

This code creates a route so that your front-end can send a request to http://localhost:3000/idioms/scrape/light and it will use our scrape() function from earlier to scrape the idioms and then save them in the database.

In order for this to work a couple more things need to happen. You’ll notice in the index.html from earlier that we linked to an index.js that we never mentioned. We’ll we’re gonna use it here to make requests to our backend. We won’t cover the details now, just now that that is it’s basic purpose. Use this code and put it in your public folder.

In our app.js file we’ll also need to require Mongoose and import the Model:

If you’re lost on all the changes we’ll share the finished repository at the end of this tutorial so just try to follow along.

Now let’s try to hit that route from the front end by entering “light” into the “Scrape Idioms by Term” field and pressing “Scrape”. You should see a table of idioms populate at the bottom. That means the scraping is working! Creating that table is handled by the index.js. In the next section when we flesh out the rest of the routes we’ll verify data is being stored by creating a route that fetches all the idioms in the database.

Conclusion of Part 3

In this third part of the tutorial we connected with Mongo and created a schema and a model that allowed us to create data in our database. In Part 4 we’ll create more routes that add functionality like retrieving all the idioms in our database and deleting items.

Check out Part 1

Check out Part 2

Check out Part 3

Check out Part 4

Check out Part 5