Create a Simple Task Queue with Flask and Redis
Introduction
When it comes to website performance, even a few seconds can be too long to keep your users waiting, especially if you want them to keep coming back to your website. Some tasks take longer to execute than others, and it can make sense to remove the lengthier ones from the typical HTTP request-and-response cycle. That’s where a task queue can be a game changer. With a task queue, you can shift tasks into the queue to be processed later, allowing you to return a response to the user immediately. In this article, we’ll show you how to create your own task queue using Flask and Redis.
Prerequisites
Before we proceed with this tutorial, let’s review a few key prerequisites that need to be in place:
You’ll need to have Python installed on your machine. The latest version of Python 3 is recommended because Python 2 has been deprecated.
You’ll also need to install some packages, and you’ll use the PIP3 package manager to install the Flask micro web framework. Use the command
pip3 -vin the terminal to verify its version number. If you’d like to see which packages are currently installed, use the commandpip3 list.
Install Redis
If you don’t have Redis installed on your machine, you can download and install the server for macOS or Windows. You can also pull Redis as a Docker image in the Docker Hub.
Install Redis in Linux
If you’re running a Debian distribution of Linux like Ubuntu or Linux Mint, the installation process is different. First, you’ll need to update the packages of the index files using the apt-get repository. This will make sure that you install the latest available version of Redis:
1 | sudo apt-get update |
Next, install the redis-server available in the apt repository. Use the command below to install Redis on your machine:
1 | sudo apt-get install redis-server |
Once you’ve installed Redis, you can enable the Redis server and reboot:
1 | sudo systemctl enable redis-server.service |
Install and set up Flask
Our next task will be to install the Flask micro web framework using the pip3 command. We’ll install this in a virtual environment in Python 3. We can start by creating a project directory where we’ll create the virtual environment in Python and then change into that directory:
1 | mkdir flask-redis $$ cd flask-redis |
To create a virtual environment, use the command shown below:
1 | virtualenv venv |
This command will create a directory called venv, which contains the Python library and others.
To start the virtual environment, run the following activation script:
1 | source venv/bin/activate |
Once it’s activated, we’ll be able to use the pip3 Python package manager to install Flask:
1 | pip3 install Flask |
NOTE: You can verify the Flask version that was installed by using the command python -m flask --version. The output will look like this:
Python 3.6.9 Flask 1.1.1 Werkzeug 0.16.0
We can also use pip3 to install Redis as a Python package. This will serve as the database that will store our data:
1 | pip3 install redis |
Next, we’ll install the Redis Queue, which is a library for processing task queues:
1 | pip3 install rq |
Create a task queued file
Now that we’ve installed all the necessary packages, let’s start building the task queue. Using a code editor such as VSCODE, SUBLIME or ATOM, we’ll create a file that will set up our Flask and Redis application:
app.py
We’ll import all of the necessary libraries from the file to create our task queue application:
1 2 3 4 5 6 7 8 9 | # Import the flask module for the application from flask import Flask, request # Import the redis module for the application import redis from rq import Queue # Import the time module to include some time delay in the application import time |
Next, we’ll create a variable named app in Flask. This will be used to set up the Redis instance and our task queue:
1 2 3 4 5 6 | # Create the application directory app = Flask(__name__) # Make a connection of the queue and redis r = redis.Redis() q = Queue(connection=r) |
We then create a function that will handle the task queue:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | # Create a working task queue def task_in_background(t): delay = 1 print("Running Task") print("Simulates the {delay} seconds") time.sleep(delay) print(len(t)) print("Completed Task") return len(t) |
Finally, we’ll create a route for the query string for the len(n) parameter:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | @app.route("/task") def add_task(): if request.args.get("t"): job= que.enqueue(task_in_background, request.args.get("t")) q_len = len(q) return f"The task {job.id} is added into the task queue at {job.enqueued_at}. {q_len} task in the queue" return "Task Queue with Flask" if __name__ == "__main__": app.run() |
Before we run the file that we just created, let’s display the worker process that is separate from the app:
1 | rq worker |
We should see output that looks like the following:
1 2 3 | 14:49:53: Worker rq:worker:697a1e6fca634471867127ad2a224dd5: started, version 1.2.0 14:49:53: *** Listening on default... 14:49:53: Cleaning registries for queue: default |
To run our task queue application, we’ll use the following command in the directory that holds our Python file:
1 | flask run |
This will run on the localhost 127.0.0.1 and append the path /task to the URL to see if the application is running.
Try going to the URL 127.0.0./task?n=hello and refresh your browser. This will create a separate work process which is connected to the Redis queue library.
At this point, the task is running in the background. Our task queue was a success.
Conclusion
Utilizing a task queue can help you manage tasks in the background while providing optimal website performance to your users. In this article, we showed you how to build a simple task queue application using Flask and Redis. With our step-by-step instructions, you’ll be able to follow along and create your own application to queue and manage tasks.
Pilot the ObjectRocket Platform Free!
Try Fully-Managed CockroachDB, Elasticsearch, MongoDB, PostgreSQL (Beta) or Redis.
Get Started