Building a Real-World Application with Celery
Building a Real-World Application with Celery
In this lesson, we will apply our knowledge of Celery to build a complete real-world application. The focus will be on creating a task queue for a simple web application that processes user-uploaded images. This application will demonstrate how to use Celery for asynchronous task processing, allowing us to handle image uploads without blocking the user interface.
Learning Objectives
By the end of this lesson, you will be able to: - Understand the architecture of a real-world application using Celery. - Create and configure a Flask web application. - Implement Celery tasks for processing images. - Handle task results and monitor task execution. - Apply best practices for using Celery in a production environment.
Understanding the Application Architecture
Before we dive into coding, let’s outline the architecture of our application. We will build a Flask web application that allows users to upload images. Once an image is uploaded, a Celery task will be triggered to process the image (e.g., resizing, filtering, etc.).
The architecture can be visualized as follows:
flowchart TD
A[User Uploads Image] --> B[Flask App]
B --> C[Celery Task Queue]
C --> D[Worker Processes Image]
D --> E[Store Processed Image]
E --> F[Notify User]
F --> A
Step 1: Setting Up the Flask Application
First, we need to set up our Flask application. If you haven’t already, install Flask using pip:
pip install Flask
Now, create a new directory for your project and create a file named app.py:
from flask import Flask, request, redirect, url_for, flash, render_template
import os
app = Flask(__name__)
app.secret_key = 'supersecretkey'
app.config['UPLOAD_FOLDER'] = 'uploads/'
@app.route('/')
def index():
return render_template('index.html')
if __name__ == '__main__':
app.run(debug=True)
This code initializes a Flask application and sets the upload folder for images. The index route renders an HTML template where users can upload images.
Step 2: Creating the HTML Template
Next, create a directory named templates in the same folder as your app.py file. Inside templates, create a file named index.html:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Image Upload</title>
</head>
<body>
<h1>Upload an Image</h1>
<form action="/upload" method="post" enctype="multipart/form-data">
<input type="file" name="file">
<input type="submit" value="Upload">
</form>
</body>
</html>
This HTML form allows users to select an image file and submit it to the server for processing.
Step 3: Handling File Uploads
Now, we will add a route to handle the file uploads in app.py:
from celery import Celery
app.config['CELERY_BROKER_URL'] = 'redis://localhost:6379/0'
app.config['CELERY_RESULT_BACKEND'] = 'redis://localhost:6379/0'
celery = Celery(app.name, broker=app.config['CELERY_BROKER_URL'])
celery.conf.update(app.config)
@celery.task
def process_image(file_path):
# Here you can add image processing logic, e.g., resizing or filtering
return f'Processed {file_path}'
@app.route('/upload', methods=['POST'])
def upload():
if 'file' not in request.files:
flash('No file part')
return redirect(url_for('index'))
file = request.files['file']
if file.filename == '':
flash('No selected file')
return redirect(url_for('index'))
file_path = os.path.join(app.config['UPLOAD_FOLDER'], file.filename)
file.save(file_path)
process_image.delay(file_path)
flash('File uploaded and processing started!')
return redirect(url_for('index'))
In this code:
- We configure Celery with Redis as the broker and result backend.
- We define a Celery task called process_image that will handle the image processing logic.
- The /upload route handles the file upload and triggers the Celery task using process_image.delay(file_path), which sends the task to the queue without blocking the user.
Step 4: Running the Application
To run your application, make sure you have Redis running on your machine. You can start Redis with the following command:
redis-server
Then, run your Flask application:
python app.py
Next, in another terminal, start the Celery worker:
celery -A app.celery worker --loglevel=info
This command starts a Celery worker that listens for tasks in the queue.
Step 5: Implementing Image Processing Logic
Now, let’s implement some basic image processing logic within the process_image task. For this, you will need to install the Pillow library:
pip install Pillow
Then, modify the process_image function:
from PIL import Image
@celery.task
def process_image(file_path):
try:
with Image.open(file_path) as img:
img = img.resize((800, 800)) # Resize image
processed_file_path = os.path.join(app.config['UPLOAD_FOLDER'], 'processed_' + os.path.basename(file_path))
img.save(processed_file_path)
return f'Processed {file_path}'
except Exception as e:
return str(e)
In this updated function:
- We use the Pillow library to open the uploaded image and resize it to 800x800 pixels.
- The processed image is saved with a new filename prefixed with processed_.
- Any exceptions during processing are caught and returned as a string.
Step 6: Handling Task Results
To handle task results, you can modify the upload route to check the status of the task:
from celery.result import AsyncResult
@app.route('/task_status/<task_id>')
def task_status(task_id):
task_result = AsyncResult(task_id)
return {'task_id': task_result.id, 'status': task_result.status, 'result': task_result.result}
This route allows you to check the status of a task by its ID. You can use this in your frontend to notify users when their image has been processed.
Best Practices
- Use environment variables to store sensitive configurations such as secret keys and database URLs.
- Handle exceptions within your Celery tasks to prevent worker crashes and ensure that logs are informative.
- Monitor your Celery workers using tools like Flower to track task execution and performance.
- Optimize task execution by grouping related tasks when possible to reduce overhead.
Common Mistakes and How to Avoid Them
- Not starting the Celery worker: Always ensure your worker is running; otherwise, tasks will not be processed.
- Forgetting to configure the broker: Ensure that your Celery configuration matches your message broker settings.
- Blocking the main thread: Use
delay()orapply_async()to ensure tasks run asynchronously.
Key Takeaways
- Celery is a powerful tool for handling asynchronous tasks in Python applications.
- Integrating Celery with Flask allows for responsive applications that can handle long-running tasks.
- Proper configuration and monitoring are essential for production-ready applications.
In this lesson, we successfully built a simple image processing application using Celery and Flask. We have covered the essential steps from setting up the application to implementing image processing tasks.
Next Steps
In the next lesson, we will explore future trends in distributed task queues, including advancements in technology and methodologies that can enhance task queue systems. Stay tuned for insights into the evolving landscape of distributed systems!
Exercises
- Exercise 1: Modify the image processing function to apply a filter (e.g., grayscale) to the uploaded images before saving them.
- Exercise 2: Create a new route that allows users to view the status of their image processing tasks using the task status route.
- Exercise 3: Implement a feature that allows users to download the processed images after they are done processing.
- Exercise 4: Set up a more complex workflow where multiple image processing tasks are chained together (e.g., resize, then apply a filter).
- Practical Assignment: Build a complete web application that allows users to upload multiple images, process them in parallel, and display the results on a dashboard with real-time status updates using Celery and Flask.
Summary
- Celery allows for asynchronous task processing in Python applications.
- Flask can be integrated with Celery for building responsive web applications.
- Proper task management and error handling are crucial for robust applications.
- Image processing can be efficiently handled with Celery tasks.
- Monitoring and optimizing Celery tasks enhances application performance.