Install Celery with Redis on VPS - Task Queue Setup

June 2026 · 9 min read

Celery is a distributed task queue framework that enables Python applications to run long-running tasks in the background instead of blocking HTTP requests. Using Redis as a message broker, Celery queues tasks between your web server and worker processes, making websites respond faster while improving scalability. You can easily add more worker instances on the same VPS or across multiple servers. This article covers installing Celery with Redis, along with Flower dashboard for real-time task monitoring.

Celery + Redis Architecture

Web App (Producer): Flask or Django application that queues tasks into Redis without waiting for completion
Redis (Message Broker): In-memory store holding task queues for workers to fetch
Celery Worker (Consumer): Process that listens for tasks from Redis and executes them asynchronously
Flower (Monitor): Web dashboard displaying worker status, task history, and queue backlog in real-time

Install Redis

sudo apt-get update && sudo apt-get install -y redis-server

# Verify Redis service is running
sudo systemctl status redis-server

# Test Redis connection
redis-cli ping
# Should return PONG

Tip: For production use, configure Redis with a password by editing /etc/redis/redis.conf, adding requirepass YOUR_STRONG_PASSWORD and binding to 127.0.0.1 only for security.

Install Celery and Dependencies

# Create a virtual environment for the project
mkdir -p /opt/myapp && cd /opt/myapp
python3 -m venv venv
source venv/bin/activate

# Install Celery, Redis client, and Flower
pip install celery redis flower

# Save requirements for reproducibility
pip freeze > requirements.txt

Create Celery App and Tasks

# Set up project structure
cat > celery_app.py << 'EOF'
from celery import Celery
import redis
import time

# Create Celery instance pointing to Redis
app = Celery(
    'myapp',
    broker='redis://127.0.0.1:6379/0',
    backend='redis://127.0.0.1:6379/1'
)

# Configure Celery
app.conf.update(
    task_serializer='json',
    accept_content=['json'],
    result_serializer='json',
    timezone='UTC',
    enable_utc=True,
    worker_prefetch_multiplier=1,  # Fetch one task at a time
    worker_max_tasks_per_child=1000,  # Restart worker every 1000 tasks
)

# Define tasks
@app.task(name='add_numbers')
def add_numbers(x, y):
    """Simple task - add two numbers"""
    return x + y

@app.task(name='send_email')
def send_email(recipient, subject, body):
    """Email sending task (simulated 5 seconds)"""
    time.sleep(5)  # Simulate actual sending
    print(f"Email sent to {recipient}: {subject}")
    return f"Email sent to {recipient}"

@app.task(name='generate_report')
def generate_report(report_type):
    """Report generation task (simulated 10 seconds)"""
    time.sleep(10)
    return f"Report {report_type} generated"

# Periodic task - runs cron-style every 1 minute
from celery.schedules import crontab

app.conf.beat_schedule = {
    'cleanup-cache': {
        'task': 'cleanup_task',
        'schedule': 60.0,  # Run every 60 seconds
    },
    'midnight-sync': {
        'task': 'sync_data',
        'schedule': crontab(hour=0, minute=0),  # Run at midnight
    },
}

@app.task(name='cleanup_task')
def cleanup_task():
    print("Running cleanup task...")
    return "Cleanup done"

@app.task(name='sync_data')
def sync_data():
    print("Syncing data at midnight...")
    return "Sync done"

if __name__ == '__main__':
    app.start()
EOF

Run Celery Workers

# Run worker with 4 concurrent processes
celery -A celery_app worker --loglevel=info --concurrency=4

# Or run as background systemd service
sudo tee /etc/systemd/system/celery.service << 'EOF'
[Unit]
Description=Celery Worker
After=network.target redis-server.service

[Service]
Type=forking
User=www-data
Group=www-data
WorkingDirectory=/opt/myapp
Environment="PATH=/opt/myapp/venv/bin"
ExecStart=/opt/myapp/venv/bin/celery -A celery_app worker --loglevel=info --concurrency=4 --logfile=/var/log/celery.log --pidfile=/var/run/celery.pid

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl start celery
sudo systemctl enable celery

Tip: For production, use supervisor or systemd socket activation for automatic worker restart on crash and better resource management.

Set Up Flower Monitoring Dashboard

# Run Flower on port 5555
flower -A celery_app --port=5555 --logfile=/var/log/flower.log

# Or create systemd service for Flower
sudo tee /etc/systemd/system/flower.service << 'EOF'
[Unit]
Description=Flower Celery Monitoring
After=network.target

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/opt/myapp
Environment="PATH=/opt/myapp/venv/bin"
ExecStart=/opt/myapp/venv/bin/flower -A celery_app --port=5555 --logfile=/var/log/flower.log

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl start flower
sudo systemctl enable flower

# Access Flower at http://localhost:5555 or behind Nginx
curl http://localhost:5555

Calling Tasks from Flask / Django App

# Flask example
cat > app.py << 'EOF'
from flask import Flask, jsonify
from celery_app import add_numbers, send_email, generate_report

app = Flask(__name__)

@app.route('/add//')
def route_add(x, y):
    # Queue task asynchronously
    result = add_numbers.delay(x, y)
    return jsonify({
        'task_id': result.id,
        'status': 'Task queued',
        'result_url': f'/result/{result.id}'
    })

@app.route('/send_email/')
def route_email(recipient):
    result = send_email.delay(
        recipient=recipient,
        subject='Hello',
        body='This is async email'
    )
    return jsonify({'task_id': result.id})

@app.route('/result/')
def get_result(task_id):
    from celery_app import app as celery_app
    result = celery_app.AsyncResult(task_id)

    return jsonify({
        'task_id': task_id,
        'status': result.status,
        'result': result.result if result.ready() else None
    })

if __name__ == '__main__':
    app.run(debug=True)
EOF

# Test with a request
curl http://localhost:5000/add/10/20
# Returns task_id immediately - no waiting for 10 seconds

Configure Nginx Reverse Proxy for Flower

sudo tee /etc/nginx/sites-available/flower << 'EOF'
server {
    listen 80;
    server_name flower.yourdomain.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name flower.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/flower.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/flower.yourdomain.com/privkey.pem;

    # Basic Auth for protection
    auth_basic "Flower Admin";
    auth_basic_user_file /etc/nginx/.htpasswd;

    location / {
        proxy_pass http://127.0.0.1:5555;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # WebSocket support for real-time updates
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}
EOF

# Create htpasswd for Basic Auth
sudo htpasswd -c /etc/nginx/.htpasswd admin
# Enter password when prompted

sudo ln -s /etc/nginx/sites-available/flower /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

# Set up SSL certificate
certbot --nginx -d flower.yourdomain.com

Celery Beat - Schedule Periodic Tasks

# Create beat service for running periodic tasks
sudo tee /etc/systemd/system/celery-beat.service << 'EOF'
[Unit]
Description=Celery Beat Scheduler
After=network.target redis-server.service

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/opt/myapp
Environment="PATH=/opt/myapp/venv/bin"
ExecStart=/opt/myapp/venv/bin/celery -A celery_app beat --loglevel=info

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl start celery-beat
sudo systemctl enable celery-beat

# Verify beat scheduled tasks
celery -A celery_app inspect scheduled

Verify Celery + Redis Is Working Correctly

# Test Redis connection
redis-cli ping
# Should return PONG

# Check queue depth directly
redis-cli LLEN celery
# Or view all keys in Redis
redis-cli KEYS '*'

# Check Celery worker status
celery -A celery_app inspect active
# Should show worker info and running tasks

# List all registered tasks
celery -A celery_app inspect registered

# View worker statistics (memory, requests, pool size)
celery -A celery_app inspect stats

# Send a test task from command line
python3 -c "from celery_app import add_numbers; result = add_numbers.delay(5, 3); print(f'Task ID: {result.id}')"

# Check task result
python3 -c "from celery_app import app; import sys; result = app.AsyncResult(sys.argv[1]); print(result.get())" TASK_ID

Tip: If worker won't accept tasks, it may be Redis connection failure or worker process crashed. Check systemd log: sudo journalctl -u celery -n 50

Troubleshooting Common Issues

Error: "Connection refused" - Cannot connect to Redis

Redis service is not running or not listening on port 6379.

# Check Redis service status
sudo systemctl status redis-server

# If inactive, start it
sudo systemctl start redis-server

# Verify Redis is listening on port 6379
sudo netstat -tlnp | grep redis
# Should show 127.0.0.1:6379 LISTEN

# Check redis.conf bind address
sudo grep "^bind" /etc/redis/redis.conf
# Should be 127.0.0.1 or 0.0.0.0 (for multi-server setup)

Error: "Unexpected end-of-stream" - Task doesn't return result

Worker process crashed or hung during task execution - result backend didn't receive the result.

# Check worker log for crashes
sudo tail -100 /var/log/celery.log

# Restart worker
sudo systemctl restart celery

# Try reducing concurrency if server resources are limited
# Edit /etc/systemd/system/celery.service
# ExecStart=... --concurrency=2  # Reduce from 4 to 2

# For tasks longer than 5 minutes, increase time limits
# In celery_app.py:
# app.conf.task_time_limit = 3600  # 1 hour
# app.conf.task_soft_time_limit = 3300  # 55 minutes

Error: "Task stuck" - Task queue doesn't decrease

Dead letter queue - tasks the worker cannot process.

# Check Redis queue depth
redis-cli LLEN celery
# If high, tasks are queueing

# Purge all pending tasks (use with caution!)
celery -A celery_app purge
# Answer y to confirm

# Try increasing worker concurrency
celery -A celery_app worker --loglevel=info --concurrency=8

# Monitor task execution times with Flower
# Visit flower.yourdomain.com → Tasks → select task to see execution time

Error: "ModuleNotFoundError" - cannot import celery_app

Virtual environment not activated or Python path is incorrect.

# Verify venv activation
source /opt/myapp/venv/bin/activate
which python3
# Should point to /opt/myapp/venv/bin/python3

# Verify celery_app.py exists
ls -la /opt/myapp/celery_app.py

# Check PYTHONPATH
export PYTHONPATH=/opt/myapp:$PYTHONPATH
celery -A celery_app inspect active

Frequently Asked Questions

Does Celery require Redis or RabbitMQ?

Celery requires a message broker - either Redis or RabbitMQ. Redis is simpler to set up for small projects, but RabbitMQ has more features and better persistence. Redis uses only memory, so messages may be lost if it crashes (unless you enable RDB/AOF persistence). For production on a VPS, Redis works well for less than 1000 tasks/second. Use RabbitMQ if you need higher reliability and ability to scale to many brokers.

Is Flower dashboard secure? Do I need Basic Auth?

Flower exposes all task data and worker status, which is sensitive - it may reveal task arguments, worker IPs, and system configuration. If accessible from the internet, it's a security risk. Always protect it with Basic Auth using htpasswd, SSL client certificates, or keep it behind a corporate firewall. Never expose port 5555 directly to the internet.

Can Celery handle heavy load? How much does it scale?

Yes, Celery scales horizontally by adding more worker instances. The limiting factors are Redis throughput (approximately 100k messages/second) and worker memory usage - each worker typically needs 100-500 MB of RAM depending on task complexity. A 4 GB VPS can comfortably run 8-10 worker processes. For larger scale, split workers across multiple dedicated VPS instances, all connecting to the same Redis broker.

Need a Thai VPS that reliably runs multiple Celery workers? asiagb Thailand VPS provides stable, low-latency hosting ideal for background processing. Audit your network infrastructure first at dnsxray.com.