Task Queues & Background Workers
Task queues are a specialized form of message queues designed for deferred work execution. Instead of processing a request synchronously and making the user wait, you enqueue a task, return immediately, and let a background worker handle the heavy lifting. The user gets a fast response; the work happens behind the scenes.
How It Differs from Message Queues
Task Queues = Message Queues + Execution
A message queue is a generic transport — it moves data between services. A task queue is specifically about scheduling and executing work:
- Message queue: "Here's some data, do whatever you want with it"
- Task queue: "Here's a function to call with these arguments — execute it later"
Task queues typically include: retry logic, priority levels, scheduling, result storage, and worker management — all built in.
Common Use Cases
Perfect Tasks for Background Processing
- Email/SMS sending: Don't make the user wait for SMTP delivery
- Image/video processing: Resize, compress, transcode — CPU-heavy, takes seconds to minutes
- Report generation: Aggregate data, generate PDF — user gets notified when ready
- Data aggregation: Recalculate dashboards, update search indexes
- Webhook delivery: POST to external URLs with retry on failure
- Batch imports: Process uploaded CSV with 100K rows
The Flow: Request → Queue → Worker
Worker Patterns
How Workers Consume Tasks
- Single consumer: One worker processes tasks sequentially. Simple but slow — limited throughput.
- Competing consumers: Multiple workers pull from the same queue. Each task goes to exactly one worker. Scale horizontally by adding more workers.
- Fan-out: One task spawns multiple sub-tasks processed in parallel (e.g., "resize image to 5 different dimensions").
Key principle: With competing consumers, if you have 10 workers and the queue has 1000 tasks, each worker processes ~100 tasks. Need to go faster? Add more workers. This is horizontal scaling for background work.
Priority Queues
Not All Tasks Are Equal
A priority queue ensures urgent tasks are processed before less important ones:
- Critical: Password reset emails — user is waiting right now
- High: Order confirmation emails — user expects within seconds
- Normal: Report generation — user will check back later
- Low: Data aggregation, cleanup jobs — run whenever there's capacity
Implementation: Separate physical queues per priority level. Workers check the critical queue first, then high, then normal, etc. This prevents a flood of low-priority batch work from blocking urgent tasks.
Scheduling: Delayed & Recurring Tasks
- Delayed tasks: "Send this reminder email in 24 hours" — the task sits in the queue with a visibility delay
- Recurring tasks (cron-like): "Generate the daily report every day at 6 AM" — a scheduler enqueues the task on a schedule
- Rate-limited tasks: "Send max 100 API calls per minute to this external service" — workers respect rate limits
Monitoring: Queue Depth Is Your Health Signal
What to Monitor
- Queue depth: How many tasks are waiting? If this grows unbounded, you need more workers.
- Worker lag: Time between a task being enqueued and being picked up. If lag grows, users wait longer.
- Processing time (p50, p95, p99): How long does each task take? Helps identify slow tasks.
- Failure rate: What percentage of tasks fail? Rising failures = bug or downstream issue.
- DLQ depth: Messages that failed all retries. Should be zero in normal operation.
Auto-scaling rule: "If queue depth > 1000 for 5 minutes, add 2 more workers. If queue depth < 10 for 30 minutes, remove a worker." Queue depth is the most reliable signal for scaling background workers.
Popular Task Queue Systems
- Celery (Python): Most popular Python task queue. Uses Redis or RabbitMQ as broker. Powers Instagram, Mozilla.
- Sidekiq (Ruby): Redis-backed, multi-threaded. The standard for Ruby on Rails background jobs. Simple API.
- Bull / BullMQ (Node.js): Redis-backed queue for Node. Supports priorities, scheduling, rate limiting.
- Temporal (language-agnostic): Durable workflow engine for complex multi-step tasks. Handles long-running processes (hours/days) with built-in state management.
Failure Handling
Retries with Exponential Backoff
When a task fails, don't retry immediately — the downstream service might still be down. Use exponential backoff:
- Attempt 1: fails → retry in 1 second
- Attempt 2: fails → retry in 4 seconds
- Attempt 3: fails → retry in 16 seconds
- Attempt 4: fails → retry in 64 seconds
- Attempt 5: fails → move to Dead Letter Queue, alert on-call engineer
Add jitter (random variation) to prevent all retries from hitting the downstream service at the same time (thundering herd).
🏢 Real-World: GitHub Webhook Delivery
When you push code to GitHub, webhooks notify external services (CI/CD, Slack, etc.). This is a massive task queue operation:
- Each push triggers dozens of webhook deliveries to different URLs
- Each delivery is an independent task in a queue
- Workers POST to each webhook URL with the event payload
- If a webhook endpoint is down: retry with exponential backoff (10s, 60s, 5min, 1hr...)
- After multiple failures: mark webhook as failing, back off to hourly retries
- GitHub shows delivery status in the UI: "delivered," "failed (retrying)," "failed (disabled)"
Without task queues, a single slow webhook endpoint would block all other deliveries and slow down the entire push experience.
🏢 Real-World: Instagram Image Processing
When you upload a photo to Instagram:
- Synchronous: Upload the raw image → return success to the user (fast)
- Async tasks enqueued:
- Apply selected filter (GPU-intensive)
- Generate multiple resolutions (thumbnail, feed, full-size)
- Run content moderation (ML model)
- Update follower feeds
- Send push notifications to tagged users
The user sees "uploaded!" in under a second. The heavy processing happens in the background over the next few seconds. Instagram uses Celery with RabbitMQ for this, processing millions of tasks per minute.
Interactive: Task Queue Manager
Add tasks with different priorities and watch workers process them. Trigger failures to see retry behavior: