Chapter 8 Interview: Async Processing

So basically, my mental model is: does the user need the result RIGHT NOW to continue their workflow? If yes, sync. If they can move on and get notified later, async.

Sync makes sense for things like authentication checks, reading data they're about to display, validating a form submission — the user is literally waiting for that response to know what to do next.

Async is for everything that's downstream of the user's immediate need. They clicked "place order" — they need confirmation that the order was accepted (sync), but they don't need to wait for the warehouse notification, the receipt email, the analytics event, or the inventory update. All of that can happen in the background.

The other big signal is latency tolerance. If an operation takes more than a second or two — processing a video, generating a report, sending to a third-party API that might be slow — make it async. Return immediately with "we're working on it" and notify when done.

And reliability is a factor too. If a downstream service might be down, async with a queue gives you retry capability. Sync means you fail when they fail. Async decouples your availability from theirs.

• User waiting? If they need the result to continue → sync
• Downstream work: notifications, analytics, integrations → async
• Latency: anything > 1-2s should be async with status polling/webhooks
• Reliability: async decouples you from downstream failures

Alright, so the key word here is "reliable" — meaning every email that should go out actually goes out, even if things fail along the way.

I'd start with a queue-based architecture. When your application needs to send an email, it doesn't call the SMTP provider directly. It writes a message to a durable queue — something like SQS or RabbitMQ. That write is your commitment: "this email will be sent."

Then you have worker processes consuming from that queue. They pick up a message, call the email provider (SendGrid, SES, whatever), and only acknowledge the message after successful delivery. If the provider is down or returns an error, the message stays in the queue for retry.

For retry logic, I'd use exponential backoff — wait 1 minute, then 5, then 15, then 60. After some max attempts (maybe 5-6), move it to a dead letter queue for manual inspection. You don't want to retry forever — the email might be to an invalid address.

Idempotency is critical. What if your worker crashes after sending but before acknowledging? The message gets redelivered. So you need a deduplication check — maybe store a hash of (recipient + template + timestamp) and skip if already sent. Nobody wants duplicate emails.

And for observability — track every email through states: queued → processing → sent → delivered/bounced. Dashboard showing queue depth, send rate, failure rate. Alert if the queue grows faster than workers can drain it.

• Queue-based: write to durable queue, workers consume and send
• Retry with backoff: exponential delays, dead letter queue after max attempts
• Idempotency: dedup check to prevent duplicate sends on redelivery
• State tracking: queued → processing → sent → delivered/bounced
• Monitoring: queue depth, send rate, failure rate, alerts on backlog

The way I see it, the fundamental difference is the consumption model. A message queue is like a to-do list — once a message is processed, it's gone. An event stream is like a log — events are appended and stay there, and multiple consumers can read them independently.

With a queue (RabbitMQ, SQS), you have competing consumers. If three workers are listening, each message goes to exactly ONE of them. Once acknowledged, it's removed. This is perfect for work distribution — "process this image," "send this email." You want it done once.

With a stream (Kafka, Kinesis), events are published to a topic and STAY there. Multiple consumer groups can each read all events independently, at their own pace. The notification service reads order events AND the analytics service reads the same events AND the billing service. They don't interfere with each other.

Streams also give you replay — if you deploy a buggy consumer that misprocessed events, you can reset its offset and reprocess from any point in time. With a queue, once consumed, it's gone forever. That replayability is incredibly valuable for debugging and recovery.

So basically: queue for task distribution (do this work once), stream for event broadcasting (multiple systems react to the same events). Different tools, different problems.

• Queue: competing consumers, message consumed once, then deleted
• Stream: append-only log, multiple independent consumers, messages persist
• Queue use case: work distribution — process once by any available worker
• Stream use case: event broadcasting — many systems react independently
• Replay: streams allow reprocessing from any point; queues don't

So there's a spectrum of failure handling, and honestly the right approach depends on whether the failure is transient or permanent.

For transient failures — network blip, downstream service momentarily overloaded — you retry. But not immediately and not forever. Exponential backoff with jitter: first retry after 1s (plus random 0-500ms), then 4s, then 16s, up to some cap. The jitter prevents thundering herd when a service comes back and all retries fire simultaneously.

After N retries (I usually set 3-5 depending on the operation), the message goes to a dead letter queue (DLQ). This is your safety net. Failed messages sit there for inspection — you can look at them, figure out why they failed, fix the bug, and replay them. Never silently drop messages.

For permanent failures — invalid data, business logic violations — retrying won't help. You want to detect these fast (maybe on first attempt) and route them directly to the DLQ or an error handling path. Don't waste retry cycles on something that will never succeed.

I'd also add alerting on the DLQ size. If it's growing, something systemic is wrong. And implement a "replay" mechanism — a tool or script that can take DLQ messages and re-inject them into the main queue after you've fixed the underlying issue.

• Retry with backoff: exponential delays + jitter for transient failures
• Dead letter queue: catch-all for messages that exhaust retries
• Classify failures: transient (retry) vs permanent (DLQ immediately)
• Replay mechanism: tool to re-inject DLQ messages after fix
• Alerting: monitor DLQ depth, alert on growth

Okay, so a notification system needs to handle multiple channels (push, email, SMS, in-app), user preferences, and high throughput without annoying users. Let me walk through how I'd structure this.

At the top, you have notification triggers — events from various services. "Order shipped," "friend request received," "payment failed." These publish to a central notification event stream. I'd use Kafka here because multiple downstream systems need to react.

Next, a notification orchestrator service consumes these events and decides: what channels should this go to? It checks user preferences (did they opt out of SMS?), suppression rules (don't send more than 3 push notifications per hour), and priority (payment failures always get through).

Then it fans out to channel-specific queues — one for push, one for email, one for SMS. Each channel has its own workers that handle the specifics: formatting templates, calling provider APIs (Firebase for push, SendGrid for email, Twilio for SMS).

For in-app notifications, I'd write directly to a notifications table/store and push via WebSocket if the user is online. Unread count, mark-as-read, that whole UX. Separate from the async external channels.

Rate limiting per-user is important — nobody wants 50 notifications in an hour. Batch similar notifications ("3 people liked your post" instead of 3 separate notifs). And always, always have a way for users to control their preferences granularly.

• Event-driven: services publish events, notification service reacts
• Orchestrator: checks preferences, suppression rules, priority
• Channel fanout: separate queues + workers per channel (push/email/SMS)
• In-app: persistent store + WebSocket for real-time delivery
• User control: per-channel preferences, rate limiting, batching