Chapter 9 Interview: Consistency & Consensus

So CAP theorem says that in a distributed system, when a network partition happens — and it WILL happen — you have to choose between consistency and availability. You can't have both during the partition.

Let me break that down practically. Consistency means every read gets the most recent write. Availability means every request gets a response (even if it might be stale). Partition tolerance means the system keeps working even when network links between nodes break.

The way I see it, partition tolerance isn't really optional — networks fail, that's reality. So the real choice is: during a network split, do you return errors to maintain consistency (CP), or do you return potentially stale data to stay available (AP)?

Real systems: a banking ledger is CP — you'd rather reject a transaction than risk inconsistency. A social media feed is AP — showing a slightly stale timeline is way better than showing an error page. Most systems aren't purely one or the other though — you can make different trade-offs per feature within the same system.

Honestly, the more useful framing these days is PACELC — which adds: even when there's NO partition, do you optimize for latency or consistency? Because that trade-off matters every single request, not just during rare partition events.

• CAP: during partition, choose Consistency (reject requests) or Availability (serve stale)
• Partition tolerance: not optional — networks fail in practice
• CP examples: banking, inventory counts, leader election
• AP examples: social feeds, DNS, shopping cart
• PACELC: extends CAP — what do you optimize when there's NO partition?

Okay so honestly, my first answer is: try really hard not to need them. Distributed transactions are painful. If you can redesign your service boundaries so a transaction stays within one service, do that instead.

But when you genuinely need coordination across services — like an order that must reserve inventory AND charge payment AND create a shipment — you have a few patterns.

The Saga pattern is my go-to. Instead of one big transaction, you have a sequence of local transactions, each in its own service. If step 3 fails, you run compensating transactions to undo steps 1 and 2. Like: charge payment → reserve inventory → create shipment. If shipment fails, you release the inventory reservation and refund the payment.

There are two saga flavors: choreography (each service emits events and others react) and orchestration (a central coordinator directs the sequence). I prefer orchestration for complex flows because the logic is in one place and easier to reason about. Choreography works for simpler 2-3 step flows.

Two-phase commit (2PC) exists but I'd avoid it in microservices. It's blocking, requires all participants to be available simultaneously, and a coordinator failure can leave everything locked. It's fine within a single database but doesn't scale across services.

• Avoid if possible: redesign boundaries to keep transactions local
• Saga pattern: sequence of local transactions + compensating actions on failure
• Orchestration vs choreography: central coordinator vs event-driven reactions
• 2PC: blocking, fragile — avoid across microservices
• Compensation: design every action with its undo/rollback counterpart

For a shopping cart, I'd go with eventual consistency — and here's why that's actually the right call even though it sounds scary.

Think about the user experience. If someone adds an item to their cart from their phone, and then opens their laptop 2 seconds later and it's not there yet — is that a disaster? Not really. They'll refresh or it'll sync momentarily. The cost of strong consistency (higher latency, reduced availability) isn't worth it for a cart.

Amazon literally wrote the Dynamo paper about this. Their shopping cart uses eventual consistency because availability matters more. A customer who can't add to cart is lost revenue. A customer who sees a slightly stale cart for a moment is mildly inconvenienced.

For conflict resolution — what if they add an item on phone and remove it on laptop simultaneously? — I'd use "last write wins" with vector clocks, or better yet, use a CRDT-style approach where adds and removes are both preserved and merged. The merge strategy: if in doubt, keep the item in the cart. It's better to show something extra (they can remove it) than to lose items they wanted.

BUT — and this is important — at checkout time, you switch to strong consistency. When they click "buy," you do a consistent read of inventory, lock the items, charge the payment. The cart is eventually consistent; the purchase is strongly consistent.

• Eventual consistency: availability > consistency for cart operations
• Conflict resolution: merge-friendly — prefer keeping items over losing them
• Amazon's approach: Dynamo paper — always-writable cart
• Checkout switches: strong consistency at purchase time for inventory/payment

So Raft is a consensus algorithm designed to be understandable — that was literally its design goal after Paxos confused everyone. It ensures a cluster of nodes agrees on a sequence of operations even if some nodes fail.

The core idea: one node is the Leader, the rest are Followers. All writes go through the Leader. The Leader appends the operation to its log, replicates it to Followers, and once a majority (quorum) acknowledges, it's committed. Only then does it respond to the client.

Leader election is the interesting part. Each node has a randomized election timeout. If a Follower doesn't hear from the Leader within that timeout, it assumes the Leader is dead, increments the term number, and starts an election by asking other nodes for votes. You need a majority to win. The randomized timeouts prevent everyone from trying to become leader simultaneously.

The log replication ensures all nodes process the same operations in the same order. If a Follower's log diverges from the Leader's (maybe it was partitioned), the Leader detects this and sends the missing entries. The Leader's log is always authoritative.

The way I think about it — Raft guarantees that as long as a majority of nodes are up, the cluster can make progress. If you have 5 nodes, you can tolerate 2 failures. Three nodes, one failure. That's your availability guarantee.

• Leader-based: one Leader handles all writes, replicates to Followers
• Quorum commits: majority must acknowledge before write is committed
• Leader election: randomized timeouts, term numbers, majority vote to win
• Log replication: Leader's log is authoritative, Followers converge to it
• Fault tolerance: N nodes tolerates (N-1)/2 failures

The way I frame this decision: what's the cost of reading stale data? If stale data causes financial loss, safety issues, or broken invariants — strong consistency. If stale data causes mild UX awkwardness — eventual is fine.

Strong consistency use cases: bank account balances (can't show wrong balance and allow overdraft), inventory counts for the last item in stock (don't sell what you don't have), user authentication state (if they change their password, old sessions must be invalid NOW), and anything with uniqueness constraints (username registration).

Eventual consistency use cases: social media feeds (seeing a post 2 seconds late is fine), like counts (off by one momentarily? nobody cares), user profiles (name change propagating in 5 seconds is acceptable), search indexes (slightly stale results are okay), and analytics dashboards (real-time isn't actually real-time anyway).

Honestly, most features in most applications can tolerate eventual consistency. We over-apply strong consistency out of fear. The key insight is that strong consistency has real costs — higher latency (coordinator round-trips), lower availability (need quorum), and reduced throughput. You're paying those costs on every operation.

My approach: default to eventual, upgrade to strong only where the business requires it. And be specific about the consistency window — "eventual" could mean 50ms or 5 minutes. Define your SLA.

• Strong when: financial data, inventory, auth state, uniqueness constraints
• Eventual when: feeds, counters, profiles, search, analytics
• Cost of strong: higher latency, lower availability, reduced throughput
• Default eventual: upgrade to strong only where business demands it
• Define the window: "eventual" needs an SLA — 100ms? 5s? 1min?