1 Why Offline Metrics Lie to You

Every model in this course has been evaluated offline: a held-out test set, a fixed metric (accuracy, AUC, RMSE, BLEU), a single number that says "this model is better than that model." That number is necessary but never sufficient. It answers a narrower question than the one the business actually cares about, and it is computed under conditions that production will not respect.

The Offline/Online Gap

Three structural reasons explain why a model that wins offline can lose online:

• Metric mismatch. Offline you optimize a proxy (AUC for click prediction); online the business cares about revenue, retention, or time-to-resolution. A recommendation model can improve click-through rate while decreasing session length, because it surfaces clickbait-y items users regret clicking. The proxy and the true objective are correlated, not identical — and a new model can exploit the gap between them.
• Selection effects. Your offline test set is a fixed, historical sample. The moment a model goes live, its own predictions change what data gets collected next: a fraud model that blocks certain transactions will never see what those transactions "would have" looked like, and a recommender that stops showing a category starves itself of feedback about that category. This is a feedback loop offline evaluation structurally cannot see.
• Distribution shift in the wild. The Lesson 68 drift story applies here too — production traffic mix, user behavior, and even adversarial responses (bots, gaming the algorithm) differ from the static historical distribution the offline test set was carved from. A model 2 AUC points better on last quarter's data is not guaranteed to be 2 points better on next week's live traffic.

The Online Controlled Experiment

An A/B test (also called an online controlled experiment, OCE) solves this by measuring the model where it actually has to perform: in front of real users, making real decisions, under real feedback loops — while using randomization to make sure any measured difference is attributable to the model change and nothing else.

2 Designing the Experiment: Hypothesis and Metrics

Before writing a single line of traffic-splitting code, an A/B test needs a written design — exactly like the Scope of Work from Lesson 9, but scoped to one specific comparison.

State a Falsifiable Hypothesis

"The new model is better" is not a hypothesis — it's a hope. A real hypothesis names the metric, the direction, and (ideally) the expected magnitude: "Replacing the v3 ranking model with v4 will increase 7-day click-through rate (CTR) on the recommendations module by at least 2 percentage points, without increasing p99 inference latency by more than 20ms."

Primary Metric vs Guardrail Metrics

Every experiment needs exactly one primary metric — the number that decides whether the test is a win. Pick more than one and you invite "metric shopping": running the test, then hunting through ten metrics until one is significant by chance. Alongside the primary metric, define guardrail metrics — things that must not get worse even if the primary metric improves.

The guardrails matter because a model can win on the metric it was optimized for while quietly damaging something the business depends on — a latency-heavy model that boosts CTR by 3% but adds 300ms to every request will increase abandonment rate enough to net negative on revenue.

3 The Randomization Unit: Why User-Level Beats Session or Request

An A/B test randomly assigns subjects to control (existing model) or treatment (new model). The crucial design decision is: what is the unit of randomization?

• Request-level: every single API call/page view is independently randomized into control or treatment. Cheap to implement, but a single user might be served the old model on one request and the new model on the next — they directly experience the inconsistency, and any effect that depends on a coherent experience across multiple interactions (e.g., a recommendation feed that should feel personalized) gets contaminated.
• Session-level: a user is randomized once per session, consistent for that session, but can flip to the other arm in their next session. Better than request-level, but a user can still accumulate exposure to both arms over time, diluting any measured difference and making metrics that span multiple sessions (7-day retention) unreliable.
• User-level: a user is assigned to one arm (typically via a hash of their user ID) and stays there for the entire duration of the experiment. This is the default correct choice for the vast majority of ML A/B tests.

User-level randomization avoids contamination: the treatment arm's effect bleeding into the control arm's measurement (or vice versa) because the same person experienced both conditions. It also lets you measure metrics that need a consistent identity over time, like multi-day retention or lifetime value, which session- or request-level splits cannot support.

import hashlib

def assign_variant(user_id: str, experiment_name: str, treatment_fraction: float = 0.5) -> str:
    """
    Deterministic, stable user-level bucketing using a hash.
    The same user_id + experiment_name always maps to the same variant,
    for the lifetime of the experiment, with no database lookup required.
    """
    # Salt the hash with the experiment name so a user's bucket in one
    # experiment is independent of their bucket in another experiment.
    key = f"{experiment_name}:{user_id}".encode("utf-8")
    digest = hashlib.sha256(key).hexdigest()
    bucket = int(digest[:8], 16) / 0xFFFFFFFF   # uniform float in [0, 1)

    return "treatment" if bucket < treatment_fraction else "control"


# Demonstration: bucket 10 users into a 50/50 split
user_ids = [f"user_{i}" for i in range(10)]
for uid in user_ids:
    variant = assign_variant(uid, experiment_name="ranker_v4_test")
    print(f"  {uid:10s} -> {variant}")

# Same user, same experiment, called again later -> identical result (stability)
print("\nRe-checking user_3 (must match):",
      assign_variant("user_3", experiment_name="ranker_v4_test"))

4 Hypothesis Testing From First Principles

Lesson 7 introduced probability distributions and the normal distribution. A/B testing builds directly on that foundation. Here is the minimum statistical machinery needed to read and trust an experiment's result.

Null and Alternative Hypotheses

Every test starts by assuming nothing changed: the null hypothesis H₀ states that the treatment and control have the same true conversion rate (or mean). The alternative hypothesis H₁ states they differ. The test never "proves" H₁ — it only measures how surprising the observed data would be if H₀ were actually true, and decides whether that surprise is large enough to reject H₀.

The p-value

The p-value is the probability of observing a difference at least as extreme as the one you measured, assuming the null hypothesis is true. A small p-value (conventionally p < 0.05) means: "if there were truly no difference between control and treatment, data this extreme would be rare" — so we reject H₀ and call the result statistically significant. A p-value is not the probability that H₁ is true, and it is not the probability you've made a mistake by shipping. It is a statement about how unlikely your data is, conditional on the boring explanation being correct.

Significance Level (α) and Confidence Intervals

You choose, in advance, a significance level α (typically 0.05) — the rate of false positives you are willing to tolerate if the null hypothesis is actually true. A 95% confidence interval for the difference in conversion rates is the range of values that are consistent with the observed data: if you repeated the experiment many times, about 95% of such intervals would contain the true difference. If a 95% CI for the lift excludes zero, that's equivalent to rejecting H₀ at α = 0.05 — and the CI additionally tells you the plausible magnitude of the effect, which a p-value alone does not.

Statistical Power and the Two Types of Error

Statistical power is 1 − β: the probability of correctly detecting a real effect of a given size, if it exists. Conventionally you design experiments to have power ≥ 0.80 — an 80% chance of detecting the effect you care about, if it's really there. Low-powered tests are dangerous specifically because a "no significant difference" result from an underpowered test is uninformative — you may simply not have collected enough data to see an effect that is genuinely there.

5 Sample Size and Power Calculation, Worked From Scratch

Before launching a test you must answer: "how many users do I need per arm?" Run too few and an underpowered test will mask a real win. Run far more than necessary and you waste time and expose more users than needed to a possibly-worse experience.

The Formula, For a Two-Proportion Test

Suppose the baseline conversion rate is p₁ and you want to detect a lift to p₂ = p₁ + Δ (the minimum detectable effect, MDE). For a two-sided test at significance level α with power 1 − β, the required sample size per arm is approximately:

n ≈ [ (z_α/2 · √(2p̄(1−p̄)) + z_β · √(p₁(1−p₁) + p₂(1−p₂)))² ] / Δ²

where p̄ = (p₁+p₂)/2 is the pooled rate, z_α/2 is the critical z-value for your significance level (1.96 for α=0.05 two-sided), and z_β is the critical z-value for your desired power (0.84 for 80% power). The intuition: required sample size grows with the variance of the metric and the precision (α, power) you demand, and shrinks quadratically as the effect size Δ you're trying to detect gets larger — small effects are expensive to detect reliably.

import numpy as np
from scipy import stats

def sample_size_two_proportions(p1: float, mde: float, alpha: float = 0.05,
                                 power: float = 0.80) -> int:
    """
    Required sample size PER ARM to detect a lift from p1 to p1+mde,
    using a two-sided two-proportion z-test.
    """
    p2 = p1 + mde
    p_bar = (p1 + p2) / 2

    z_alpha = stats.norm.ppf(1 - alpha / 2)   # two-sided critical value
    z_beta  = stats.norm.ppf(power)           # one-sided power requirement

    numerator = (
        z_alpha * np.sqrt(2 * p_bar * (1 - p_bar)) +
        z_beta  * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))
    ) ** 2
    n = numerator / (mde ** 2)
    return int(np.ceil(n))


# Worked example: current add-to-cart rate is 8%. We want to detect
# a lift to 8.8% (a +0.8 percentage point / 10% relative improvement),
# with the standard alpha=0.05, power=0.80.
baseline_rate = 0.08
mde           = 0.008   # absolute lift we want to be able to detect

n_per_arm = sample_size_two_proportions(baseline_rate, mde, alpha=0.05, power=0.80)
print(f"Baseline rate:           {baseline_rate:.1%}")
print(f"Minimum detectable lift: {mde:.1%} (absolute)")
print(f"Required users per arm:  {n_per_arm:,}")
print(f"Total experiment size:   {2 * n_per_arm:,}")

# Sensitivity: smaller effects need dramatically more users
print("\nHow sample size scales with the effect you want to detect:")
for mde_test in [0.004, 0.008, 0.016, 0.032]:
    n = sample_size_two_proportions(baseline_rate, mde_test)
    print(f"  MDE = {mde_test:.1%}  ->  n/arm = {n:,}")

Notice the relationship is quadratic: halving the effect size you want to detect roughly quadruples the required sample size. This is why "just A/B test everything" is harder than it sounds — detecting a genuinely small but real improvement (which is the common case once a product matures) requires a lot of traffic and a lot of patience.

Translating Sample Size to Calendar Time

# Given the required sample size, how long must the test run?
daily_eligible_users = 4_000   # users per day who qualify for this experiment
traffic_per_arm_fraction = 0.5  # 50/50 split

required_n_per_arm = n_per_arm
daily_users_per_arm = daily_eligible_users * traffic_per_arm_fraction

days_needed = required_n_per_arm / daily_users_per_arm
print(f"Daily users entering the experiment: {daily_eligible_users:,}")
print(f"Users per arm needed:                {required_n_per_arm:,}")
print(f"Estimated test duration:             {days_needed:.1f} days")
print("Round up to a full number of weeks to avoid day-of-week seasonality bias "
      "(weekday vs weekend behavior often differs).")

6 Running the Test: Traffic Splitting Strategies

Once you know the sample size, you must decide how to roll out traffic to the new model.

• Fixed 50/50 split: half of eligible users get control, half get treatment, for the entire duration. Maximizes statistical power per day (the fastest way to reach your required sample size), at the cost of exposing the maximum number of users to a new model that might turn out to be worse.
• Ramped rollout: start treatment at a small fraction (e.g., 1-5%), monitor guardrail metrics closely for catastrophic regressions, then progressively increase the treatment share (5% → 25% → 50%) as confidence builds. Slower to reach full statistical power, but dramatically reduces blast radius if the new model has a serious bug or a much worse failure mode that wasn't visible offline.
• Hybrid: ramp quickly through an initial "canary" phase purely to catch operational disasters (crashes, error-rate spikes, latency regressions), then settle into a fixed 50/50 (or whatever split your power calculation requires) for the statistical comparison itself.

import pandas as pd
import numpy as np

def ramp_schedule(total_days: int, ramp_stages: list[tuple[int, float]]) -> pd.DataFrame:
    """
    Build a ramped rollout schedule.
    ramp_stages: list of (day_to_start, treatment_fraction) tuples.
    """
    days = np.arange(1, total_days + 1)
    fractions = np.zeros(total_days)

    for start_day, frac in sorted(ramp_stages):
        fractions[start_day - 1:] = frac

    return pd.DataFrame({"day": days, "treatment_fraction": fractions})


schedule = ramp_schedule(
    total_days=21,
    ramp_stages=[(1, 0.01), (3, 0.05), (7, 0.25), (10, 0.50)],
)
print(schedule[schedule['day'].isin([1, 3, 7, 10, 14, 21])].to_string(index=False))

Most production ML teams default to a canary-then-fixed-split hybrid for any model that touches revenue, safety, or compliance-sensitive decisions (fraud, credit, medical triage) — the cost of a silent catastrophic failure at 50% traffic is simply too high to skip the canary stage, even though it delays reaching full statistical power.

7 Analyzing Results: t-tests, z-tests, and the Danger of Peeking

With data collected, the analysis itself is usually a few lines of scipy.stats — the discipline is in doing it once, correctly, at the pre-registered sample size.

Two-Proportion Z-Test (for rates: CTR, conversion, add-to-cart)

import numpy as np
from scipy import stats

def two_proportion_z_test(conversions_a, n_a, conversions_b, n_b, alpha=0.05):
    """
    Two-sided two-proportion z-test.
    a = control, b = treatment.
    Returns the z-statistic, p-value, and a 95% CI for the difference (b - a).
    """
    p_a = conversions_a / n_a
    p_b = conversions_b / n_b
    p_pool = (conversions_a + conversions_b) / (n_a + n_b)

    se_pooled = np.sqrt(p_pool * (1 - p_pool) * (1 / n_a + 1 / n_b))
    z_stat = (p_b - p_a) / se_pooled
    p_value = 2 * (1 - stats.norm.cdf(abs(z_stat)))   # two-sided

    # CI for the difference uses the UNPOOLED standard error
    se_unpooled = np.sqrt(p_a * (1 - p_a) / n_a + p_b * (1 - p_b) / n_b)
    z_crit = stats.norm.ppf(1 - alpha / 2)
    diff = p_b - p_a
    ci_low, ci_high = diff - z_crit * se_unpooled, diff + z_crit * se_unpooled

    return {
        "rate_control":    p_a,
        "rate_treatment":  p_b,
        "absolute_lift":   diff,
        "relative_lift":   diff / p_a,
        "z_stat":          z_stat,
        "p_value":         p_value,
        "ci_95":           (ci_low, ci_high),
        "significant":     p_value < alpha,
    }


# ── Simulate a realistic A/B test on synthetic data ──
np.random.seed(42)
n_per_arm = 14_732          # matches the power calculation from Section 5
true_rate_control   = 0.080
true_rate_treatment = 0.088  # the model really does have a small true lift

control_conversions   = np.random.binomial(n_per_arm, true_rate_control)
treatment_conversions = np.random.binomial(n_per_arm, true_rate_treatment)

result = two_proportion_z_test(control_conversions, n_per_arm,
                                treatment_conversions, n_per_arm)

print(f"Control:    {control_conversions:,} / {n_per_arm:,}  = {result['rate_control']:.4f}")
print(f"Treatment:  {treatment_conversions:,} / {n_per_arm:,}  = {result['rate_treatment']:.4f}")
print(f"Absolute lift: {result['absolute_lift']:+.4f}  ({result['relative_lift']:+.1%} relative)")
print(f"z-statistic:   {result['z_stat']:.3f}")
print(f"p-value:       {result['p_value']:.4f}")
print(f"95% CI for lift: [{result['ci_95'][0]:+.4f}, {result['ci_95'][1]:+.4f}]")
print(f"Statistically significant at alpha=0.05? {result['significant']}")

Two-Sample t-Test (for continuous metrics: revenue per user, latency, session time)

import numpy as np
from scipy import stats

np.random.seed(7)

# Revenue per user is rarely normal (many zeros, a long right tail of big spenders) —
# but with thousands of users per arm, the Central Limit Theorem makes the t-test's
# sampling distribution of the MEAN approximately normal regardless.
control_revenue   = np.random.exponential(scale=12.50, size=10_000)     # mean ~$12.50
treatment_revenue = np.random.exponential(scale=13.10, size=10_000)     # mean ~$13.10

t_stat, p_value = stats.ttest_ind(treatment_revenue, control_revenue, equal_var=False)

mean_diff = treatment_revenue.mean() - control_revenue.mean()
se_diff = np.sqrt(treatment_revenue.var(ddof=1) / len(treatment_revenue) +
                   control_revenue.var(ddof=1) / len(control_revenue))
ci_low  = mean_diff - 1.96 * se_diff
ci_high = mean_diff + 1.96 * se_diff

print(f"Control mean revenue/user:   ${control_revenue.mean():.2f}")
print(f"Treatment mean revenue/user: ${treatment_revenue.mean():.2f}")
print(f"Difference:                  ${mean_diff:+.2f}")
print(f"95% CI for difference:       [${ci_low:+.2f}, ${ci_high:+.2f}]")
print(f"t-statistic: {t_stat:.3f}   p-value: {p_value:.4f}")
print(f"Significant at alpha=0.05? {p_value < 0.05}")

The Peeking Problem: Why You Cannot Just "Check Every Day Until It's Significant"

The single most common way teams accidentally invalidate a perfectly well-designed A/B test is peeking: computing the p-value every day and stopping the moment it first crosses below 0.05. This is also called the optional stopping problem, and it is a form of the broader multiple testing problem. Each daily check is a fresh opportunity for random noise to produce a "significant" result purely by chance — and if you check often enough, you are nearly guaranteed to see a spurious p < 0.05 at some point, even when the null hypothesis (no real difference) is exactly true.

import numpy as np
from scipy import stats

def simulate_peeking_inflation(n_simulations=2000, daily_n_per_arm=200,
                                n_days=20, true_effect=0.0, alpha=0.05, seed=0):
    """
    Simulate experiments where control and treatment have IDENTICAL true rates
    (true_effect=0, i.e., H0 is true), then check 'how often would a daily
    peeker have stopped early and falsely declared significance?'
    """
    rng = np.random.default_rng(seed)
    base_rate = 0.10
    false_positive_if_peek_daily = 0
    false_positive_if_wait_to_end = 0

    for _ in range(n_simulations):
        control_cum, treatment_cum = 0, 0
        control_n, treatment_n = 0, 0
        stopped_early = False

        for day in range(1, n_days + 1):
            control_cum   += rng.binomial(daily_n_per_arm, base_rate)
            treatment_cum += rng.binomial(daily_n_per_arm, base_rate + true_effect)
            control_n     += daily_n_per_arm
            treatment_n   += daily_n_per_arm

            p_a, p_b = control_cum / control_n, treatment_cum / treatment_n
            p_pool = (control_cum + treatment_cum) / (control_n + treatment_n)
            se = np.sqrt(p_pool * (1 - p_pool) * (1 / control_n + 1 / treatment_n))
            z = (p_b - p_a) / se if se > 0 else 0
            p_value = 2 * (1 - stats.norm.cdf(abs(z)))

            if p_value < alpha and not stopped_early:
                stopped_early = True   # a daily "peeker" would stop right here

        if stopped_early:
            false_positive_if_peek_daily += 1
        # The "wait to the end" analyst only looks at the final day's p-value
        if p_value < alpha:
            false_positive_if_wait_to_end += 1

    return (false_positive_if_peek_daily / n_simulations,
            false_positive_if_wait_to_end / n_simulations)


peek_fpr, fixed_fpr = simulate_peeking_inflation()
print(f"Nominal alpha:                              0.05")
print(f"False positive rate if you peek every day:  {peek_fpr:.3f}")
print(f"False positive rate if you wait to the end: {fixed_fpr:.3f}")

Even though control and treatment had identical true conversion rates in this simulation, daily peeking inflated the false-positive rate from the nominal 5% to over 30%. This is one of the most consequential and most frequently violated rules in experimentation.

Sequential Testing as the Principled Alternative

Sequential testing methods (e.g., Wald's Sequential Probability Ratio Test, or modern variants like always-valid p-values / mSPRT used by experimentation platforms at large tech companies) are explicitly designed to let you check results continuously without inflating the false-positive rate. They work by using error-spending boundaries that get stricter the more often you look, so the *cumulative* probability of a false positive across all your peeks stays bounded at α. If your organization needs "peek anytime, stop as soon as you're confident" as a workflow, use a library or platform that implements proper sequential testing rather than re-running a fixed-horizon z-test every day.

8 Common Pitfalls: Simpson's Paradox, Novelty Effects, and Interference

Even a well-powered, properly-randomized, non-peeked test can mislead you if you don't watch for these failure modes.

Simpson's Paradox

An aggregate result can reverse direction once you split by a subgroup. A classic A/B testing version: the new ranking model wins overall, but loses within both the mobile segment and the desktop segment individually — because the treatment arm happened to get a higher proportion of mobile traffic (which converts better generally), not because the model is actually better on either platform.

import pandas as pd

# Simpson's paradox demonstration: aggregate favors treatment,
# but EVERY individual segment favors control.
data = pd.DataFrame({
    "segment":   ["mobile", "mobile", "desktop", "desktop"],
    "variant":   ["control", "treatment", "control", "treatment"],
    "conversions": [180, 1400, 1400, 180],
    "users":       [1000, 10000, 10000, 1000],
})
data["rate"] = data["conversions"] / data["users"]
print(data.to_string(index=False))

agg = data.groupby("variant")[["conversions", "users"]].sum()
agg["rate"] = agg["conversions"] / agg["users"]
print("\nAggregate (ignoring segment):")
print(agg)
print("\nWithin mobile:   control =", data.query("segment=='mobile' and variant=='control'")["rate"].iloc[0],
      " treatment =", data.query("segment=='mobile' and variant=='treatment'")["rate"].iloc[0])
print("Within desktop:  control =", data.query("segment=='desktop' and variant=='control'")["rate"].iloc[0],
      " treatment =", data.query("segment=='desktop' and variant=='treatment'")["rate"].iloc[0])

The fix: always verify that the randomization actually balanced segment composition between arms (a "sample ratio mismatch" / segment-balance check), and report results sliced by major known segments, not only in aggregate.

Novelty and Primacy Effects

Users sometimes react to a change simply because it's new — clicking out of curiosity, or experiencing temporary irritation at a changed UI — independent of whether the underlying model is actually better. A novelty effect inflates short-term metrics that fade once users adapt; a primacy effect (users initially resist a change, then come to prefer it) can mask a real win in the early days. The mitigation: run the test long enough to see metrics stabilize, and where feasible compare a fresh cohort of new users (who have no "old way" to be novel relative to) against an existing-user cohort.

Network Effects and Interference (SUTVA Violations)

The standard analysis (Sections 5–7) assumes the Stable Unit Treatment Value Assumption (SUTVA): one user's assigned variant doesn't affect another user's outcome. This breaks down in social or marketplace products — a treatment user who gets better recommendations and shares content more might influence control users in their network; a ride-sharing pricing experiment can shift driver supply availability for control-arm riders in the same city. When interference is plausible, randomize at a coarser unit that contains the interaction (e.g., by city or by social cluster) instead of by individual user, even though this reduces your effective sample size.

Underpowered Tests: The Quiet Failure

An underpowered test produces a false sense of having "checked" something. Section 4's Type II error rate (β) is exactly the rate at which a real improvement is missed — and teams that don't compute power up front routinely run tests with 20-30% power, where a coin flip would have nearly as much chance of detecting a true win.

9 Beyond Fixed Splits: Multi-Armed Bandits

A fixed 50/50 A/B test deliberately "wastes" conversions: every user routed to the losing arm for the full duration of the test is a user who didn't get the better experience, purely so you could measure the difference precisely. A multi-armed bandit reframes the problem as an explore/exploit trade-off: keep learning which arm is better while increasingly favoring whichever arm currently looks best, to minimize cumulative regret (the gap between what you earned and what you would have earned by always playing the best arm).

Thompson Sampling

Thompson Sampling is a simple, strong-performing bandit algorithm: maintain a probability distribution over each arm's true conversion rate (a Beta distribution is the natural conjugate prior for a binary conversion outcome), and on each new user, draw a random sample from each arm's current belief distribution and route the user to whichever arm drew the highest sample. Arms with more promising and more certain evidence get more traffic automatically — no manual ramp schedule required.

import numpy as np

class ThompsonSamplingBandit:
    """
    Beta-Bernoulli Thompson Sampling for a binary outcome (e.g., conversion).
    Each arm's belief about its true conversion rate is a Beta(alpha, beta)
    distribution, updated after every observation.
    """
    def __init__(self, arm_names: list[str]):
        self.arm_names = arm_names
        # Start with an uninformative Beta(1, 1) = Uniform(0, 1) prior per arm
        self.alpha = {name: 1.0 for name in arm_names}
        self.beta  = {name: 1.0 for name in arm_names}

    def select_arm(self) -> str:
        samples = {
            name: np.random.beta(self.alpha[name], self.beta[name])
            for name in self.arm_names
        }
        return max(samples, key=samples.get)

    def update(self, arm: str, reward: int):
        """reward = 1 for a conversion, 0 otherwise."""
        self.alpha[arm] += reward
        self.beta[arm]  += (1 - reward)

    def estimated_rates(self):
        return {name: self.alpha[name] / (self.alpha[name] + self.beta[name])
                for name in self.arm_names}


# Simulate: "current_model" truly converts at 8.0%, "new_model" truly at 9.5%
np.random.seed(3)
true_rates = {"current_model": 0.080, "new_model": 0.095}
bandit = ThompsonSamplingBandit(list(true_rates.keys()))

n_rounds = 20_000
traffic_log = {"current_model": 0, "new_model": 0}

for _ in range(n_rounds):
    chosen_arm = bandit.select_arm()
    reward = np.random.binomial(1, true_rates[chosen_arm])
    bandit.update(chosen_arm, reward)
    traffic_log[chosen_arm] += 1

print("Traffic allocation after 20,000 rounds:")
for arm, count in traffic_log.items():
    print(f"  {arm:15s}: {count:,} users ({count/n_rounds:.1%})  | true rate = {true_rates[arm]:.1%}")

print("\nBandit's estimated conversion rates:")
for arm, rate in bandit.estimated_rates().items():
    print(f"  {arm:15s}: {rate:.4f}")

Notice the bandit automatically shifted the majority of traffic to the better-performing arm well before the experiment "ended" — there is no fixed end date in the bandit framing, traffic allocation just keeps adapting.

A/B Testing vs Bandits: When to Use Which

10 The Full ML Lifecycle: From Lesson 1 to Production

This is the 67th and final lesson of the curriculum, which makes it a good moment to step back and see the whole shape of what you've built, rather than the individual pieces.

It started with foundations: NumPy arrays and Pandas DataFrames gave you a way to hold and manipulate data; probability, distributions, and linear algebra (Lessons 6-8) gave you the mathematical vocabulary everything downstream is written in. Lesson 9's ML project lifecycle then gave you the map before the territory — the 8-stage loop of SOW → data → EDA → features → model → tuning → deployment → monitoring that, in hindsight, this entire curriculum has been a deep dive into, one stage at a time.

From there you built the classical ML toolkit — linear and logistic regression, trees and ensembles, SVMs, clustering — learning not just how each algorithm works but when each one is the right tool. Then deep learning: backpropagation, CNNs, RNNs, and the architectures that made vision and sequence modeling tractable at scale. Then the modern era — transformers and LLMs, attention mechanisms, fine-tuning, retrieval-augmented generation, and multimodal models (Lesson 62) that read, see, and listen.

Phase 6 closed the loop on what happens once a model exists. Optuna (Lesson 63) taught you to tune a model's hyperparameters systematically instead of by hand. SHAP taught you to explain what a model learned, so its decisions are auditable rather than opaque. MLflow taught you to track every experiment so "which run produced this model, with which data and which parameters" is never a mystery. FastAPI, Docker, and SageMaker (Lesson 66) taught you to turn a trained model into a service the rest of the world can actually call. PSI and CSI (Lesson 68) taught you to watch that service in production and notice, quantitatively, the moment the world it was trained on stops matching the world it's serving. And this lesson taught you the final piece: how to prove, with the same statistical rigor you'd demand of any scientific claim, that a candidate replacement is actually an improvement before you trust it with all of your traffic.

Put together, the lifecycle is a loop, not a line:

build → tune (Optuna) → explain (SHAP) → track (MLflow) → deploy (FastAPI / Docker / SageMaker) → monitor (PSI / CSI) → validate (A/B testing) → repeat

Monitoring (Lesson 68) is what tells you when to act — drift, degradation, a changed world. A/B testing (this lesson) is what tells you whether the thing you built in response actually helped, with a number you can defend in a room full of skeptical stakeholders, not just a hunch from a higher offline AUC. Every production ML system that survives more than a few months runs this loop continuously: something drifts, a new candidate is built and tuned and explained and tracked, it's deployed behind a flag, it's A/B tested against the incumbent, and — win or lose — the result becomes input to the next iteration of the loop. That loop, not any single model, is the actual deliverable of a mature ML practice. You now have every piece of it.