1 Descriptive vs Inferential Statistics

Descriptive statistics summarize the data you actually have — your sample. They answer: "What does my dataset look like?" Mean, median, standard deviation, histograms, and correlation coefficients are all descriptive tools.

Inferential statistics use a sample to draw conclusions about a larger population you haven't measured. They answer: "Based on what I observed, what can I say about the world?" Hypothesis tests, confidence intervals, and p-values are inferential tools.

This lesson focuses on descriptive statistics. Inferential statistics appear later in the curriculum when we cover model evaluation and A/B testing. Both are essential; descriptive statistics come first because you must understand your data before you can reason about populations.

2 Measures of Central Tendency

Central tendency statistics describe the "center" or "typical value" of a distribution. They're the most basic summary of a feature, but choosing the wrong one can be very misleading.

Mean (Arithmetic Average)

The mean is the sum of all values divided by the count. It uses every data point in the calculation, which makes it both powerful and sensitive to outliers. One extreme value can pull the mean far from where most data lives.

Median

The median is the middle value when data is sorted. If there's an even number of values, it's the average of the two middle values. The median is robust to outliers — adding a billion-dollar value to an income dataset barely moves the median, but dramatically shifts the mean.

Mode

The mode is the most frequently occurring value. It's the only central tendency measure that works for categorical data. Numeric data can be bimodal (two modes) or multimodal.

import pandas as pd
import numpy as np

# Salary dataset with one extreme outlier (the CEO)
salaries = pd.Series([45000, 52000, 48000, 55000, 61000, 49000,
                      58000, 47000, 53000, 2_500_000])  # CEO salary

print(f"Mean:   ${salaries.mean():>12,.0f}")   # $337,700 — distorted by CEO
print(f"Median: ${salaries.median():>12,.0f}") # $52,500  — representative
print(f"Mode:   {salaries.mode().values}")     # no repeated values here

# Mode is more meaningful for categorical data
cities = pd.Series(['NYC', 'LA', 'NYC', 'Chicago', 'NYC', 'LA', 'NYC'])
print(f"Most common city: {cities.mode()[0]}")  # NYC

# Practical: choose mean vs median based on skewness
print(f"\nSkewness of salaries: {salaries.skew():.2f}")
# High positive skew → use median for "typical salary"

The same effect is visible at any scale. Below is a small 9-point salary dataset (in $k) — drag the slider to add one extra "outlier" salary and watch how far the mean drifts away from the bulk of the data while the median barely moves:

3 Percentiles & Quartiles

A percentile tells you what value a given percentage of data falls below. The 90th percentile income means 90% of people earn less than that amount. Percentiles are used everywhere in ML: defining thresholds, creating features, detecting outliers, and evaluating model performance.

Quartiles are the three percentile points that divide sorted data into four equal parts:

• Q1 (25th percentile): 25% of data falls below this value
• Q2 (50th percentile): the median — 50% below, 50% above
• Q3 (75th percentile): 75% of data falls below this value

Together with the minimum and maximum, they form the five-number summary — what df.describe() shows you.

import numpy as np
import pandas as pd

np.random.seed(42)
data = np.random.normal(loc=50, scale=15, size=1000).clip(0, 100)

# ── Using numpy ──
q1  = np.percentile(data, 25)
q2  = np.percentile(data, 50)   # same as median
q3  = np.percentile(data, 75)
p90 = np.percentile(data, 90)

print(f"Q1  (25th): {q1:.2f}")
print(f"Q2  (50th): {q2:.2f}")
print(f"Q3  (75th): {q3:.2f}")
print(f"P90 (90th): {p90:.2f}")

# ── Using pandas ──
s = pd.Series(data)
print(s.quantile([0.25, 0.50, 0.75, 0.90]))
# 0.25    39.58
# 0.50    50.07
# 0.75    60.18
# 0.90    68.67

# Five-number summary (describe includes more, but these are the core 5)
five_num = {
    'min':    data.min(),
    'Q1':     q1,
    'median': q2,
    'Q3':     q3,
    'max':    data.max()
}
for k, v in five_num.items():
    print(f"{k:>8}: {v:.2f}")

4 Measures of Dispersion: Range & IQR

Measures of dispersion describe how spread out the data is. Two datasets can have the same mean but completely different spreads — which matters enormously for model behavior.

Range

Range = maximum − minimum. Simple to compute, but completely dominated by the two extreme values. One bad data point can make the range useless.

Interquartile Range (IQR)

IQR = Q3 − Q1. This is the spread of the middle 50% of the data. Because it ignores the top 25% and bottom 25%, it's completely robust to outliers. IQR is the preferred spread measure for skewed distributions and outlier detection.

import numpy as np
import pandas as pd
from scipy import stats

np.random.seed(0)
# Normal income data + one extreme outlier
incomes = np.append(np.random.normal(55000, 12000, 99), 2_500_000)

q1 = np.percentile(incomes, 25)
q3 = np.percentile(incomes, 75)
iqr = q3 - q1

print(f"Min:    ${incomes.min():>12,.0f}")
print(f"Max:    ${incomes.max():>12,.0f}")
print(f"Range:  ${incomes.max() - incomes.min():>12,.0f}")  # Completely distorted
print(f"Q1:     ${q1:>12,.0f}")
print(f"Q3:     ${q3:>12,.0f}")
print(f"IQR:    ${iqr:>12,.0f}")                             # Unaffected by outlier
print(f"Std:    ${incomes.std():>12,.0f}")                   # Also distorted

# IQR via pandas
s = pd.Series(incomes)
iqr_pandas = s.quantile(0.75) - s.quantile(0.25)
print(f"\nIQR (pandas): ${iqr_pandas:,.0f}")

# scipy has a convenience function
from scipy.stats import iqr as scipy_iqr
print(f"IQR (scipy):  ${scipy_iqr(incomes):,.0f}")

A box plot is just a picture of the five-number summary. Here's the box plot for the incomes array above (99 normal incomes plus the one $2.5M outlier) — the box spans Q1 to Q3, the line inside is the median, the whiskers extend to the most extreme non-outlier points, and anything beyond the 1.5×IQR fences is plotted as an individual dot:

5 Outlier Detection Using the IQR Rule

The Tukey fence method (also called the 1.5×IQR rule) is the standard approach for identifying outliers in non-normally-distributed data. It defines two fences:

• Lower fence = Q1 − 1.5 × IQR
• Upper fence = Q3 + 1.5 × IQR

Any value below the lower fence or above the upper fence is classified as an outlier. This is exactly the rule that box plots use to determine where to draw whiskers and where to plot dots.

import numpy as np
import pandas as pd

np.random.seed(42)
# Transaction amounts: mostly normal, with some fraudulent high-value transactions
amounts = np.append(np.random.normal(150, 40, 200),
                    [850, 920, 1100, -50, -200])  # outliers

# ── Step 1: Compute IQR fences ──
q1  = np.percentile(amounts, 25)
q3  = np.percentile(amounts, 75)
iqr = q3 - q1

lower_fence = q1 - 1.5 * iqr
upper_fence = q3 + 1.5 * iqr

print(f"Q1:           {q1:.2f}")
print(f"Q3:           {q3:.2f}")
print(f"IQR:          {iqr:.2f}")
print(f"Lower fence:  {lower_fence:.2f}")
print(f"Upper fence:  {upper_fence:.2f}")

# ── Step 2: Boolean mask to flag outliers ──
is_outlier = (amounts < lower_fence) | (amounts > upper_fence)
print(f"\nTotal outliers detected: {is_outlier.sum()}")
print(f"Outlier values: {amounts[is_outlier]}")

# ── Step 3: Using a Pandas DataFrame ──
df = pd.DataFrame({'amount': amounts})
df['is_outlier'] = (df['amount'] < lower_fence) | (df['amount'] > upper_fence)
print(df[df['is_outlier']])

# ── Step 4: Filter to keep only non-outliers ──
df_clean = df[~df['is_outlier']].drop(columns='is_outlier')
print(f"\nOriginal size: {len(df)}, After removing outliers: {len(df_clean)}")

6 Outlier Treatment Methods

Detecting outliers is only half the job. Deciding what to do with them requires domain knowledge and careful judgment. There is no single correct answer — the right treatment depends on why the outlier exists.

Method 1: Removal

Simply drop rows with outlier values. Safe when: the outlier is clearly a data error (e.g., a negative age, a weight of 999 kg), the fraction of outliers is small (<1–2%), and the outliers won't appear in production data.

df_no_outliers = df[~is_outlier]
print(f"Removed {is_outlier.sum()} rows")

Method 2: Capping / Flooring (Winsorization)

Cap values at the fence boundaries instead of removing them. This preserves the sample size while limiting the influence of extremes. Useful when the outlier might be real (genuine large transaction) but you don't want it to dominate the model.

df['amount_capped'] = df['amount'].clip(lower=lower_fence, upper=upper_fence)
# Values below lower_fence become lower_fence; above upper_fence become upper_fence
print(df['amount_capped'].describe())

Method 3: Log Transform (for Right-Skewed Data)

A log transform compresses the right tail of a distribution, making it approximately normal. It's the standard treatment for right-skewed features like income, house prices, and transaction amounts. The outliers still exist in the original scale, but they're no longer extreme after transformation.

import numpy as np

# Only works for positive values — add 1 if there are zeros
df['log_amount'] = np.log1p(df['amount'].clip(lower=0))  # log(1 + amount)

print(f"Original skew: {df['amount'].skew():.2f}")
print(f"Log skew:      {df['log_amount'].skew():.2f}")
# log skew is much closer to 0 — a more symmetric, bell-like shape

Method 4: Impute with Median

For extreme outliers caused by missing data encoded as sentinel values (like -999 or 9999), replace them with the median. This is a special case of the missing value imputation strategies from Lesson 04.

SENTINEL = 9999
df.loc[df['amount'] == SENTINEL, 'amount'] = df['amount'].median()

7 Variance & Standard Deviation

Variance measures the average squared deviation of each data point from the mean. Squaring the deviations ensures that positive and negative deviations don't cancel out, and it penalizes larger deviations more heavily.

The formula for sample variance (used when computing from a sample to estimate a population) is:

s² = Σ(xᵢ − x̄)² / (n − 1)

The denominator is (n−1) rather than n — this is called Bessel's correction and it adjusts for the fact that the sample mean is used instead of the true population mean. Use ddof=1 in NumPy for sample variance (the default in Pandas).

Standard deviation is simply the square root of variance: s = √s². It's in the same units as the original data, making it interpretable. "One standard deviation above the mean" means something concrete in real units.

import numpy as np
import pandas as pd

data = np.array([10, 12, 23, 23, 16, 23, 21, 16])

# ── Variance ──
# Population variance (ddof=0): all data IS the population
pop_var    = np.var(data, ddof=0)
# Sample variance (ddof=1): data is a sample from a larger population
sample_var = np.var(data, ddof=1)

print(f"Population variance (ddof=0): {pop_var:.4f}")
print(f"Sample variance    (ddof=1): {sample_var:.4f}")

# ── Standard deviation ──
pop_std    = np.std(data, ddof=0)
sample_std = np.std(data, ddof=1)

print(f"Population std (ddof=0): {pop_std:.4f}")
print(f"Sample std     (ddof=1): {sample_std:.4f}")

# ── Pandas defaults ──
s = pd.Series(data)
print(f"\npd.Series.var(): {s.var():.4f}")   # ddof=1 by default
print(f"pd.Series.std(): {s.std():.4f}")   # ddof=1 by default

# ── Practical interpretation ──
heights = pd.Series([165, 170, 172, 168, 175, 180, 162, 171, 169, 173])
mean_h  = heights.mean()
std_h   = heights.std()

print(f"\nMean height: {mean_h:.1f} cm")
print(f"Std height:  {std_h:.1f} cm")
print(f"68% of heights fall between {mean_h - std_h:.1f} and {mean_h + std_h:.1f} cm")
# ~68% of data falls within 1 std of the mean when values follow
# the classic bell curve — Lesson 07 makes this rule precise

8 Correlation: Pearson & Spearman

Correlation measures the strength and direction of the relationship between two variables. It ranges from −1 to +1 regardless of the original units, making it a standardized measure.

Pearson Correlation (Linear)

Pearson correlation measures the strength of the linear relationship between two continuous variables. It assumes both variables are roughly normally distributed and that the relationship is linear. This is the default in df.corr().

Spearman Correlation (Rank-Based / Monotonic)

Spearman correlation measures the strength of any monotonic relationship — one that consistently goes up or consistently goes down, but not necessarily at a constant rate. It works by converting values to ranks first, making it robust to outliers and valid for non-normal distributions. Use df.corr(method='spearman').

import numpy as np
import pandas as pd
from scipy import stats

np.random.seed(0)
x = np.random.uniform(1, 100, 200)
y_linear  = 2 * x + np.random.normal(0, 10, 200)   # linear relationship
y_rank    = x ** 2 + np.random.normal(0, 500, 200)  # monotonic but nonlinear
y_none    = np.random.normal(50, 20, 200)            # no relationship

df = pd.DataFrame({'x': x, 'y_linear': y_linear, 'y_rank': y_rank, 'y_none': y_none})

# Pearson correlations
print("=== Pearson Correlations ===")
print(df.corr(method='pearson')['x'].round(3))

# Spearman correlations
print("\n=== Spearman Correlations ===")
print(df.corr(method='spearman')['x'].round(3))

# x vs y_linear: Pearson ≈ 0.98, Spearman ≈ 0.98  (linear → both work)
# x vs y_rank:   Pearson ≈ 0.77, Spearman ≈ 0.99  (nonlinear → Spearman finds it)
# x vs y_none:   Pearson ≈ 0.05, Spearman ≈ 0.05  (no correlation → both ~0)

# Using scipy for a single pair with p-value
r, p = stats.pearsonr(x, y_linear)
print(f"\nPearson r={r:.3f}, p-value={p:.2e}")
# p < 0.05 → statistically significant correlation

rho, p_sp = stats.spearmanr(x, y_rank)
print(f"Spearman ρ={rho:.3f}, p-value={p_sp:.2e}")

Correlation coefficients are easiest to build intuition for by looking at the scatter plots they summarize. Switch between three datasets shaped like x, y_linear, and y_none from the code above, plus a strong negative example, and watch both the point cloud and the Pearson/Spearman readouts change: