🎯 What You'll Learn
- Perform element-wise and aggregate operations on arrays
- Understand and apply broadcasting — NumPy's rules for arithmetic on different-shaped arrays
- Compute dot products and matrix multiplication — the most important operation in ML
- Use statistical functions:
mean,std,sum,max,minalong axes - Write vectorized code instead of Python loops — and understand why it matters
- Normalize features for machine learning: zero-mean, unit-variance scaling
1 Element-Wise Operations
NumPy operators (+, -, *, /, **) apply element-by-element when arrays have the same shape. No loops needed.
import numpy as np
a = np.array([1, 2, 3, 4])
b = np.array([10, 20, 30, 40])
print(a + b) # [11, 22, 33, 44] — element-wise addition
print(a * b) # [10, 40, 90, 160] — element-wise multiplication
print(b / a) # [10. 10. 10. 10.] — element-wise division
print(a ** 2) # [ 1 4 9 16] — element-wise squaring
print(np.sqrt(a)) # [1. 1.414 1.732 2. ] — element-wise sqrtThe same logic applies to 2D arrays. Every operation targets corresponding positions:
M = np.array([[1, 2], [3, 4]])
N = np.array([[5, 6], [7, 8]])
print(M + N)
# [[ 6 8]
# [10 12]]
print(M * N) # element-wise, NOT matrix multiplication
# [[ 5 12]
# [21 32]]M * N is NOT matrix multiplication
In NumPy, * always means element-wise multiplication. For the linear-algebra matrix product you need in ML (summing the dot products of rows and columns), you must use np.dot(M, N) or the @ operator: M @ N. This is one of the most common mistakes for beginners.
2 Broadcasting — Math Across Different Shapes
Broadcasting is how NumPy handles arithmetic between arrays of different shapes — without copying any data. It's what makes code like arr - arr.mean() work on a 2D array, and it's fundamental to how ML computations are written efficiently.
The simplest case: array + scalar
arr = np.array([[1, 2, 3],
[4, 5, 6]])
# Adding a scalar to a 2D array
result = arr + 10
print(result)
# [[11 12 13]
# [14 15 16]]
# The scalar 10 is "broadcast" to every element — no copyingBroadcasting a 1D array across rows of a 2D array
X = np.array([[1, 2, 3], # shape (3, 3)
[4, 5, 6],
[7, 8, 9]])
offset = np.array([10, 20, 30]) # shape (3,)
# NumPy broadcasts offset to every row of X
result = X + offset
print(result)
# [[11 22 33]
# [14 25 36]
# [17 28 39]]Broadcasting in action: offset has shape (3,) — just one row. NumPy doesn't physically copy it; it conceptually repeats the row to match every row of X (the faded, dashed cells), then adds element-wise. No extra memory is ever allocated for the repeated copies.
NumPy compares shapes element by element, starting from the trailing (rightmost) dimension. Two dimensions are compatible if they are equal, or if one of them is 1. If the shorter array has fewer dimensions, NumPy pads 1s on the left.
Example: (3, 3) + (3,) → pad to (3, 3) + (1, 3) → broadcast row to match → result (3, 3). ✓
When broadcasting FAILS
a = np.ones((3, 4)) # shape (3, 4)
b = np.ones((3,)) # shape (3,)
# Attempt to add: trailing dimensions are 4 and 3 — neither is 1
try:
result = a + b
except ValueError as e:
print(e)
# operands could not be broadcast together with shapes (3,4) (3,)To broadcast along columns instead, you need to add an axis to b so its shape becomes (3, 1):
b_col = b.reshape(-1, 1) # shape: (3, 1)
result = a + b_col # shapes (3, 4) + (3, 1) → broadcast to (3, 4) ✓
print(result.shape) # (3, 4)Same a (shape (3, 4), all ones), same b (shape (3,)) — but reshaping b changes which axis it broadcasts along. Toggle between the two to see how the result heatmap changes:
a + b: the row vector [1,2,3,4] is repeated for all 3 rows of a.
Practical Use: Adjusting Every Row of a Dataset at Once
Suppose 100 weather stations each report 5 sensor readings, and you know each sensor type has a fixed calibration error. Broadcasting lets you correct the entire dataset in one line — no loop over rows:
# Readings from 100 weather stations, 5 sensors each
readings = np.random.randn(100, 5) # shape (100, 5)
# Known calibration offset for each of the 5 sensor types
offsets = np.array([0.5, -0.2, 0.0, 1.1, -0.7]) # shape (5,)
# Broadcasting: apply each sensor's offset to every station at once
corrected = readings + offsets # shape (100, 5)
print(corrected.shape) # (100, 5)This "one row of per-column values applied to the whole table" pattern is everywhere in ML code — you'll use it in this very lesson to normalize features, and constantly after that.
3 Dot Products & Matrix Multiplication
This is the most important mathematical operation in machine learning. Nearly every model you'll build in this course — from the simplest one in Phase 2 to the deep networks in Phase 4 — spends most of its time doing exactly this operation. Learn it well here, with plain numbers, and everything later gets easier.
Dot Product of Two Vectors
The dot product multiplies corresponding elements and sums the results. For two vectors of length n, it produces a single scalar.
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
dot = np.dot(a, b) # 1×4 + 2×5 + 3×6 = 4 + 10 + 18 = 32
print(dot) # 32
# Equivalently using the @ operator (Python 3.5+)
dot2 = a @ b
print(dot2) # 32The dot product has a very natural meaning: it's a weighted sum. Here's an example you already know from school — computing a final course grade where homework counts 20%, the midterm 30%, and the final exam 50%:
scores = np.array([72, 85, 91]) # homework, midterm, final exam
weights = np.array([0.2, 0.3, 0.5]) # how much each one counts
final_grade = scores @ weights # 0.2×72 + 0.3×85 + 0.5×91
print(final_grade) # 85.4Keep this picture in mind: a dot product combines several values into one number, using a set of weights that says how important each value is. When you reach Phase 2, you'll discover that this is exactly how ML models make predictions — they just learn the weights from data instead of being handed them.
Matrix-Vector Multiplication
Now suppose the school is considering 4 different grading schemes — each weighting homework, midterm, and final differently. Stack the schemes as rows of a matrix, and one matrix-vector multiplication grades the student under all 4 schemes at once. Each row of W does its own dot product with x:
# W: 4 grading schemes × 3 scores, one scheme per row
W = np.array([[0.2, 0.3, 0.5], # standard
[0.4, 0.3, 0.3], # homework-heavy
[0.0, 0.5, 0.5], # exams only
[1/3, 1/3, 1/3]]) # equal weights — shape (4, 3)
x = np.array([72, 85, 91]) # one student's scores, shape (3,)
# Each row of W computes a dot product with x
grades = W @ x # shape (4,) — one grade per scheme
print(grades) # [85.4 81.6 88. 82.67]Matrix-Matrix Multiplication: The Whole Table at Once
In practice you rarely process one sample at a time — you process the whole dataset in a single operation. With 5 students (rows of X) and 4 grading schemes (rows of W), one matrix multiplication computes all 20 grades:
# X: 5 students, 3 scores each
X = np.array([[72, 85, 91],
[95, 60, 70],
[88, 88, 88],
[50, 75, 92],
[67, 79, 83]]) # shape (5, 3)
# Every row of X (a student) dot-products with every row of W (a scheme)
# (5, 3) @ (3, 4) → (5, 4)
grades = X @ W.T # shape (5, 4)
print(grades.shape) # (5, 4) — 5 students × 4 schemesZooming into a single output cell makes the rule concrete: each entry of the result is one dot product between a row of X and a row of W (a column of W.T). Here's student 0's scores against grading scheme 0 — the same numbers from the code above:
Matrix multiplication, one cell at a time: grades[0, 0] is the dot product of row 0 of X (student 0's scores) with row 0 of W (scheme 0's weights) — multiply each corresponding pair, then sum. Every other cell of the (5, 4) result is computed the exact same way, just with a different student/scheme pairing.
For A @ B to work: A's last dimension must equal B's first dimension. The result has A's leading dimensions and B's trailing dimensions.
(m, k) @ (k, n) → (m, n)
Example: (5, 3) @ (3, 4) → (5, 4). The inner dimension k=3 must match.
4 Aggregate Functions and the axis Parameter
NumPy provides fast aggregate functions — sum, mean, std, min, max, argmin, argmax — that can operate over the whole array or along specific dimensions.
arr = np.array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]]) # shape (3, 3)
# No axis — operates over ALL elements
print(np.sum(arr)) # 45
print(np.mean(arr)) # 5.0
print(np.max(arr)) # 9
# axis=0 — collapses rows, result is per-column
print(np.sum(arr, axis=0)) # [12, 15, 18] — sum of each column
print(np.mean(arr, axis=0)) # [4., 5., 6.] — mean of each column
# axis=1 — collapses columns, result is per-row
print(np.sum(arr, axis=1)) # [6, 15, 24] — sum of each row
print(np.mean(arr, axis=1)) # [2., 5., 8.] — mean of each rowThink: "axis=0 collapses along rows" means you're computing one statistic per column. "axis=1 collapses along columns" means one statistic per row. For a dataset with shape (n_samples, n_features): axis=0 gives stats per feature; axis=1 gives stats per sample.
argmin and argmax — where is the extreme value?
temps = np.array([21.9, 19.6, 17.3, 20.7, 15.2, 18.4]) # daily highs
coldest_day = np.argmin(temps) # index of the smallest value
print(f"Coldest day: {coldest_day}, temp: {temps[coldest_day]}")
# Coldest day: 4, temp: 15.2
# On a 2D array: which product sold the most in each store?
sales = np.array([[10, 80, 10], # store 0: product 1 wins
[70, 20, 10], # store 1: product 0 wins
[30, 30, 40]]) # store 2: product 2 wins
best_product = np.argmax(sales, axis=1) # best column per row
print(best_product) # [1, 0, 2] — column indices, not values5 Vectorization: Replace Loops with NumPy
In Python, when you want to apply an operation to every element of a list, you write a for loop. In NumPy, you don't — you write the operation once and let NumPy apply it across the whole array in compiled C. This is called vectorization, and it's the single most important performance principle in ML code.
import numpy as np
import time
# Task: compute Euclidean distance between 100,000 pairs of 128-D vectors
n = 100_000
A = np.random.randn(n, 128)
B = np.random.randn(n, 128)
# ❌ Python loop — slow
start = time.time()
distances_loop = []
for i in range(n):
diff = A[i] - B[i]
dist = np.sqrt(np.sum(diff ** 2))
distances_loop.append(dist)
print(f"Loop: {time.time() - start:.3f}s")
# ✅ Vectorized — fast
start = time.time()
distances_vec = np.sqrt(np.sum((A - B) ** 2, axis=1))
print(f"Vectorized: {time.time() - start:.3f}s")The vectorized version is over 100× faster. Training an ML model means repeating operations like this thousands upon thousands of times — so this is the difference between a model that trains in minutes and one that takes hours.
Whenever you find yourself writing a for loop over elements of a NumPy array, pause and ask: "Can I express this as a NumPy operation?" Almost always, you can. This reflex will dramatically improve both your code's performance and its readability.
6 Feature Normalization — A Practical ML Workflow
Before feeding data into most ML models, you need to normalize the features — rescale every column so they all live on a comparable range. Why? Many models (you'll meet them in Phase 2) work by comparing and combining feature values. If one feature ranges 0–1,000,000 (e.g., salary) and another ranges 0–1 (e.g., a yes/no flag), the large-scale feature drowns out the small one — not because it's more important, but simply because its numbers are bigger. Normalization removes that accidental advantage.
Min-Max Normalization (scale to [0, 1])
X = np.array([[200, 0.5, 3], # feature: salary, ratio, count
[500, 0.1, 7],
[150, 0.9, 1],
[800, 0.3, 5]]) # shape (4, 3)
X_min = X.min(axis=0) # min per feature (column), shape (3,)
X_max = X.max(axis=0) # max per feature (column), shape (3,)
X_norm = (X - X_min) / (X_max - X_min)
print(X_norm)
# [[0.196 0.5 0.333]
# [0.542 0. 1. ]
# [0. 1. 0. ]
# [1. 0.25 0.667]]Standardization / Z-score Normalization (mean=0, std=1)
The other classic recipe: subtract each column's mean, then divide by its standard deviation (a measure of how spread out the values are — Lesson 06 covers it properly). Each feature ends up centered on 0 with a spread of 1, and this is the more common choice in practice:
X_mean = X.mean(axis=0) # mean per feature
X_std = X.std(axis=0) # std per feature
X_std_norm = (X - X_mean) / X_std
print(f"Mean after standardization: {X_std_norm.mean(axis=0).round(10)}")
print(f"Std after standardization: {X_std_norm.std(axis=0).round(4)}")
# Mean after standardization: [0. 0. 0.]
# Std after standardization: [1. 1. 1.]The "salary" column above — [200, 500, 150, 800] — has a very different scale than the ratio or count columns. Here's what each normalization technique does to that single column, using the exact numbers from the code:
Raw salary values — scale 150–800, no relation to the unit-scale features it will be combined with.
Same 4 values after min-max scaling (→ [0, 1]) and after z-score standardization (mean 0, std 1).
Real ML projects always set aside part of the data to test the model on later — data the model never sees during learning. When that time comes, there's an important rule: compute X_mean and X_std from the learning portion only, and reuse those same numbers on the held-back portion. Lesson 10 explains why this matters (it prevents a subtle form of cheating called leakage). For now, just remember that the mean and std you normalize with are themselves values you must be careful about.
7 Useful NumPy Functions Cheat Sheet
arr = np.random.randn(100, 5)
# Statistics
np.mean(arr) # overall mean
np.mean(arr, axis=0) # mean per column (per feature)
np.std(arr, axis=0) # std per feature
np.var(arr) # variance
np.median(arr, axis=0) # median per feature
np.percentile(arr, 75) # 75th percentile
np.cumsum(arr, axis=0) # cumulative sum along rows
# Shape manipulation
arr.reshape(20, 25) # reshape (100, 5) → (20, 25)
arr.flatten() # → (500,)
arr.T # transpose: (100, 5) → (5, 100)
np.concatenate([arr, arr], axis=0) # stack vertically → (200, 5)
np.concatenate([arr, arr], axis=1) # stack horizontally → (100, 10)
np.stack([arr, arr], axis=0) # new axis: → (2, 100, 5)
# Set/test operations
np.unique(np.array([1, 2, 2, 3])) # [1, 2, 3]
np.where(arr > 0, arr, 0) # replace negatives with 0, keep positives
np.clip(arr, -1, 1) # clamp values to [-1, 1]
np.sort(arr, axis=0) # sort along axis
# Linear algebra
np.dot(A, B) # dot product / matmul
A @ B # matrix multiplication (same)
np.linalg.norm(arr) # L2 (Euclidean) norm
np.linalg.norm(arr, ord=1) # L1 norm
np.linalg.inv(A) # matrix inverse (square A)
np.linalg.svd(A) # singular value decomposition (advanced — Phase 3)Real-World Spotlight: Feature Normalization at a Fintech Startup
Imagine you're building a credit-scoring model for a fintech company. Your training dataset has 50,000 loan applications with features like:
- Annual income: ranges from $20,000 to $500,000
- Credit score: ranges from 300 to 850
- Debt-to-income ratio: ranges from 0.01 to 0.95
- Number of late payments: 0 to 30
- Loan amount requested: $1,000 to $50,000
Without normalization, the income feature (scale ~100,000) numerically drowns out every other column — a model learning from this data would behave as if income were the only feature that exists. The full normalization workflow in NumPy:
import numpy as np
# Simulated credit application data: 50,000 applicants, 5 features
np.random.seed(0)
n = 50_000
income = np.random.uniform(20_000, 500_000, n)
credit_score = np.random.uniform(300, 850, n)
dti_ratio = np.random.uniform(0.01, 0.95, n)
late_payments = np.random.randint(0, 31, n).astype(float)
loan_amount = np.random.uniform(1_000, 50_000, n)
# Stack into design matrix: shape (50_000, 5)
X = np.column_stack([income, credit_score, dti_ratio, late_payments, loan_amount])
print(f"X shape: {X.shape}") # (50000, 5)
# Set aside the last 20% of rows as "held-back" data the model
# will never learn from (the why is coming in Lesson 10 — here,
# notice it's just the slicing you learned in Lesson 01)
split = int(0.8 * n)
X_train, X_test = X[:split], X[split:]
# Compute normalization parameters on the learning portion only
train_mean = X_train.mean(axis=0) # shape (5,)
train_std = X_train.std(axis=0) # shape (5,)
# Apply to both parts (broadcasting handles the (5,) vs (n, 5) shape)
X_train_norm = (X_train - train_mean) / train_std
X_test_norm = (X_test - train_mean) / train_std # same stats — no peeking!
print(f"Train mean after norm: {X_train_norm.mean(axis=0).round(4)}")
print(f"Train std after norm: {X_train_norm.std(axis=0).round(4)}")
# Train mean: [0. 0. 0. 0. 0.]
# Train std: [1. 1. 1. 1. 1.]Now all five features live on the same scale, so a model trained on this data can weigh each one on its merits rather than its magnitude. This normalization step is standard practice before training almost any ML model — and in Phase 2 you'll meet a tool (scikit-learn's StandardScaler) that packages exactly the logic you just wrote by hand.
✍️ Practice Exercise
- Create a 5×3 matrix of random floats. Compute the mean of each column (feature means). Subtract the column means from every row (zero-centering). Verify that the result has column means very close to 0.
- Write a vectorized function that computes the softmax of a 1D array:
softmax(x) = exp(x) / sum(exp(x)). Verify that the output sums to exactly 1.0. (Softmax turns any list of numbers into positive fractions that sum to 1 — a recipe you'll use again and again from Phase 2 onward.) - Given two matrices A of shape (4, 3) and B of shape (3, 6), multiply them using
@. What shape is the result? Now try to multiply them in the wrong order: what error do you get and why?
▶ Show Solution
import numpy as np
# Task 1: zero-centering
X = np.random.rand(5, 3)
col_means = X.mean(axis=0) # shape (3,)
X_centered = X - col_means # broadcasting: (5,3) - (3,) → (5,3)
print(X_centered.mean(axis=0)) # ≈ [0., 0., 0.]
# Task 2: softmax
def softmax(x):
e = np.exp(x - x.max()) # subtract max for numerical stability
return e / e.sum()
scores = np.array([2.0, 1.0, 0.5, -1.0])
probs = softmax(scores)
print(probs) # [0.653, 0.24, 0.0..., ...] — sums to 1
print(probs.sum()) # 1.0
# Task 3: matrix multiplication shapes
A = np.random.rand(4, 3)
B = np.random.rand(3, 6)
C = A @ B
print(C.shape) # (4, 6) ← (m, k) @ (k, n) → (m, n)
try:
bad = B @ A # (3, 6) @ (4, 3) — inner dims 6 ≠ 4
except ValueError as e:
print(e) # matmul: Input operand 1 has a mismatch in its core dimension📚 Primary Source for This Lesson
NumPy Broadcasting Guide — official NumPy documentation
The authoritative, visual explanation of broadcasting rules with worked examples. After this lesson, read it end-to-end — the diagrams will cement your understanding of how shapes interact.