🎯 What You'll Learn
- Why preprocessing is essential, and which algorithms are sensitive to feature scale
- Encode categorical variables using LabelEncoder, OrdinalEncoder, and OneHotEncoder — and when to use each
- Scale numerical features with StandardScaler, MinMaxScaler, and RobustScaler
- Apply the golden rule: fit on training data only, then transform both train and test
- Build robust preprocessing pipelines with
sklearn.PipelineandColumnTransformer
All tools in this lesson are part of scikit-learn: pip install scikit-learn pandas numpy. Import conventions used throughout: import pandas as pd, import numpy as np, from sklearn.preprocessing import ....
1 Why Preprocessing Matters
Raw data rarely comes in a form that ML algorithms can directly consume. Text categories, missing values, wildly different numerical scales — these all need to be addressed before training. But beyond convenience, preprocessing is mathematically necessary for many algorithm families:
(The algorithm names below are a preview of Lessons 11–28 — you don't need to know them yet. What matters is that every algorithm falls into one of these families, and each family has different preprocessing needs.)
- Distance-based algorithms (e.g., KNN — Lesson 16): these compute distances between data points. If one feature ranges 0–100 and another ranges 0–0.001, the first feature dominates every distance calculation — the second is effectively ignored. Scaling is essential.
- Gradient-based algorithms (e.g., linear and logistic regression — Lessons 11, 15): gradients are proportional to feature magnitudes. Features with large scales produce large gradients, causing instability and slow convergence. Scaling smooths the loss landscape.
- Tree-based algorithms (Lessons 22–24): these split on individual feature thresholds and are completely scale-invariant. Scaling doesn't help or hurt them.
- Algorithms that assume numeric input: virtually all sklearn estimators require numeric arrays. Categorical text columns must be encoded before training.
All preprocessing transformers must be fit on the training set only and then applied to both train and test sets. Fitting on the full dataset contaminates training with test statistics — a form of data leakage that inflates evaluation metrics. Using sklearn Pipelines enforces this rule automatically.
2 Categorical Encoding
Machine learning models work with numbers. Categorical features — "city", "product_type", "education_level" — must be converted to numeric representations. The method you choose depends on the nature of the category.
Label Encoding (for target variable or binary categories)
Maps each category to an integer. Appropriate for the target variable in classification, and for truly binary categories (yes/no). Never use for nominal features with more than 2 categories in tree-free models — it imposes a false ordinal relationship.
from sklearn.preprocessing import LabelEncoder
import pandas as pd
le = LabelEncoder()
y = ['cat', 'dog', 'cat', 'fish', 'dog']
y_encoded = le.fit_transform(y)
print(y_encoded) # [0 1 0 2 1]
print(le.classes_) # ['cat' 'dog' 'fish']
# Inverse: recover original labels from predictions
print(le.inverse_transform([0, 2])) # ['cat' 'fish']Ordinal Encoding (for ordered categories)
When a categorical variable has a meaningful order (Small < Medium < Large, Bronze < Silver < Gold), OrdinalEncoder lets you specify that order explicitly.
from sklearn.preprocessing import OrdinalEncoder
oe = OrdinalEncoder(categories=[['Low', 'Medium', 'High']])
X = [['Low'], ['High'], ['Medium'], ['Low'], ['High']]
X_encoded = oe.fit_transform(X)
print(X_encoded.ravel()) # [0. 2. 1. 0. 2.]
# Low→0, Medium→1, High→2 — order is preservedOne-Hot Encoding (for nominal categories)
For categories without natural ordering (city names, product types, job titles), one-hot encoding creates a new binary column for each category. A row gets a 1 in the column for its category and 0 in all others. This avoids the false ordinal relationship.
from sklearn.preprocessing import OneHotEncoder
import numpy as np
# Nominal feature: no natural order between cities
cities = [['London'], ['Paris'], ['Tokyo'], ['London'], ['Berlin']]
ohe = OneHotEncoder(sparse_output=False) # dense matrix for readability
encoded = ohe.fit_transform(cities)
print(ohe.categories_) # [array(['Berlin', 'London', 'Paris', 'Tokyo'])]
print(encoded)
# [[0. 1. 0. 0.] # London
# [0. 0. 1. 0.] # Paris
# [0. 0. 0. 1.] # Tokyo
# [0. 1. 0. 0.] # London
# [1. 0. 0. 0.]] # Berlin
# Pandas equivalent for quick exploration
df = pd.DataFrame({'city': ['London', 'Paris', 'Tokyo', 'London']})
print(pd.get_dummies(df, drop_first=True))
# city_Paris city_Tokyo city_London (Berlin dropped as reference)
# Use drop_first=True to avoid the dummy variable trapOne-hot encoding the cities column from the code above: 1 nominal column with 4 categories becomes 4 binary columns. Hover a cell to see which row/category it represents.
If you have k categories and create k binary columns, the last column is perfectly predictable from the others — they always sum to 1. This creates multicollinearity, which destabilises linear models. The fix: drop one category column (use drop='first' in OneHotEncoder or drop_first=True in pd.get_dummies). Tree-based models are immune — you can skip drop='first' for them.
Handling Unseen Categories
# OneHotEncoder by default raises an error on unseen categories in test set.
# Use handle_unknown='ignore' to replace unseen categories with all zeros.
ohe = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
ohe.fit(X_train_cats)
# Test data might have a city the model never saw in training:
X_test_transformed = ohe.transform(X_test_cats) # no error — unseen → all zeros3 Feature Scaling
Feature scaling transforms numerical features so they occupy comparable ranges. The three most important scalers in scikit-learn each have different assumptions and use cases:
StandardScaler — z-score normalization
Centers each feature to mean=0 and standard deviation=1. The resulting values are called z-scores. This is the default choice for most gradient-based and distance-based algorithms.
from sklearn.preprocessing import StandardScaler
import numpy as np
X = np.array([[1000, 2],
[2000, 3],
[3000, 5],
[4000, 4]])
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
print("Means: ", scaler.mean_) # [2500. 3.5]
print("Std: ", scaler.scale_) # [1118.03 1.12]
print("Scaled:\n", X_scaled)
# [[-1.342 -1.342]
# [-0.447 -0.447]
# [ 0.447 1.342]
# [ 1.342 0.447]]
# Each column now has mean≈0 and std=1MinMaxScaler — range normalization
Scales features to a fixed range, default [0, 1]. Preserves zero values and is useful when you need bounded outputs (e.g., feeding into a sigmoid layer or image pixel normalization).
from sklearn.preprocessing import MinMaxScaler
mms = MinMaxScaler(feature_range=(0, 1))
X_mms = mms.fit_transform(X)
print(X_mms)
# [[0. 0. ]
# [0.333 0.333]
# [0.667 1. ]
# [1. 0.667]]
# Values now in [0, 1] for both columns
# MinMaxScaler is sensitive to outliers — one extreme value compresses everything else
# e.g., if one house sold for $50M in a dataset of typical homes, all others get squished near 0RobustScaler — median and IQR normalization
Uses the median and interquartile range (IQR) instead of mean and standard deviation. Since median and IQR are insensitive to outliers, this scaler is the right choice when your data contains significant outliers you want to preserve (not remove).
from sklearn.preprocessing import RobustScaler
X_with_outliers = np.array([[1, 2], [2, 3], [3, 4], [4, 5], [100, 6]])
rs = RobustScaler()
X_robust = rs.fit_transform(X_with_outliers)
print("Median: ", rs.center_) # [3. 4.]
print("IQR: ", rs.scale_) # [2. 2.]
print(X_robust)
# [[-1. -1. ]
# [-0.5 -0.5 ]
# [ 0. 0. ]
# [ 0.5 0.5 ]
# [48.5 1. ]] ← outlier (100) still stands out but doesn't crush other values| Scaler | Formula | Output range | When to use |
|---|---|---|---|
| StandardScaler | (x − μ) / σ | Unbounded, mean=0 std=1 | ⭐ Default for linear models, SVMs, KNN, PCA |
| MinMaxScaler | (x − min) / (max − min) | [0, 1] (or custom) | Image data, neural network input layers |
| RobustScaler | (x − median) / IQR | Unbounded, robust to outliers | Data with significant outliers you want to keep |
Scaling changes the axis a feature lives on, but it never changes the shape of its distribution. Below is a simulated sqft feature (modeled on the house-size column used throughout this lesson) shown raw, then after each scaler — notice the histogram outline stays identical, only the x-axis numbers move.
Before — raw sqft values
After — StandardScaler output
4 The Critical Rule: Fit on Train Only
This is the most commonly violated rule in data preprocessing, and it produces the most insidious bugs — because the code runs fine, evaluations look great, and the problem only surfaces in production.
The Wrong Way (Data Leakage)
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
import numpy as np
X = np.random.randn(1000, 5)
y = np.random.randint(0, 2, 1000)
# ❌ WRONG — fit on the FULL dataset before splitting
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X) # uses test statistics!
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)
# The test set's mean and std leaked into the scaler
# Evaluation metrics are optimistically biasedThe Right Way
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
import numpy as np
X = np.random.randn(1000, 5)
y = np.random.randint(0, 2, 1000)
# ✓ CORRECT — split FIRST, then fit scaler on training data only
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) # learns mean/std FROM TRAIN ONLY
X_test_scaled = scaler.transform(X_test) # applies train statistics to test
# The distinction between fit_transform and transform:
# fit_transform(X_train) = fit(X_train) → transform(X_train) [computes AND applies]
# transform(X_test) = apply training stats to test data [applies only]
print("Train scaler mean:", scaler.mean_) # computed from training set onlyIf you scale using the full dataset, the scaler learns statistics (mean, std) that include test data. During evaluation, the model "saw" test statistics during preprocessing — it's like giving away answers before the exam. The model appears to generalize better than it truly does. When deployed on genuinely new data, it performs worse than expected.
5 Train/Test Split Best Practices
from sklearn.model_selection import train_test_split
import pandas as pd
df = pd.read_csv('house_prices.csv')
X = df.drop('price', axis=1)
y = df['price']
# Basic split
X_train, X_test, y_train, y_test = train_test_split(
X, y,
test_size=0.2, # 80% train, 20% test
random_state=42, # reproducibility — always set this
)
print(X_train.shape, X_test.shape) # (1168, 79) (292, 79)
# For CLASSIFICATION — use stratify to preserve class proportions
df_clf = pd.read_csv('churn.csv')
X_clf = df_clf.drop('churned', axis=1)
y_clf = df_clf['churned']
X_train, X_test, y_train, y_test = train_test_split(
X_clf, y_clf,
test_size=0.2,
random_state=42,
stratify=y_clf, # ensures train/test have same churn rate
)
# Verify stratification worked
print("Train churn rate:", y_train.mean()) # e.g., 0.142
print("Test churn rate: ", y_test.mean()) # e.g., 0.142 ← same!If your dataset is 95% class 0 and 5% class 1, a random split might put all class-1 examples in the training set by chance, leaving you with an empty class in the test set. stratify=y guarantees both splits maintain the original 95/5 ratio. Always use it for classification.
6 sklearn Pipelines: The Right Way to Chain Steps
The manual approach to preprocessing — split, fit scaler, transform train, transform test — is error-prone and verbose. The elegant solution is a Pipeline: a single object that chains multiple steps and applies the fit-on-train rule automatically.
A single Pipeline (with a ColumnTransformer inside it) chains every preprocessing step and the model into one object — so cross_val_score and .fit() always refit preprocessing on the correct training fold only.
A pipeline needs a model at the end, but the models themselves start next lesson. So here we borrow two from scikit-learn (LogisticRegression, RandomForestClassifier) and treat them as black boxes with three buttons: .fit(X, y) learns from data, .predict(X) makes predictions, and .score(X, y) reports accuracy — the fraction of predictions that were correct. What happens inside these boxes is the subject of Lessons 11–23. Today, keep your eyes on everything that happens before the model.
Simple Pipeline: Scaler + Classifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
# Build the pipeline
pipeline = Pipeline([
('scaler', StandardScaler()), # step 1: scale features
('classifier', LogisticRegression(max_iter=1000)) # step 2: fit model
])
# Training: fit() calls scaler.fit_transform(X_train) then classifier.fit(X_train_scaled, y_train)
pipeline.fit(X_train, y_train)
# Prediction: transform() calls scaler.transform(X_test) then classifier.predict(X_test_scaled)
y_pred = pipeline.predict(X_test)
print("Test accuracy:", pipeline.score(X_test, y_test)) # 0.9737
# Cross-validation — leakage-safe because Pipeline fits scaler separately in each fold
cv_scores = cross_val_score(pipeline, X_train, y_train, cv=5, scoring='accuracy')
print(f"CV accuracy: {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")
# CV accuracy: 0.9758 ± 0.0085cross_val_score — what those 5 numbers are
That last call did something clever: it split the training data into 5 equal folds, trained the pipeline on 4 folds and scored it on the 5th, then rotated so each fold took one turn as the scorer — 5 trainings, 5 scores, and we report their mean ± spread. This is k-fold cross-validation, and it gives a far more trustworthy estimate than a single split, because no one lucky or unlucky split can mislead you. You'll use it in nearly every lesson from here on, and Lesson 20 refines it for classification problems.
When you use cross_val_score(pipeline, ...), sklearn fits the entire pipeline independently on each training fold. The scaler's statistics are computed only from that fold's training data — not from validation data. Without a Pipeline, if you pre-scale before cross-validation, you've leaked validation statistics into every fold's training process.
7 ColumnTransformer for Mixed Data Types
Real datasets typically have a mix of numerical and categorical columns, each requiring different preprocessing. ColumnTransformer applies different transformers to different subsets of columns and concatenates the results.
import pandas as pd
import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
# Sample dataset: house price classification (affordable vs not)
data = {
'sqft': [1200, 2300, 1800, 3200, 950],
'bedrooms': [2, 4, 3, 5, 1],
'neighborhood': ['Downtown', 'Suburbs', 'Downtown', 'Rural', 'Suburbs'],
'house_style': ['Ranch', 'Colonial', 'Ranch', 'Colonial', 'Ranch'],
'affordable': [1, 0, 1, 0, 1],
}
df = pd.DataFrame(data)
X = df.drop('affordable', axis=1)
y = df['affordable']
# Identify column types
numeric_cols = ['sqft', 'bedrooms']
categorical_cols = ['neighborhood', 'house_style']
# Build transformers for each column type
numeric_transformer = Pipeline([
('scaler', StandardScaler()),
])
categorical_transformer = Pipeline([
('onehot', OneHotEncoder(handle_unknown='ignore', drop='first')),
])
# Combine into a ColumnTransformer
preprocessor = ColumnTransformer(
transformers=[
('num', numeric_transformer, numeric_cols),
('cat', categorical_transformer, categorical_cols),
],
remainder='drop' # drop any columns not listed above
)
# Wrap preprocessor + model in a final Pipeline
full_pipeline = Pipeline([
('preprocessor', preprocessor),
('classifier', LogisticRegression(max_iter=500)),
])
# Train and evaluate
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)
full_pipeline.fit(X_train, y_train)
print("Test accuracy:", full_pipeline.score(X_test, y_test))
# Inspect the output shape after preprocessing
preprocessor.fit(X_train)
X_transformed = preprocessor.transform(X_train)
print("Transformed shape:", X_transformed.shape)
# (3, 5): 2 numeric features + (3-1=2) neighborhood cols + (2-1=1) style col = 5 totalUsing make_column_transformer (shorthand)
from sklearn.compose import make_column_transformer
# Shorter syntax for simple cases
preprocessor = make_column_transformer(
(StandardScaler(), numeric_cols),
(OneHotEncoder(handle_unknown='ignore'), categorical_cols),
remainder='passthrough' # keep other columns unchanged
)8 Handling Missing Values in Preprocessing
Missing values must be handled before most sklearn estimators can run. The cleanest approach is to add imputation as the first step in a Pipeline or ColumnTransformer, so it too is fit on training data only.
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
import numpy as np
import pandas as pd
# Simulate a dataset with missing values
np.random.seed(42)
n = 200
df = pd.DataFrame({
'age': np.random.normal(40, 10, n),
'income': np.random.normal(60000, 15000, n),
'credit_score': np.random.normal(700, 50, n),
'employment': np.random.choice(['Full-time', 'Part-time', 'Self-employed', None], n),
'home_owner': np.random.choice(['Yes', 'No', None], n),
})
# Introduce some missing values
df.loc[df.sample(20).index, 'age'] = np.nan
df.loc[df.sample(15).index, 'income'] = np.nan
numeric_cols = ['age', 'income', 'credit_score']
categorical_cols = ['employment', 'home_owner']
# Numeric pipeline: impute with mean, then scale
numeric_pipeline = Pipeline([
('imputer', SimpleImputer(strategy='mean')), # missing → column mean
('scaler', StandardScaler()),
])
# Categorical pipeline: impute with most frequent value, then one-hot encode
categorical_pipeline = Pipeline([
('imputer', SimpleImputer(strategy='most_frequent')), # missing → mode
('encoder', OneHotEncoder(handle_unknown='ignore', sparse_output=False)),
])
preprocessor = ColumnTransformer([
('num', numeric_pipeline, numeric_cols),
('cat', categorical_pipeline, categorical_cols),
])
# Final pipeline with classifier
model_pipeline = Pipeline([
('prep', preprocessor),
('model', RandomForestClassifier(n_estimators=100, random_state=42)),
])
# imputer.fit() learns mean from TRAINING set only → no leakage
y = np.random.randint(0, 2, n)
X = df
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model_pipeline.fit(X_train, y_train)
print("Accuracy:", model_pipeline.score(X_test, y_test))strategy='mean' — best for roughly symmetric numeric distributions. strategy='median' — better for skewed numeric distributions (e.g., income, house prices). strategy='most_frequent' — for categorical features and binary variables. strategy='constant', fill_value=0 — when absence of a value is itself informative (e.g., "no prior loans"). For complex patterns, consider IterativeImputer (experimental) which uses other features to predict missing values.
Real-World Spotlight: House Price Prediction Pipeline
The Ames Housing dataset is a classic real-world dataset with 79 features — a mix of numeric and categorical — and significant missing values. Here's a complete preprocessing pipeline:
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error
# Load Ames Housing (from kaggle or sklearn.datasets)
df = pd.read_csv('ames_housing.csv')
X = df.drop('SalePrice', axis=1)
y = np.log1p(df['SalePrice']) # log-transform target to reduce skew
# Identify column types
numeric_features = X.select_dtypes(include=['int64', 'float64']).columns.tolist()
categorical_features = X.select_dtypes(include=['object']).columns.tolist()
print(f"Numeric features: {len(numeric_features)}") # e.g., 36
print(f"Categorical features: {len(categorical_features)}") # e.g., 43
# Build full preprocessing pipeline
numeric_transformer = Pipeline([
('imputer', SimpleImputer(strategy='median')),
('scaler', StandardScaler()),
])
categorical_transformer = Pipeline([
('imputer', SimpleImputer(strategy='most_frequent')),
('encoder', OneHotEncoder(handle_unknown='ignore', sparse_output=False)),
])
preprocessor = ColumnTransformer([
('num', numeric_transformer, numeric_features),
('cat', categorical_transformer, categorical_features),
])
full_pipeline = Pipeline([
('prep', preprocessor),
('model', Ridge(alpha=10.0)),
])
# Train/test split — no stratify for regression
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
full_pipeline.fit(X_train, y_train)
y_pred = full_pipeline.predict(X_test)
rmse = mean_squared_error(y_test, y_pred, squared=False)
print(f"Test RMSE (log scale): {rmse:.4f}") # ~0.131
print(f"Test R²: {full_pipeline.score(X_test, y_test):.4f}") # ~0.881
# Cross-validation to verify
cv_rmse = cross_val_score(full_pipeline, X_train, y_train,
cv=5, scoring='neg_root_mean_squared_error')
print(f"CV RMSE: {(-cv_rmse).mean():.4f} ± {cv_rmse.std():.4f}")Quick Check
✍️ Practice Exercises
- Load any dataset with a mix of numerical and categorical columns. Identify which columns need OrdinalEncoder (there's an ordering) and which need OneHotEncoder (no ordering). Build a ColumnTransformer that applies the appropriate encoder to each.
- Write code that intentionally does it wrong (fit scaler on full data before splitting) and correctly (fit on train only). Use
cross_val_scoreto measure accuracy for both approaches. Observe the inflated performance from the wrong approach. - Build a complete Pipeline for the Titanic dataset: impute 'Age' with median, 'Embarked' with most_frequent; one-hot encode 'Sex' and 'Embarked'; StandardScale 'Age', 'Fare', 'Pclass'; train a LogisticRegression. Evaluate with 5-fold cross-validated accuracy.
- After fitting a pipeline, extract the fitted preprocessing steps to inspect what the scaler learned:
pipeline.named_steps['scaler'].mean_. Do the same for a pipeline with a ColumnTransformer. How do you access the numeric sub-pipeline's scaler?
▶ Show Titanic pipeline solution
import pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
df = pd.read_csv('titanic.csv')
X = df[['Age', 'Fare', 'Pclass', 'Sex', 'Embarked']]
y = df['Survived']
numeric_transformer = Pipeline([
('imputer', SimpleImputer(strategy='median')),
('scaler', StandardScaler()),
])
categorical_transformer = Pipeline([
('imputer', SimpleImputer(strategy='most_frequent')),
('encoder', OneHotEncoder(drop='first', handle_unknown='ignore', sparse_output=False)),
])
preprocessor = ColumnTransformer([
('num', numeric_transformer, ['Age', 'Fare', 'Pclass']),
('cat', categorical_transformer, ['Sex', 'Embarked']),
])
pipeline = Pipeline([
('prep', preprocessor),
('model', LogisticRegression(max_iter=200)),
])
scores = cross_val_score(pipeline, X, y, cv=5, scoring='accuracy')
print(f"CV Accuracy: {scores.mean():.4f} ± {scores.std():.4f}")📚 Primary Source for This Lesson
scikit-learn User Guide: Preprocessing Data
The official, authoritative documentation for all preprocessing tools covered in this lesson. Includes examples for every scaler and encoder, explanations of the mathematical transformations, and guidance on when to use each. Read sections 6.3 (preprocessing), 6.5 (pipelines), and 6.7 (column transformer).
df.info() output and your pipeline code — your AI tutor will help you debug it.