🎯 What You'll Learn
- What Pandas is and why it exists alongside NumPy
- Understand
Series(1D) andDataFrame(2D) — Pandas' two core structures - Load data from CSV files into a DataFrame
- Explore a dataset:
head,info,describe,shape,dtypes,value_counts - Select data using column names,
.loc(label-based), and.iloc(position-based) - Filter rows using boolean conditions
- Sort, add, and remove columns
Install with pip install pandas. Convention: import pandas as pd. You'll also want pip install matplotlib for the plots referenced in this lesson.
1 Why Pandas — What NumPy Can't Do
NumPy is excellent at fast numerical computation, but it has one major constraint: every element in an array must be the same type. Real-world datasets don't look like that. A customer record might have: name (string), age (int), signup date (datetime), account balance (float), and subscription tier (category). NumPy can't naturally handle this mix.
Pandas was created specifically for tabular, heterogeneous data — the kind you find in CSVs, SQL tables, Excel files, and database exports. It's what virtually every ML engineer uses to:
- Load and explore raw datasets
- Clean and fix data quality issues
- Engineer new features
- Split data into training and test sets
- Export processed data back to NumPy arrays for model training
Raw data → Pandas (explore, clean, engineer features) → NumPy array → model training (with tools you'll meet in Phases 2 and 4). Pandas is the gateway. Even if you eventually use frameworks that abstract away some of this, understanding Pandas is non-negotiable for every serious ML engineer.
2 The Two Core Data Structures
Series — 1D Labeled Array
A Series is a one-dimensional array with an index. Think of a single column in a spreadsheet. The index labels each value — by default it's 0, 1, 2, … but it can be anything (strings, dates, etc.).
import pandas as pd
import numpy as np
# Create a Series from a list
ages = pd.Series([28, 34, 45, 22, 38])
print(ages)
# 0 28
# 1 34
# 2 45
# 3 22
# 4 38
# dtype: int64
# Series with a custom index
scores = pd.Series(
[92, 85, 78, 95],
index=['Alice', 'Bob', 'Charlie', 'Diana']
)
print(scores['Alice']) # 92
print(scores.mean()) # 87.5
print(scores.dtype) # int64
# Series methods work like NumPy
print(scores[scores > 85])
# Alice 92
# Diana 95
# dtype: int64DataFrame — 2D Labeled Table
A DataFrame is a collection of Series objects that share the same index — essentially a table where each column is a Series. This is the structure you'll work with 95% of the time.
# Create a DataFrame from a dictionary of lists
data = {
'age': [28, 34, 45, 22, 38],
'salary': [65_000, 82_000, 110_000, 48_000, 95_000],
'years_exp':[3, 7, 15, 1, 12],
'promoted': [False, True, True, False, True],
'dept': ['Eng', 'Eng', 'Mgmt', 'Sales', 'Eng']
}
df = pd.DataFrame(data)
print(df)
# age salary years_exp promoted dept
# 0 28 65000 3 False Eng
# 1 34 82000 7 True Eng
# 2 45 110000 15 True Mgmt
# 3 22 48000 1 False Sales
# 4 38 95000 12 True Eng
print(type(df['salary'])) # <class 'pandas.core.series.Series'>
print(df.shape) # (5, 5) — 5 rows, 5 columns
print(df.dtypes)
# age int64
# salary int64
# years_exp int64
# promoted bool
# dept object ← strings are stored as 'object'Anatomy of the df built above: the blue-outlined column (age) is itself a Series — a 1D labeled array. Every column shares the same index (left, violet) so rows always line up across columns.
Because each column is just a Series of numbers, anything in a DataFrame can be plotted directly — no extra conversion needed. Here's the same 5-employee df from above, plotted two ways:
Each point is one row of df; hover to see the department (color) and promotion status (marker shape).
3 Loading Real Data
In practice, you almost never type data by hand. You load it from a file. CSV (comma-separated values) is the most common format.
import pandas as pd
# Load a CSV file from disk
df = pd.read_csv('data/customers.csv')
# Or load directly from a URL (great for practice datasets)
url = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
titanic = pd.read_csv(url)
# Other common formats
excel_df = pd.read_excel('data/report.xlsx')
json_df = pd.read_json('data/records.json')
sql_df = pd.read_sql('SELECT * FROM customers', connection) # with SQLAlchemy
# Save back to CSV
df.to_csv('data/cleaned_customers.csv', index=False) # index=False avoids saving the row numbersThroughout Phase 1–2 of this curriculum, we'll use the Titanic dataset (survival prediction) and the California Housing dataset (house price prediction). Both are freely available on Kaggle and scikit-learn's built-in datasets.
4 Exploring a Dataset — Your First 5 Minutes
Every time you open a new dataset, there's a standard sequence of commands you run to understand what you're dealing with. Here they are, illustrated with the Titanic dataset.
import pandas as pd
# Simulating the Titanic dataset for illustration
# (In practice: df = pd.read_csv('titanic.csv'))
df = pd.DataFrame({
'PassengerId': range(1, 9),
'Survived': [0, 1, 1, 1, 0, 0, 0, 1],
'Pclass': [3, 1, 3, 1, 3, 3, 1, 3],
'Name': ['Braund, Mr. Owen', 'Cumings, Mrs. John', 'Heikkinen, Miss. Laina',
'Futrelle, Mrs. Jacques', 'Allen, Mr. William', 'Moran, Mr. James',
'McCarthy, Mr. Timothy', 'Palsson, Master. Gosta'],
'Sex': ['male', 'female', 'female', 'female', 'male', 'male', 'male', 'male'],
'Age': [22.0, 38.0, 26.0, 35.0, None, None, 54.0, 2.0],
'Fare': [7.25, 71.28, 7.92, 53.1, 8.05, 8.46, 51.86, 21.07]
})
# ① How big is the dataset?
print(df.shape) # (8, 7) — 8 rows, 7 columns
# ② First few rows
print(df.head(3)) # first 3 rows (default: 5)
print(df.tail(2)) # last 2 rows
# ③ Column names and data types
print(df.dtypes)
# PassengerId int64
# Survived int64
# Pclass int64
# Name object
# Sex object
# Age float64 ← float because of NaN values
# Fare float64
# ④ Memory usage and non-null counts — catch missing values immediately
print(df.info())
# <class 'pandas.core.frame.DataFrame'>
# RangeIndex: 8 entries, 0 to 7
# Data columns (total 7 columns):
# # Column Non-Null Count Dtype
# --- ------ -------------- -----
# 0 PassengerId 8 non-null int64
# ...
# 5 Age 6 non-null float64 ← 2 missing!
# ...
# ⑤ Statistical summary of numerical columns
print(df.describe())
# PassengerId Survived Pclass ...
# count 8.000000 8.000000 8.000000
# mean 4.500000 0.500000 2.375000
# std 2.449490 0.534522 0.916048
# min 1.000000 0.000000 1.000000
# 25% 2.750000 0.000000 1.750000
# 50% 4.500000 0.500000 3.000000
# 75% 6.250000 1.000000 3.000000
# max 8.000000 1.000000 3.000000
# ⑥ For categorical columns, use value_counts
print(df['Sex'].value_counts())
# male 5
# female 3
# Name: Sex, dtype: int64
print(df['Survived'].value_counts(normalize=True)) # as proportions
# 0 0.5 ← 50% died
# 1 0.5 ← 50% survived5 Selecting Data: Columns, .loc, and .iloc
Pandas has three primary ways to select data. Knowing when to use each prevents both bugs and performance issues.
Column Selection
# Single column → returns a Series
ages = df['Age'] # preferred way — always works
ages = df.Age # attribute-style — only works if name has no spaces
# Multiple columns → returns a DataFrame
subset = df[['Age', 'Fare', 'Survived']]
print(subset.shape) # (8, 3).loc — Label-Based Selection
Use .loc when you want to select by row label (index value) and/or column name. This is the most readable approach.
df.loc[0] # row with index label 0 — returns a Series
df.loc[0, 'Age'] # row 0, column 'Age' → 22.0
df.loc[0:3, 'Age'] # rows 0–3 inclusive, 'Age' column ← NOTE: .loc slices are INCLUSIVE
df.loc[:, 'Age':'Fare'] # all rows, columns from 'Age' to 'Fare'
# .loc with a boolean condition (most common use in ML)
passengers_1st = df.loc[df['Pclass'] == 1]
print(passengers_1st.shape) # (2, 7)
survivors = df.loc[df['Survived'] == 1, ['Name', 'Age', 'Fare']]
print(survivors).iloc — Integer Position-Based Selection
Use .iloc when you want to select by position — like NumPy array indexing, but for DataFrames.
df.iloc[0] # first row by position
df.iloc[0, 2] # row 0, column index 2 (Pclass)
df.iloc[0:3, :] # first 3 rows, all columns ← NOTE: .iloc slices are EXCLUSIVE
df.iloc[:, 1:4] # all rows, columns at positions 1, 2, 3
# The "hold back some rows" pattern from Lesson 02, now in Pandas
n = len(df)
first_80 = df.iloc[:int(0.8 * n)] # first 80% of rows
last_20 = df.iloc[int(0.8 * n):] # last 20% of rowsdf.loc[0:3] returns rows labeled 0, 1, 2, and 3. df.iloc[0:3] returns rows at positions 0, 1, and 2 (position 3 excluded). This asymmetry is a notorious source of off-by-one bugs. Remember: loc = label (inclusive), iloc = integer position (exclusive like Python slices).
The three selection styles highlight different shapes of the same 8×3 grid (Name, Age, Fare columns from the Titanic-style df). Click a button to compare what each selection actually grabs:
Green = selected cells. Column selection grabs one full column; .loc[0:3, 'Age'] grabs rows labeled 0–3 (inclusive) in the Age column; .iloc[0:3, :] grabs rows at positions 0–2 (exclusive of 3) across every column.
6 Filtering, Sorting, and Adding Columns
Filtering Rows
# Single condition
adults = df[df['Age'] >= 18]
# Multiple conditions: use & (and), | (or), ~ (not)
# IMPORTANT: wrap each condition in parentheses!
young_1st = df[(df['Age'] < 30) & (df['Pclass'] == 1)]
survivors = df[(df['Survived'] == 1) | (df['Fare'] > 50)]
males = df[~(df['Sex'] == 'female')] # NOT female
# .isin() — check membership in a list
upper_class = df[df['Pclass'].isin([1, 2])]
# .between() — range filter (inclusive)
midage = df[df['Age'].between(25, 40)]and for Pandas boolean conditions
Python's and operator works on scalars. When you have two Series of True/False values, you need & (bitwise AND) instead. The same applies to | vs or, and ~ vs not. Forgetting the parentheses around each condition is also a very common mistake.
Sorting
df_sorted = df.sort_values('Age') # ascending by default
df_sorted = df.sort_values('Age', ascending=False) # descending
df_sorted = df.sort_values(['Pclass', 'Age']) # sort by multiple columnsAdding and Removing Columns
# Add a new column (computed from existing ones)
df['fare_per_age'] = df['Fare'] / df['Age'] # can produce NaN where Age=NaN
# Add a binary column
df['is_adult'] = (df['Age'] >= 18).astype(int) # True→1, False→0
# Add based on a condition
df['fare_category'] = pd.cut(
df['Fare'],
bins=[0, 20, 60, 600],
labels=['Low', 'Mid', 'High']
)
# Remove columns
df_no_name = df.drop(columns=['Name', 'PassengerId']) # returns new df
df.drop(columns=['Name'], inplace=True) # modifies in-place
print(df.columns.tolist())
# ['Survived', 'Pclass', 'Sex', 'Age', 'Fare', 'fare_per_age', 'is_adult', 'fare_category']7 Extracting Data for ML: Back to NumPy
After using Pandas to explore and clean your data, the final step is converting back to NumPy arrays — the format that model-training tools expect. Two new vocabulary words here: the columns a model learns from are the features (conventionally called X), and the column it learns to predict is the target (conventionally y). For the Titanic data, the features are things like age and fare, and the target is Survived — did this passenger live or die?
import pandas as pd
import numpy as np
# Suppose df has been cleaned and contains only numeric, non-null values
features = ['Age', 'Fare', 'Pclass', 'is_adult']
target = 'Survived'
X = df[features].values # .values converts DataFrame → NumPy array
y = df[target].values
print(type(X)) # <class 'numpy.ndarray'>
print(X.shape) # (n_samples, 4)
print(y.shape) # (n_samples,)
# Alternatively with .to_numpy(), which can also set the dtype
X = df[features].to_numpy(dtype=np.float32)Real-World Spotlight: First Look at the Titanic Dataset
The Titanic dataset (passenger survival from the 1912 disaster) is the classic first dataset in ML — the eventual goal is to predict who survived based on their other attributes. Companies like Airbnb and Spotify run exactly this kind of tabular analysis on their own data every day. Here's how a real ML engineer takes a first look:
import pandas as pd
# Load the dataset
url = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
df = pd.read_csv(url)
print(f"Dataset shape: {df.shape}") # (891, 12)
print(f"\nColumns:\n{df.columns.tolist()}")
# ['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp',
# 'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked']
# Survival rate overall
survival_rate = df['Survived'].mean()
print(f"\nOverall survival rate: {survival_rate:.1%}") # 38.4%
# Survival rate by gender — filter rows, then take the mean
# (next lesson you'll learn groupby, which does this in one line)
women = df[df['Sex'] == 'female']
men = df[df['Sex'] == 'male']
print(f"\nWomen survived: {women['Survived'].mean():.1%}") # 74.2%
print(f"Men survived: {men['Survived'].mean():.1%}") # 18.9%
# Survival rate by passenger class — same filter-then-mean pattern
print("\nSurvival rate by passenger class:")
for pclass in [1, 2, 3]:
rate = df[df['Pclass'] == pclass]['Survived'].mean()
print(f"Class {pclass}: {rate:.1%}")
# Class 1: 63.0% ← 63% of 1st class survived
# Class 2: 47.3%
# Class 3: 24.2% ← only 24% of 3rd class
# Age distribution
print(f"\nAge stats:\n{df['Age'].describe().round(1)}")
# count 714.0 ← 177 missing values!
# mean 29.7
# std 14.5
# min 0.4
# 25% 20.1
# 50% 28.0
# 75% 38.0
# max 80.0
# Missing values — critical insight
missing = df.isnull().sum()
print(f"\nMissing values:\n{missing[missing > 0]}")
# Age 177 ← must handle in Lesson 4
# Cabin 687 ← 77% missing! likely drop or engineer
# Embarked 2 ← easy to fillIn 15 lines of Pandas code, you've uncovered three major insights that will drive your entire modeling strategy: women and first-class passengers had much higher survival rates (→ Sex and Pclass will be strong features), and Age has significant missingness that needs to be addressed before training. This is what Exploratory Data Analysis looks like.
✍️ Practice Exercise
Load the Titanic dataset using the URL above and answer the following questions using Pandas:
- How many rows and columns does the full Titanic dataset have? How many passengers (rows) survived?
- Select only the
Name,Age, andFarecolumns for passengers in 1st class (Pclass == 1) who survived. How many rows does this have? - Add a new column
'family_size'=SibSp+Parch+ 1 (includes the passenger). Using boolean filtering, compare the mean family size of survivors vs non-survivors. - Use
.ilocto extract the first 100 rows and columns at positions 1, 2, 4, and 5 (Survived, Pclass, Sex, Age). What is the shape of the result?
▶ Show Solution
import pandas as pd
url = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
df = pd.read_csv(url)
# Task 1
print(df.shape) # (891, 12)
print(df['Survived'].sum()) # 342 survived
# Task 2
first_class_survivors = df.loc[
(df['Pclass'] == 1) & (df['Survived'] == 1),
['Name', 'Age', 'Fare']
]
print(first_class_survivors.shape) # (136, 3)
# Task 3
df['family_size'] = df['SibSp'] + df['Parch'] + 1
died_avg = df[df['Survived'] == 0]['family_size'].mean()
surv_avg = df[df['Survived'] == 1]['family_size'].mean()
print(f"Non-survivors: {died_avg:.2f}")
print(f"Survivors: {surv_avg:.2f}")
# Run it yourself — the two averages are close, which already
# hints family_size alone won't separate survivors from victims
# Task 4
subset = df.iloc[:100, [1, 2, 4, 5]]
print(subset.shape) # (100, 4)
print(subset.columns.tolist()) # ['Survived', 'Pclass', 'Sex', 'Age']📚 Primary Source for This Lesson
Pandas: Getting Started Tutorials — official Pandas documentation
Ten official tutorials from the Pandas team, covering DataFrames, indexing, aggregation, and more. This is the best single starting point for hands-on Pandas practice, written by the people who built the library.