🔵 Intermediate

File I/O

📖 Lesson 14 ⏱ 35 min 🧪 5 questions 💻 3 exercises

🎯 Learning Objectives

Open, read, and write text and binary files safely
Use the with statement to guarantee file closure
Choose the right read method for the job (read, readline, iteration)
Work with file paths using pathlib.Path
Read and write CSV and JSON files
Apply best practices: atomic writes, large-file streaming, encoding awareness

Why File I/O?

Programs that can only work with in-memory data lose everything when they stop. File I/O is how Python programs persist data — reading configuration, saving results, processing logs, exchanging data with other systems.

Python makes file operations simple, but there are a handful of rules that prevent resource leaks, data corruption, and encoding headaches. This lesson covers them all.

Opening Files — `open()`

The built-in open() function returns a file object. Its two most important arguments are the path and the mode:

f = open("data.txt", "r")   # open for reading (default)
# ... do work ...
f.close()                    # MUST close to flush buffers and release the OS handle

open_basic.py

Mode	Meaning	File must exist?
`"r"`	Read (default)	Yes — raises `FileNotFoundError`
`"w"`	Write — truncates existing content	No — creates if absent
`"a"`	Append — writes at end	No — creates if absent
`"x"`	Exclusive create — fails if file exists	No — creates only
`"r+"`	Read + write (no truncate)	Yes
`"b"`	Binary mode — combine with above: `"rb"`, `"wb"`	—
`"t"`	Text mode (default) — combine: `"rt"`, `"wt"`	—

Always specify an explicit encoding for text files. The default varies by OS — UTF-8 on Linux/macOS, often CP1252 on Windows. Use encoding="utf-8" to be safe and portable.

The `with` Statement

Using with open(...) is the only correct way to open files in production code. It guarantees the file is closed even if an exception is raised:

# ✓ Correct — file is always closed when the block exits
with open("data.txt", "r", encoding="utf-8") as f:
    content = f.read()
# f is now closed — even if an exception was raised inside the block

# ❌ Fragile — if an exception occurs, close() is never called
f = open("data.txt", "r")
content = f.read()
f.close()  # might not be reached!

with_statement.py

You can open multiple files in one with block:

with open("input.txt", "r", encoding="utf-8") as src, \
     open("output.txt", "w", encoding="utf-8") as dst:
    for line in src:
        dst.write(line.upper())

multi_with.py

with works with any context manager — an object that implements __enter__ and __exit__. File objects do this automatically. You'll see with used for database connections, locks, and more (Lesson 23).

Reading Files

with open("data.txt", "r", encoding="utf-8") as f:

    # read() — entire file as one string
    content = f.read()
    print(type(content))   # <class 'str'>

# --- reopen to reset position ---

with open("data.txt", "r", encoding="utf-8") as f:
    # readline() — one line at a time (includes the trailing '\n')
    first  = f.readline()
    second = f.readline()

with open("data.txt", "r", encoding="utf-8") as f:
    # readlines() — all lines as a list of strings
    lines = f.readlines()
    print(lines)   # ['line 1\n', 'line 2\n', 'line 3\n']

with open("data.txt", "r", encoding="utf-8") as f:
    # Iteration — best for large files; reads one line at a time
    for line in f:
        print(line.rstrip("\n"))   # strip the trailing newline

reading.py

Method	Returns	Best for
`f.read()`	One big string	Small files you need whole
`f.read(n)`	Up to n characters	Fixed-size chunks
`f.readline()`	One line (with `\n`)	Parsing line by line with state
`f.readlines()`	List of all lines	Small files; need random access to lines
`for line in f`	One line per iteration	Large files — memory efficient

For large files always iterate with for line in f or read in chunks. f.read() loads the entire file into memory — a 2 GB log file will use 2 GB of RAM.

Writing Files

# write() — write a string; returns the number of characters written
with open("output.txt", "w", encoding="utf-8") as f:
    f.write("Hello, world!\n")
    f.write("Second line\n")

# writelines() — write a list/iterable of strings (no newlines added!)
lines = ["apple\n", "banana\n", "cherry\n"]
with open("fruits.txt", "w", encoding="utf-8") as f:
    f.writelines(lines)

# print() also writes to files via the file= parameter
with open("log.txt", "w", encoding="utf-8") as f:
    print("Error: something failed", file=f)
    print("Code: 42", file=f)

writing.py

Opening a file in "w" mode immediately erases all its existing content, even before you write a single byte. If you want to preserve existing content, use "a" (append) instead.

Appending to a file

import datetime

def log_event(message, path="events.log"):
    timestamp = datetime.datetime.now().isoformat(timespec="seconds")
    with open(path, "a", encoding="utf-8") as f:
        f.write(f"[{timestamp}] {message}\n")

log_event("Server started")
log_event("User logged in: alice")
log_event("Request processed in 120ms")

append.py

Working with Paths — `pathlib`

pathlib.Path is the modern, object-oriented way to work with file system paths. It's cross-platform (handles / vs \) and much cleaner than string manipulation.

from pathlib import Path

# Create a Path object — doesn't touch the filesystem yet
p = Path("data") / "reports" / "summary.txt"
print(p)            # data/reports/summary.txt
print(p.name)       # summary.txt
print(p.stem)       # summary
print(p.suffix)     # .txt
print(p.parent)     # data/reports

# Check existence
print(p.exists())   # False (doesn't exist yet)
print(p.is_file())  # False
print(p.is_dir())   # False

# Create directories
p.parent.mkdir(parents=True, exist_ok=True)

# Read/write convenience methods
p.write_text("Hello from pathlib!\n", encoding="utf-8")
content = p.read_text(encoding="utf-8")
print(content)   # Hello from pathlib!

# List directory contents
src = Path(".")
for item in src.iterdir():
    print(item.name, "DIR" if item.is_dir() else "FILE")

# Glob — find files by pattern
for py_file in src.glob("**/*.py"):
    print(py_file)

pathlib_tour.py

from pathlib import Path

p = Path("data/reports/summary.txt")

# Rename / move
p.rename("data/reports/summary_v2.txt")

# Delete a file
Path("data/reports/summary_v2.txt").unlink(missing_ok=True)

# Delete an empty directory
Path("data/reports").rmdir()

# File size in bytes
size = Path("large_file.bin").stat().st_size
print(f"{size:,} bytes")

pathlib_ops.py

Use Path(__file__).parent to get the directory of the currently running script — reliable regardless of the working directory when you invoke Python.

CSV Files

CSV (Comma-Separated Values) is one of the most common data exchange formats. Python's csv module handles quoting, escaping, and dialect differences correctly — never parse CSV with a plain split(",").

import csv

# ── Writing CSV ──
employees = [
    ["name", "department", "salary"],
    ["Alice", "Engineering", 95000],
    ["Bob", "Marketing", 78000],
    ["Carol", "Engineering", 102000],
]

with open("employees.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerows(employees)

# ── Reading CSV ──
with open("employees.csv", "r", newline="", encoding="utf-8") as f:
    reader = csv.reader(f)
    header = next(reader)         # consume the header row
    for row in reader:
        print(row)
# ['Alice', 'Engineering', '95000']
# ['Bob', 'Marketing', '78000']
# ['Carol', 'Engineering', '102000']

csv_basic.py

import csv

# ── DictWriter — write with headers as keys ──
employees = [
    {"name": "Alice", "department": "Engineering", "salary": 95000},
    {"name": "Bob",   "department": "Marketing",   "salary": 78000},
]

with open("employees.csv", "w", newline="", encoding="utf-8") as f:
    fields = ["name", "department", "salary"]
    writer = csv.DictWriter(f, fieldnames=fields)
    writer.writeheader()
    writer.writerows(employees)

# ── DictReader — each row is a dict with header keys ──
with open("employees.csv", "r", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(f"{row['name']} earns £{int(row['salary']):,}")

csv_dict.py

Always pass newline="" when opening CSV files — the csv module handles line endings itself. Without it you may get blank rows on Windows (\r\n double-counted).

JSON Files

JSON is the lingua franca of data exchange on the web. Python's json module converts between Python objects and JSON strings seamlessly.

import json

data = {
    "name": "Alice",
    "age": 30,
    "languages": ["Python", "Rust", "Go"],
    "active": True,
    "score": 9.5
}

# ── Write JSON to file ──
with open("user.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2)   # indent makes it human-readable

# ── Read JSON from file ──
with open("user.json", "r", encoding="utf-8") as f:
    loaded = json.load(f)

print(loaded["name"])        # Alice
print(loaded["languages"])   # ['Python', 'Rust', 'Go']
print(type(loaded["active"]))  # <class 'bool'>

# ── String ↔ object (without files) ──
json_str = json.dumps(data, indent=2)   # Python → JSON string
obj       = json.loads(json_str)         # JSON string → Python

json_files.py

Python type	JSON type
`dict`	object `{}`
`list`, `tuple`	array `[]`
`str`	string
`int`, `float`	number
`True` / `False`	`true` / `false`
`None`	`null`

import json
from datetime import datetime

# Custom types (e.g. datetime) need a custom encoder
class DateTimeEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime):
            return obj.isoformat()
        return super().default(obj)

event = {"name": "deploy", "time": datetime.now()}
print(json.dumps(event, cls=DateTimeEncoder))
# {"name": "deploy", "time": "2024-03-15T10:30:00.000000"}

json_custom.py

Binary Files

Use binary mode ("rb" / "wb") when working with images, PDFs, audio, or any file that is not plain text. Data is read/written as bytes objects, not strings.

# Copy any file in binary mode — works for text AND binary
def copy_file(src, dst):
    with open(src, "rb") as f_in, open(dst, "wb") as f_out:
        while chunk := f_in.read(65536):   # read 64 KB at a time
            f_out.write(chunk)

copy_file("photo.jpg", "photo_backup.jpg")

# Read PNG header to check magic bytes
with open("photo.png", "rb") as f:
    header = f.read(8)
    is_png = header[:4] == b'\x89PNG'
    print("PNG file:", is_png)

binary.py

The walrus operator (:=) in while chunk := f.read(65536) assigns and tests in one step — the loop stops when read() returns an empty bytes object at end-of-file.

Common Patterns

Atomic write — prevent partial writes on error

import os
from pathlib import Path

def atomic_write(path, content, encoding="utf-8"):
    """Write to a temp file first, then rename — guarantees the final
    file is either the old version or the new version, never corrupt."""
    tmp = Path(str(path) + ".tmp")
    try:
        tmp.write_text(content, encoding=encoding)
        tmp.replace(path)   # atomic rename on the same filesystem
    except Exception:
        tmp.unlink(missing_ok=True)
        raise

atomic_write("config.json", '{"version": 2}')

atomic_write.py

Read a config file with a default fallback

import json
from pathlib import Path

DEFAULT_CONFIG = {"debug": False, "port": 8080, "host": "localhost"}

def load_config(path="config.json"):
    p = Path(path)
    if not p.exists():
        return DEFAULT_CONFIG.copy()
    with p.open(encoding="utf-8") as f:
        user_cfg = json.load(f)
    return {**DEFAULT_CONFIG, **user_cfg}   # user overrides defaults

config = load_config()
print(config["port"])  # 8080 (or whatever the file says)

config_load.py

Process a large file line-by-line

from pathlib import Path

def count_errors(log_path):
    """Count lines containing 'ERROR' without loading the whole file."""
    count = 0
    with open(log_path, "r", encoding="utf-8", errors="replace") as f:
        for line in f:
            if "ERROR" in line:
                count += 1
    return count

print(count_errors("application.log"))

large_file.py

Best Practices

Always use with open(…) — never call open() without a context manager.
Always specify encoding="utf-8" for text files — don't rely on the platform default.
Iterate over large files (for line in f) rather than calling f.read().
Use pathlib.Path over raw string path manipulation for clarity and cross-platform safety.
Use newline="" when opening CSV files.
Use "x" mode (exclusive create) when creating a file that must not overwrite an existing one.
Atomic writes for critical data — write to a temp file, then rename.
Handle FileNotFoundError and PermissionError explicitly — don't let I/O errors crash your entire program silently.

from pathlib import Path

def safe_read(path, default=""):
    try:
        return Path(path).read_text(encoding="utf-8")
    except FileNotFoundError:
        return default
    except PermissionError as e:
        raise RuntimeError(f"Cannot read {path}: permission denied") from e

print(safe_read("notes.txt", default="(no notes yet)"))

safe_read.py

Primary sources: Python Docs — Reading and Writing Files · Python Docs — pathlib · Python Docs — csv · Python Docs — json

🤖

Ask your AI tutor! Getting a UnicodeDecodeError? Not sure whether to use read() or iterate line-by-line? Need to parse a non-standard CSV? These are great questions to work through together.

💻 Exercises

01 Log File Analyser

Write a function analyse_log(path) that reads a plain-text log file (one entry per line) and returns a dict with:

"total": total number of lines
"errors": count of lines containing "ERROR"
"warnings": count of lines containing "WARNING"
"first_error": the first line containing "ERROR", or None

Show solution

def analyse_log(path):
    stats = {"total": 0, "errors": 0, "warnings": 0, "first_error": None}
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            line = line.rstrip("\n")
            stats["total"] += 1
            if "ERROR" in line:
                stats["errors"] += 1
                if stats["first_error"] is None:
                    stats["first_error"] = line
            if "WARNING" in line:
                stats["warnings"] += 1
    return stats

# Test — create a sample log first:
from pathlib import Path
Path("sample.log").write_text(
    "INFO server started\nWARNING disk at 80%\nERROR connection refused\nINFO request ok\n",
    encoding="utf-8"
)
print(analyse_log("sample.log"))
# {'total': 4, 'errors': 1, 'warnings': 1, 'first_error': 'ERROR connection refused'}

02 CSV Transformer

Read employees.csv (columns: name, department, salary) and write a new file employees_raised.csv where every salary has been increased by 10 %. Use DictReader / DictWriter.

Show solution

import csv

with open("employees.csv", "r", newline="", encoding="utf-8") as fin, \
     open("employees_raised.csv", "w", newline="", encoding="utf-8") as fout:

    reader = csv.DictReader(fin)
    writer = csv.DictWriter(fout, fieldnames=reader.fieldnames)
    writer.writeheader()

    for row in reader:
        row["salary"] = round(float(row["salary"]) * 1.10, 2)
        writer.writerow(row)

# Verify
with open("employees_raised.csv", "r", newline="", encoding="utf-8") as f:
    for row in csv.DictReader(f):
        print(row)

03 Config File Manager

Write a class ConfigManager that:

Takes a file path and a dict of defaults in __init__
load() — reads the JSON file, merges with defaults (file values override defaults)
save() — writes the current config to the JSON file atomically (write to .tmp, then rename)
get(key) / set(key, value) — read and write individual keys

Show solution

import json
from pathlib import Path

class ConfigManager:
    def __init__(self, path, defaults=None):
        self.path = Path(path)
        self._data = dict(defaults or {})
        self.load()

    def load(self):
        if self.path.exists():
            with self.path.open(encoding="utf-8") as f:
                file_data = json.load(f)
            self._data = {**self._data, **file_data}

    def save(self):
        tmp = Path(str(self.path) + ".tmp")
        try:
            tmp.write_text(json.dumps(self._data, indent=2), encoding="utf-8")
            tmp.replace(self.path)
        except Exception:
            tmp.unlink(missing_ok=True)
            raise

    def get(self, key, default=None):
        return self._data.get(key, default)

    def set(self, key, value):
        self._data[key] = value

# Usage
cfg = ConfigManager("app_config.json", defaults={"debug": False, "port": 8080})
print(cfg.get("port"))    # 8080
cfg.set("port", 9000)
cfg.set("debug", True)
cfg.save()

# Reload and verify
cfg2 = ConfigManager("app_config.json", defaults={"debug": False, "port": 8080})
print(cfg2.get("port"))   # 9000
print(cfg2.get("debug"))  # True

File I/O

🎯 Learning Objectives

Why File I/O?

Opening Files — open()

The with Statement

Reading Files

Writing Files

Appending to a file

Working with Paths — pathlib

CSV Files

JSON Files

Binary Files

Common Patterns

Atomic write — prevent partial writes on error

Read a config file with a default fallback

Process a large file line-by-line

Best Practices

💻 Exercises

Opening Files — `open()`

The `with` Statement

Working with Paths — `pathlib`