Packaging & pyproject.toml

1 · Why Package?

In Python a module is a single .py file, a package is a directory containing an __init__.py (or, since Python 3.3, a namespace package without one), and a distribution package is the archive you upload to PyPI so others can pip install it.

Why bother turning your code into a proper distribution package?

• Reusability — install the same library in many projects without copy-pasting.
• Dependency pinning — declare exactly what you need; pip resolves and installs it.
• Reproducible installs — lockfiles and wheels guarantee identical environments.
• Sharing — publish to PyPI and anyone in the world can pip install yourpackage.

The tooling has evolved significantly over the years:

1. distutils (stdlib, now deprecated and removed in 3.12)
2. setuptools with an imperative setup.py
3. Declarative setup.cfg alongside a shim setup.py
4. pyproject.toml — the modern standard defined by PEP 517, PEP 518, and PEP 621

2 · Project Layout

The recommended layout uses a src/ directory to hold your importable package:

mypackage/                  ← repo root
├── src/
│   └── mypackage/
│       ├── __init__.py
│       ├── core.py
│       └── utils.py
├── tests/
│   ├── __init__.py
│   └── test_core.py
├── pyproject.toml
├── README.md
└── LICENSE

Why the src/ layout?

• Prevents accidentally importing your uninstalled source tree (Python adds . to sys.path, so without src/ you'd import from the repo root instead of the installed version).
• Forces you to install the package (pip install -e .) before testing — catching packaging bugs early.
• Produces cleaner wheel contents — only the code under src/ ships.

The tests/ directory lives outside src/ so it is not included in the built wheel by default.

A minimal __init__.py sets the public API and version:

"""mypackage — a demonstration package."""

__version__ = "0.1.0"

from mypackage.core import process_data
from mypackage.utils import slugify

__all__ = ["process_data", "slugify"]

3 · pyproject.toml Anatomy

A pyproject.toml is composed of three logical sections:

1. [build-system] — tells pip/build which backend to use.
2. [project] — standard metadata (PEP 621): name, version, dependencies, etc.
3. Tool-specific tables — e.g. [tool.hatch], [tool.pytest.ini_options], [tool.ruff].

Here is a complete, annotated example using Hatchling:

[build-system]
requires      = ["hatchling"]
build-backend = "hatchling.build"

[project]
name            = "mypackage"
version         = "0.1.0"
description     = "A short description of what mypackage does"
readme          = "README.md"
license         = { text = "MIT" }
requires-python = ">=3.10"
authors         = [{ name = "Alice Dev", email = "alice@example.com" }]
keywords        = ["example", "packaging"]
classifiers     = [
    "Development Status :: 3 - Alpha",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
]

dependencies = [
    "httpx>=0.27,<1",
    "pydantic>=2.0",
]

[project.optional-dependencies]
dev  = ["pytest>=8", "pytest-cov", "mypy", "ruff"]
docs = ["mkdocs", "mkdocs-material"]

[project.urls]
Homepage      = "https://github.com/alice/mypackage"
Documentation = "https://mypackage.readthedocs.io"
"Bug Tracker" = "https://github.com/alice/mypackage/issues"

[project.scripts]
mypackage-cli = "mypackage.cli:main"

Field-by-field breakdown:

• name — the distribution name on PyPI. Must be unique; normalised to lowercase with hyphens replaced by dashes (e.g. My_Package → my-package).
• version — follows SemVer. Can also be dynamic (read from source).
• description — one-line summary shown on PyPI.
• readme — path to the long description file (rendered as the PyPI project page).
• license — SPDX expression or { text = "..." }.
• requires-python — minimum Python version using PEP 440 specifiers.
• authors — list of { name, email } tables.
• classifiers — Trove classifiers for discovery on PyPI.
• dependencies — runtime requirements using PEP 440 version specifiers.
• optional-dependencies — extras installed via pip install mypackage[dev].
• urls — links shown on the PyPI sidebar.
• scripts — console entry points. "mypackage.cli:main" means "import main from mypackage.cli and call it".

4 · Version Specifiers & Dependency Pinning

PEP 440 defines how to express version constraints. Mastering these is critical for writing correct dependencies.

Tools for generating lockfiles from loose constraints:

• pip-compile (from pip-tools) — pip-compile pyproject.toml -o requirements.lock
• uv lock (from uv) — extremely fast Rust-based resolver

5 · Build Backends

PEP 517 introduced a clean separation between build frontends (like pip or python -m build) and build backends that actually produce the wheel. You pick a backend via [build-system]:

Example [build-system] blocks:

# Hatchling (recommended default)
[build-system]
requires      = ["hatchling"]
build-backend = "hatchling.build"

# Flit
[build-system]
requires      = ["flit_core>=3.2.0"]
build-backend = "flit_core.buildapi"

# Setuptools (modern declarative)
[build-system]
requires      = ["setuptools>=61", "wheel"]
build-backend = "setuptools.build_meta"

# Maturin (Rust)
[build-system]
requires      = ["maturin>=1.0"]
build-backend = "maturin"

6 · Building a Distribution

Install the standard build frontend:

pip install build

Then build your package:

# Build both sdist and wheel (recommended)
python -m build

# Build only wheel
python -m build --wheel

# Build only source distribution
python -m build --sdist

After a successful build, the dist/ directory contains:

• mypackage-0.1.0.tar.gz — the sdist (source distribution). Contains your source code plus pyproject.toml; anyone can build a wheel from it.
• mypackage-0.1.0-py3-none-any.whl — the wheel. Pre-built, installs in milliseconds with no build step required.

Wheel filename anatomy:

{name}-{version}-{python_tag}-{abi_tag}-{platform_tag}.whl

mypackage-0.1.0-py3-none-any.whl
│             │   │    │     └─ any platform
│             │   │    └─ no ABI dependency
│             │   └─ Python 3 (any minor)
│             └─ version
└─ distribution name

A pure-Python wheel uses py3-none-any. Packages with C extensions produce platform-specific wheels like cp312-cp312-manylinux_2_17_x86_64.whl.

7 · Publishing to PyPI

The full publish workflow:

# 1 — Install twine
pip install twine

# 2 — Check the distribution files for common errors
twine check dist/*

# 3 — Upload to TestPyPI first (always test before real PyPI)
twine upload --repository testpypi dist/*

# 4 — Install from TestPyPI to verify
pip install --index-url https://test.pypi.org/simple/ mypackage

# 5 — Publish to real PyPI
twine upload dist/*

Key points:

• Create accounts at pypi.org and test.pypi.org (they are separate).
• Use API tokens — not username/password. Store them in ~/.pypirc or as environment variables.
• Package names must be globally unique on PyPI.
• Once a version is uploaded it cannot be overwritten — always bump the version for fixes.

A minimal ~/.pypirc:

[distutils]
index-servers = pypi testpypi

[pypi]
repository = https://upload.pypi.org/legacy/
username   = __token__
password   = pypi-<YOUR_API_TOKEN>

[testpypi]
repository = https://test.pypi.org/legacy/
username   = __token__
password   = pypi-<YOUR_TESTPYPI_TOKEN>

8 · Automating Releases with CI

A minimal GitHub Actions workflow that publishes to PyPI whenever you push a version tag:

name: Publish to PyPI

on:
  push:
    tags: ["v*"]

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install build twine
      - run: python -m build
      - run: twine upload dist/*
        env:
          TWINE_USERNAME: __token__
          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}

Trusted Publishing (OIDC) — the modern, zero-secret alternative. Instead of storing a long-lived API token, you configure PyPI to trust your GitHub Actions workflow directly via OpenID Connect. No secrets to rotate, no tokens to leak. See the PyPI Trusted Publishers documentation.

Automating version bumps with bump-my-version:

# Install
pip install bump-my-version

# Bump patch: 0.1.0 → 0.1.1 (commits + tags automatically)
bump-my-version bump patch

# Bump minor: 0.1.1 → 0.2.0
bump-my-version bump minor

# Bump major: 0.2.0 → 1.0.0
bump-my-version bump major

# Push the tag to trigger the CI publish workflow
git push --follow-tags

Configure it in pyproject.toml:

[tool.bumpversion]
current_version = "0.1.0"
commit          = true
tag             = true

[[tool.bumpversion.files]]
filename = "pyproject.toml"
search   = 'version = "{current_version}"'
replace  = 'version = "{new_version}"'

[[tool.bumpversion.files]]
filename = "src/mypackage/__init__.py"
search   = '__version__ = "{current_version}"'
replace  = '__version__ = "{new_version}"'

With this setup, a single bump-my-version bump patch && git push --follow-tags updates the version everywhere, commits, tags, and pushes — triggering your CI to build and publish automatically.

Editable Installs & Development Workflow

During development you want changes to your source files to be reflected immediately without reinstalling. An editable install (also called a "develop install") links the package into your environment rather than copying it.

# Install your package in editable mode
pip install -e .

# Install with optional dev extras too
pip install -e ".[dev]"

# Verify it is listed as editable
pip show mypackage        # Location points to your src/ directory

With an editable install, editing src/mypackage/core.py is immediately visible to any script that imports mypackage — no reinstall required. Tests, scripts, and notebooks all see live changes.

Recommended Daily Workflow

1. Create and activate a virtual environment: python -m venv .venv && source .venv/bin/activate
2. Editable-install with dev extras: pip install -e ".[dev]"
3. Code, run tests with pytest, lint with ruff check .
4. Bump version: bump-my-version bump patch
5. Build: python -m build — inspect dist/
6. Publish: twine upload dist/* (or push a tag to trigger CI)

Best Practices

• Always use the src/ layout — it prevents your uninstalled package from being importable by accident and mirrors how it will behave once installed.
• Specify requires-python — prevents installation on unsupported Python versions and communicates your support matrix clearly.
• Pin loosely in library dependencies — use ranges like httpx>=0.27,<1 in pyproject.toml; reserve tight pins for application lockfiles (requirements.txt, uv.lock).
• Use API tokens, not passwords — create per-project tokens on PyPI with minimal scope; rotate them regularly.
• Test on TestPyPI first — once a version is uploaded to PyPI it cannot be deleted or overwritten. Always verify with TestPyPI.
• Automate with CI — tag-triggered GitHub Actions workflows remove human error from the release process.
• Include a LICENSE file — without one, legally no one can use your code. MIT or Apache-2.0 are standard open-source choices.
• Write a useful README.md — it becomes your PyPI landing page. Include install instructions, a quick-start example, and a link to full docs.

Exercises

[build-system]
requires      = ["hatchling"]
build-backend = "hatchling.build"

[project]
name            = "texttools"
version         = "0.1.0"
description     = "Handy text transformation utilities"
readme          = "README.md"
requires-python = ">=3.10"
dependencies    = []

[project.optional-dependencies]
dev = ["pytest>=8"]

import re
from collections import Counter

def slugify(text: str) -> str:
    text = text.lower().strip()
    text = re.sub(r"[^\w\s-]", "", text)
    return re.sub(r"[\s_]+", "-", text)

def truncate(text: str, max_len: int, suffix: str = "…") -> str:
    if len(text) <= max_len:
        return text
    return text[: max_len - len(suffix)] + suffix

def word_count(text: str) -> dict[str, int]:
    words = re.findall(r"\b\w+\b", text.lower())
    return dict(Counter(words))

import sys
from texttools.transforms import slugify, word_count

def main() -> None:
    if len(sys.argv) < 3:
        print("Usage: texttools-cli  ")
        sys.exit(1)
    cmd, text = sys.argv[1], sys.argv[2]
    if cmd == "slugify":
        print(slugify(text))
    elif cmd == "wordcount":
        for word, count in sorted(word_count(text).items()):
            print(f"{word}: {count}")
    else:
        print(f"Unknown command: {cmd}")
        sys.exit(1)

[project.scripts]
texttools-cli = "texttools.cli:main"

[project.optional-dependencies]
dev  = ["pytest>=8"]
http = ["httpx>=0.27,<1"]

# Build
python -m build

# Check metadata
twine check dist/*

# Upload to TestPyPI
TWINE_USERNAME=__token__ TWINE_PASSWORD=pypi-<token> \
  twine upload --repository-url https://test.pypi.org/legacy/ dist/*

# Fresh environment test
python -m venv /tmp/test-env
source /tmp/test-env/bin/activate
pip install --index-url https://test.pypi.org/simple/ texttools
texttools-cli slugify "Hello World"    # → hello-world