Why Engineers Need This

You don't need to implement a transformer from scratch. But understanding how LLMs work lets you:

  • Predict when they'll fail (and why)
  • Estimate costs accurately (tokens = money)
  • Write better prompts (you understand what the model "sees")
  • Debug unexpected outputs (temperature, context truncation)

Tokenization: How Text Becomes Numbers

LLMs don't read characters or words — they read tokens. A token is a chunk of text, typically 3–4 characters. The model's vocabulary is a fixed set of tokens learned during training using Byte-Pair Encoding (BPE).

Tokenization Example

The word unhappiness gets broken into tokens:

un happ iness

Each token maps to an integer ID in the model's vocabulary. GPT-4 has ~100,000 tokens in its vocabulary.

BPE in 30 Seconds

Byte-Pair Encoding starts with individual bytes, then repeatedly merges the most common adjacent pairs into new tokens. This creates a vocabulary that efficiently represents common words as single tokens while still handling rare words by splitting them into sub-tokens.

Result: common words like "the" = 1 token. Rare words like "defenestration" = 3-4 tokens.

Why this matters for cost

API pricing is per-token. A 1,000-word English paragraph is roughly 1,300 tokens. Code is typically more token-dense than English prose. Always estimate token counts before sending large payloads.

The Transformer (Bird's-Eye View)

The transformer architecture is the engine behind every modern LLM. You don't need to implement it, but understanding the flow helps you reason about capabilities and limitations.

The token prediction loop — input tokens flow through the transformer to predict the next one

Key Components

  1. Token Embedding — Each token ID is converted to a high-dimensional vector (e.g., 4096 dimensions in GPT-4)
  2. Positional Encoding — Position information is added so the model knows word order
  3. Self-Attention Layers — The model decides which other tokens to "pay attention to" for each position. This is where reasoning happens.
  4. Feed-Forward Layers — Dense neural network layers that transform representations
  5. Output Projection — Final layer produces a probability distribution over all possible next tokens

Attention: The Key Insight

Self-attention lets every token "look at" every other token in the context. When the model processes "The cat sat on the ___", attention lets it connect "sat" with likely completions like "mat" or "floor".

This is also why context length matters — attention has O(n²) computational cost with sequence length, making long contexts expensive.

Next-Token Prediction

The fundamental operation of an LLM is breathtakingly simple: predict the next token. That's it. Everything else — conversations, code generation, reasoning — emerges from this single objective.

The autoregressive generation loop — each predicted token becomes input for the next step

Autoregressive Generation

To generate a full response, the model:

  1. Takes the full prompt as input tokens
  2. Predicts one token (the most likely next token)
  3. Appends that token to the input
  4. Repeats until it produces a stop token or hits the max length

This is why generation is slow — each token requires a full forward pass through the network. A 500-token response needs 500 sequential passes.

Context Windows

The context window is the maximum number of tokens a model can process in a single call. It includes both your input (prompt) AND the model's output.

Model Context Window Approximate Pages
GPT-3.5 16K tokens ~25 pages
GPT-4o 128K tokens ~200 pages
Claude 3.5 Sonnet 200K tokens ~300 pages
Gemini 1.5 Pro 1M tokens ~1,500 pages
Context ≠ Memory

Models don't "remember" previous conversations unless you explicitly include them in the context. Each API call is stateless. Conversation history must be sent every time, consuming tokens (and money) with each turn.

Temperature & Sampling Parameters

After the model computes probabilities for each possible next token, sampling parameters control how the final token is chosen.

Temperature

Temperature scales the logits (raw scores) before converting to probabilities. It controls randomness:

  • Temperature = 0 — Always pick the highest-probability token. Deterministic, repetitive.
  • Temperature = 0.7 — Moderate randomness. Good default for most tasks.
  • Temperature = 1.5 — High randomness. Creative but potentially incoherent.
Effect of temperature on output distribution — lower = more peaked, higher = more uniform

Top-p (Nucleus Sampling)

Top-p limits sampling to the smallest set of tokens whose cumulative probability exceeds p. With top_p=0.9, the model only considers tokens covering 90% of probability mass — cutting the long tail of unlikely tokens.

Top-k

Top-k simply limits sampling to the k most probable tokens. top_k=50 means only the top 50 tokens are considered, regardless of their probability mass.

Practical defaults

For factual/coding tasks: temperature=0, top_p=1. For creative writing: temperature=0.8, top_p=0.95. Generally, adjust either temperature OR top_p, not both — they interact in non-obvious ways.

Mini-Project: Token Counter & Cost Estimator

🛠️ Build a Token Counter

Build a CLI tool that counts tokens in text using tiktoken (OpenAI's tokenizer) and estimates API costs across different models.

Setup

terminal
pip install tiktoken

The Code

token_counter.py
"""
Token Counter & Cost Estimator
Counts tokens and estimates API costs for different models.
"""
import tiktoken
import sys

# Pricing per 1M tokens (input) as of 2024
MODEL_PRICING = {
    "gpt-4o": {"input": 2.50, "output": 10.00, "encoding": "o200k_base"},
    "gpt-4o-mini": {"input": 0.15, "output": 0.60, "encoding": "o200k_base"},
    "gpt-4-turbo": {"input": 10.00, "output": 30.00, "encoding": "cl100k_base"},
    "gpt-3.5-turbo": {"input": 0.50, "output": 1.50, "encoding": "cl100k_base"},
}


def count_tokens(text: str, model: str = "gpt-4o") -> int:
    """Count the number of tokens in a text string."""
    encoding_name = MODEL_PRICING[model]["encoding"]
    enc = tiktoken.get_encoding(encoding_name)
    tokens = enc.encode(text)
    return len(tokens)


def show_tokenization(text: str, model: str = "gpt-4o") -> list[str]:
    """Show how text is split into individual tokens."""
    encoding_name = MODEL_PRICING[model]["encoding"]
    enc = tiktoken.get_encoding(encoding_name)
    tokens = enc.encode(text)
    return [enc.decode([t]) for t in tokens]


def estimate_cost(
    input_tokens: int,
    output_tokens: int = 0,
    model: str = "gpt-4o"
) -> dict:
    """Estimate cost in USD for a given token count."""
    pricing = MODEL_PRICING[model]
    input_cost = (input_tokens / 1_000_000) * pricing["input"]
    output_cost = (output_tokens / 1_000_000) * pricing["output"]
    return {
        "model": model,
        "input_tokens": input_tokens,
        "output_tokens": output_tokens,
        "input_cost": f"${input_cost:.6f}",
        "output_cost": f"${output_cost:.6f}",
        "total_cost": f"${input_cost + output_cost:.6f}",
    }


def main():
    if len(sys.argv) > 1:
        text = " ".join(sys.argv[1:])
    else:
        print("Enter text (Ctrl+D to finish):")
        text = sys.stdin.read()

    print(f"\n{'='*50}")
    print(f"Input text: {text[:100]}{'...' if len(text) > 100 else ''}")
    print(f"Character count: {len(text)}")
    print(f"{'='*50}\n")

    # Show tokenization for first model
    tokens = show_tokenization(text)
    print(f"Tokens: {tokens[:20]}{'...' if len(tokens) > 20 else ''}\n")

    # Compare across models
    print(f"{'Model':<16} {'Tokens':<10} {'Input Cost':<14} {'Cost per 1K calls'}")
    print("-" * 60)

    for model in MODEL_PRICING:
        n_tokens = count_tokens(text, model)
        cost = estimate_cost(n_tokens, n_tokens, model)  # assume output ≈ input
        per_1k = (n_tokens / 1_000_000) * (
            MODEL_PRICING[model]["input"] + MODEL_PRICING[model]["output"]
        ) * 1000
        print(f"{model:<16} {n_tokens:<10} {cost['input_cost']:<14} ${per_1k:.4f}")


if __name__ == "__main__":
    main()

Example Output

terminal output
$ python token_counter.py "The quick brown fox jumps over the lazy dog"

==================================================
Input text: The quick brown fox jumps over the lazy dog
Character count: 43
==================================================

Tokens: ['The', ' quick', ' brown', ' fox', ' jumps', ' over', ' the', ' lazy', ' dog']

Model            Tokens     Input Cost     Cost per 1K calls
------------------------------------------------------------
gpt-4o           9          $0.000023      $0.1125
gpt-4o-mini      9          $0.000001      $0.0068
gpt-4-turbo      9          $0.000090      $0.3600
gpt-3.5-turbo    9          $0.000005      $0.0180

Exercises

  1. Try tokenizing code vs prose — which is more token-dense?
  2. Tokenize text in different languages. How does Japanese compare to English in token count?
  3. Add a --file flag to read from a file and estimate the cost of using it as context
  4. Calculate: if you send 200K tokens of context with each API call, what's the hourly cost at 10 requests/minute?

Key Takeaways

  • LLMs process tokens, not words — tokenization determines cost and context usage
  • The transformer uses self-attention to let each token "see" all others in the context
  • Generation is autoregressive: one token at a time, each dependent on all previous
  • Context window = input + output; it's shared and finite
  • Temperature controls randomness; top-p/top-k trim the probability distribution
  • Always estimate token counts before designing your system — they directly determine cost