The Request Lifecycle

Every LLM API call follows the same basic flow, whether you're using OpenAI, Anthropic, or any other provider:

Request lifecycle: from your code to streaming tokens back

Key things happening at each stage:

  • Build messages — Assemble system prompt + conversation history + user input
  • API request — HTTP POST with auth headers, model selection, parameters
  • Model inference — Provider runs your input through the model (100ms–30s)
  • Stream response — Tokens arrive via Server-Sent Events as they're generated
  • Process output — Parse, validate, store, display to user

Message Roles

Both OpenAI and Anthropic use a structured message format with roles that tell the model who's "speaking":

The Three Roles

  • system — Sets the AI's persona, rules, and constraints. Processed once at the start. The model treats this as authoritative instructions.
  • user — Human messages. Questions, commands, data to process.
  • assistant — The AI's previous responses. Include these to maintain conversation history.
message structure
messages = [
    {"role": "system", "content": "You are a helpful coding assistant. Be concise."},
    {"role": "user", "content": "What's a list comprehension in Python?"},
    {"role": "assistant", "content": "A list comprehension is a concise way to create lists..."},
    {"role": "user", "content": "Show me an example with filtering"},
]
Anthropic's difference

Anthropic separates the system prompt from messages. Instead of a system message in the array, you pass it as a top-level system parameter. Messages must strictly alternate between user and assistant roles.

OpenAI vs Anthropic: API Shapes

Feature OpenAI Anthropic
Endpoint /v1/chat/completions /v1/messages
System prompt Message with role "system" Top-level system param
Max output param max_tokens (optional) max_tokens (required)
Streaming stream: true, SSE chunks stream: true, SSE events
Stop reason finish_reason stop_reason
Token counting In response usage object In response usage object
Python SDK openai anthropic
Auth header Authorization: Bearer sk-... x-api-key: sk-ant-...

Making Your First API Call

OpenAI

openai_hello.py
"""Basic OpenAI API call."""
import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain recursion in one sentence."},
    ],
    temperature=0.7,
    max_tokens=150,
)

print(response.choices[0].message.content)
print(f"Tokens used: {response.usage.total_tokens}")

Anthropic

anthropic_hello.py
"""Basic Anthropic API call."""
import os
import anthropic

client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=150,
    system="You are a helpful assistant.",
    messages=[
        {"role": "user", "content": "Explain recursion in one sentence."},
    ],
    temperature=0.7,
)

print(response.content[0].text)
print(f"Tokens used: {response.usage.input_tokens + response.usage.output_tokens}")

Streaming Responses

Streaming is essential for good UX. Without it, users stare at a blank screen for 2–10 seconds. With streaming, they see tokens appear in real-time.

streaming_openai.py
"""Streaming with OpenAI."""
from openai import OpenAI
import os

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Write a haiku about Python."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)

print()  # newline at end
streaming_anthropic.py
"""Streaming with Anthropic."""
import anthropic
import os

client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

with client.messages.stream(
    model="claude-sonnet-4-20250514",
    max_tokens=150,
    messages=[{"role": "user", "content": "Write a haiku about Python."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

print()
⚠️ Never hardcode API keys

API keys should NEVER appear in source code, git history, or client-side code. Always use environment variables or a secrets manager:

  • Local dev: .env file + python-dotenv (add .env to .gitignore)
  • Production: secrets manager (AWS Secrets Manager, GCP Secret Manager, Vault)
  • CI/CD: encrypted environment variables

If a key leaks, rotate it immediately. OpenAI and Anthropic both allow instant key rotation in their dashboards.

Retry Logic & Rate Limits

APIs fail. Networks drop. Rate limits hit. Production code must handle these gracefully.

Common Error Codes

  • 429 — Rate limited. Back off and retry.
  • 500/502/503 — Server error. Retry with exponential backoff.
  • 401 — Invalid API key. Don't retry; fix the key.
  • 400 — Bad request (too many tokens, invalid params). Don't retry; fix the request.
retry_logic.py
"""Robust API calls with retry logic."""
import time
import random
from openai import OpenAI, RateLimitError, APIError, APITimeoutError
import os

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])


def call_with_retry(
    messages: list,
    model: str = "gpt-4o",
    max_retries: int = 3,
    base_delay: float = 1.0,
) -> str:
    """Call the API with exponential backoff retry."""
    for attempt in range(max_retries + 1):
        try:
            response = client.chat.completions.create(
                model=model,
                messages=messages,
                temperature=0.7,
            )
            return response.choices[0].message.content

        except RateLimitError as e:
            if attempt == max_retries:
                raise
            # Exponential backoff with jitter
            delay = base_delay * (2 ** attempt) + random.uniform(0, 1)
            print(f"Rate limited. Retrying in {delay:.1f}s... (attempt {attempt + 1})")
            time.sleep(delay)

        except APITimeoutError:
            if attempt == max_retries:
                raise
            delay = base_delay * (2 ** attempt)
            print(f"Timeout. Retrying in {delay:.1f}s...")
            time.sleep(delay)

        except APIError as e:
            if e.status_code and e.status_code >= 500:
                if attempt == max_retries:
                    raise
                time.sleep(base_delay * (2 ** attempt))
            else:
                raise  # Client errors (400, 401) — don't retry
SDK built-in retries

Both the openai and anthropic Python SDKs have built-in retry logic. The OpenAI client retries 429/500+ errors twice by default. You can configure it: OpenAI(max_retries=5). For most cases, the built-in retry is sufficient.

Async & Concurrent Calls

When you need to make multiple independent API calls (e.g., evaluating 100 prompts), use async to parallelize:

async_calls.py
"""Async concurrent API calls."""
import asyncio
import os
from openai import AsyncOpenAI

client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])


async def classify(text: str) -> str:
    """Classify a single piece of text."""
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Classify as positive/negative/neutral. Reply with one word."},
            {"role": "user", "content": text},
        ],
        temperature=0,
        max_tokens=10,
    )
    return response.choices[0].message.content.strip()


async def classify_batch(texts: list[str], max_concurrent: int = 10) -> list[str]:
    """Classify multiple texts with bounded concurrency."""
    semaphore = asyncio.Semaphore(max_concurrent)

    async def limited_classify(text):
        async with semaphore:
            return await classify(text)

    tasks = [limited_classify(t) for t in texts]
    return await asyncio.gather(*tasks)


# Usage
texts = ["Great product!", "Terrible service.", "It was okay."]
results = asyncio.run(classify_batch(texts))
print(list(zip(texts, results)))
Concurrency limits

Don't fire 1,000 concurrent requests — you'll hit rate limits instantly. Use a semaphore (as shown) to cap at 10–50 concurrent requests depending on your tier. OpenAI's rate limits are per-minute, so spread requests over time for large batches.

Project: CLI Chatbot with Streaming

🛠️ Build a CLI Chatbot

A full-featured terminal chatbot with streaming output, conversation history, system prompt configuration, and graceful error handling.

chatbot.py
"""
CLI Chatbot with streaming and conversation history.
Usage: python chatbot.py
"""
import os
import sys
from openai import OpenAI, APIError

# ─── Configuration ───────────────────────────────────────────
MODEL = "gpt-4o"
SYSTEM_PROMPT = """You are a helpful AI assistant. You are concise but thorough.
When showing code, always include brief comments explaining key parts."""

MAX_HISTORY = 20  # Keep last N messages to manage context window


def create_client() -> OpenAI:
    """Create OpenAI client with API key from environment."""
    api_key = os.environ.get("OPENAI_API_KEY")
    if not api_key:
        print("Error: OPENAI_API_KEY environment variable not set.")
        print("Set it with: export OPENAI_API_KEY='sk-...'")
        sys.exit(1)
    return OpenAI(api_key=api_key)


def stream_response(client: OpenAI, messages: list) -> str:
    """Stream a response and return the full text."""
    try:
        stream = client.chat.completions.create(
            model=MODEL,
            messages=messages,
            stream=True,
            temperature=0.7,
        )

        full_response = []
        print("\n\033[36mAssistant:\033[0m ", end="")

        for chunk in stream:
            delta = chunk.choices[0].delta
            if delta.content:
                print(delta.content, end="", flush=True)
                full_response.append(delta.content)

        print("\n")
        return "".join(full_response)

    except APIError as e:
        print(f"\n\033[31mAPI Error: {e.message}\033[0m\n")
        return ""


def main():
    client = create_client()
    history = []

    print("=" * 50)
    print("  CLI Chatbot (type 'quit' to exit, 'clear' to reset)")
    print("=" * 50)
    print(f"  Model: {MODEL}")
    print(f"  System: {SYSTEM_PROMPT[:60]}...")
    print("=" * 50 + "\n")

    while True:
        try:
            user_input = input("\033[33mYou:\033[0m ").strip()
        except (KeyboardInterrupt, EOFError):
            print("\n\nGoodbye!")
            break

        if not user_input:
            continue
        if user_input.lower() == "quit":
            print("Goodbye!")
            break
        if user_input.lower() == "clear":
            history.clear()
            print("History cleared.\n")
            continue
        if user_input.lower() == "history":
            print(f"Messages in history: {len(history)}")
            for msg in history:
                role = msg["role"]
                content = msg["content"][:80]
                print(f"  [{role}] {content}...")
            print()
            continue

        # Add user message to history
        history.append({"role": "user", "content": user_input})

        # Trim history if too long
        if len(history) > MAX_HISTORY:
            history = history[-MAX_HISTORY:]

        # Build full message list
        messages = [{"role": "system", "content": SYSTEM_PROMPT}] + history

        # Stream response
        response_text = stream_response(client, messages)

        if response_text:
            history.append({"role": "assistant", "content": response_text})


if __name__ == "__main__":
    main()

Running It

terminal
# Install dependency
pip install openai

# Set your API key
export OPENAI_API_KEY="sk-..."

# Run the chatbot
python chatbot.py

Enhancements to Try

  1. Add a /model command to switch models mid-conversation
  2. Add token counting to show cost per message (use tiktoken from Lesson 2)
  3. Save/load conversation history to a JSON file
  4. Add a /system command to change the system prompt on the fly
  5. Implement the same chatbot using the Anthropic SDK for comparison

Key Takeaways

  • LLM APIs are stateless — you must send the full conversation history each time
  • Message roles (system/user/assistant) structure the conversation for the model
  • Always stream responses in user-facing applications for better UX
  • Implement retry logic with exponential backoff for production reliability
  • Never hardcode API keys — use environment variables or secrets managers
  • Use async + semaphores for batch processing with bounded concurrency