The Request Lifecycle
Every LLM API call follows the same basic flow, whether you're using OpenAI, Anthropic, or any other provider:
Key things happening at each stage:
- Build messages — Assemble system prompt + conversation history + user input
- API request — HTTP POST with auth headers, model selection, parameters
- Model inference — Provider runs your input through the model (100ms–30s)
- Stream response — Tokens arrive via Server-Sent Events as they're generated
- Process output — Parse, validate, store, display to user
Message Roles
Both OpenAI and Anthropic use a structured message format with roles that tell the model who's "speaking":
The Three Roles
- system — Sets the AI's persona, rules, and constraints. Processed once at the start. The model treats this as authoritative instructions.
- user — Human messages. Questions, commands, data to process.
- assistant — The AI's previous responses. Include these to maintain conversation history.
messages = [
{"role": "system", "content": "You are a helpful coding assistant. Be concise."},
{"role": "user", "content": "What's a list comprehension in Python?"},
{"role": "assistant", "content": "A list comprehension is a concise way to create lists..."},
{"role": "user", "content": "Show me an example with filtering"},
]
Anthropic separates the system prompt from messages. Instead of a system message in the array, you pass it as a top-level system parameter. Messages must strictly alternate between user and assistant roles.
OpenAI vs Anthropic: API Shapes
| Feature | OpenAI | Anthropic |
|---|---|---|
| Endpoint | /v1/chat/completions |
/v1/messages |
| System prompt | Message with role "system" | Top-level system param |
| Max output param | max_tokens (optional) |
max_tokens (required) |
| Streaming | stream: true, SSE chunks |
stream: true, SSE events |
| Stop reason | finish_reason |
stop_reason |
| Token counting | In response usage object |
In response usage object |
| Python SDK | openai |
anthropic |
| Auth header | Authorization: Bearer sk-... |
x-api-key: sk-ant-... |
Making Your First API Call
OpenAI
"""Basic OpenAI API call."""
import os
from openai import OpenAI
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain recursion in one sentence."},
],
temperature=0.7,
max_tokens=150,
)
print(response.choices[0].message.content)
print(f"Tokens used: {response.usage.total_tokens}")
Anthropic
"""Basic Anthropic API call."""
import os
import anthropic
client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=150,
system="You are a helpful assistant.",
messages=[
{"role": "user", "content": "Explain recursion in one sentence."},
],
temperature=0.7,
)
print(response.content[0].text)
print(f"Tokens used: {response.usage.input_tokens + response.usage.output_tokens}")
Streaming Responses
Streaming is essential for good UX. Without it, users stare at a blank screen for 2–10 seconds. With streaming, they see tokens appear in real-time.
"""Streaming with OpenAI."""
from openai import OpenAI
import os
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
stream = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Write a haiku about Python."}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta
if delta.content:
print(delta.content, end="", flush=True)
print() # newline at end
"""Streaming with Anthropic."""
import anthropic
import os
client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
with client.messages.stream(
model="claude-sonnet-4-20250514",
max_tokens=150,
messages=[{"role": "user", "content": "Write a haiku about Python."}],
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
print()
API keys should NEVER appear in source code, git history, or client-side code. Always use environment variables or a secrets manager:
- Local dev:
.envfile +python-dotenv(add.envto.gitignore) - Production: secrets manager (AWS Secrets Manager, GCP Secret Manager, Vault)
- CI/CD: encrypted environment variables
If a key leaks, rotate it immediately. OpenAI and Anthropic both allow instant key rotation in their dashboards.
Retry Logic & Rate Limits
APIs fail. Networks drop. Rate limits hit. Production code must handle these gracefully.
Common Error Codes
429— Rate limited. Back off and retry.500/502/503— Server error. Retry with exponential backoff.401— Invalid API key. Don't retry; fix the key.400— Bad request (too many tokens, invalid params). Don't retry; fix the request.
"""Robust API calls with retry logic."""
import time
import random
from openai import OpenAI, RateLimitError, APIError, APITimeoutError
import os
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
def call_with_retry(
messages: list,
model: str = "gpt-4o",
max_retries: int = 3,
base_delay: float = 1.0,
) -> str:
"""Call the API with exponential backoff retry."""
for attempt in range(max_retries + 1):
try:
response = client.chat.completions.create(
model=model,
messages=messages,
temperature=0.7,
)
return response.choices[0].message.content
except RateLimitError as e:
if attempt == max_retries:
raise
# Exponential backoff with jitter
delay = base_delay * (2 ** attempt) + random.uniform(0, 1)
print(f"Rate limited. Retrying in {delay:.1f}s... (attempt {attempt + 1})")
time.sleep(delay)
except APITimeoutError:
if attempt == max_retries:
raise
delay = base_delay * (2 ** attempt)
print(f"Timeout. Retrying in {delay:.1f}s...")
time.sleep(delay)
except APIError as e:
if e.status_code and e.status_code >= 500:
if attempt == max_retries:
raise
time.sleep(base_delay * (2 ** attempt))
else:
raise # Client errors (400, 401) — don't retry
Both the openai and anthropic Python SDKs have built-in retry logic. The OpenAI client retries 429/500+ errors twice by default. You can configure it: OpenAI(max_retries=5). For most cases, the built-in retry is sufficient.
Async & Concurrent Calls
When you need to make multiple independent API calls (e.g., evaluating 100 prompts), use async to parallelize:
"""Async concurrent API calls."""
import asyncio
import os
from openai import AsyncOpenAI
client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
async def classify(text: str) -> str:
"""Classify a single piece of text."""
response = await client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Classify as positive/negative/neutral. Reply with one word."},
{"role": "user", "content": text},
],
temperature=0,
max_tokens=10,
)
return response.choices[0].message.content.strip()
async def classify_batch(texts: list[str], max_concurrent: int = 10) -> list[str]:
"""Classify multiple texts with bounded concurrency."""
semaphore = asyncio.Semaphore(max_concurrent)
async def limited_classify(text):
async with semaphore:
return await classify(text)
tasks = [limited_classify(t) for t in texts]
return await asyncio.gather(*tasks)
# Usage
texts = ["Great product!", "Terrible service.", "It was okay."]
results = asyncio.run(classify_batch(texts))
print(list(zip(texts, results)))
Don't fire 1,000 concurrent requests — you'll hit rate limits instantly. Use a semaphore (as shown) to cap at 10–50 concurrent requests depending on your tier. OpenAI's rate limits are per-minute, so spread requests over time for large batches.
Project: CLI Chatbot with Streaming
🛠️ Build a CLI Chatbot
A full-featured terminal chatbot with streaming output, conversation history, system prompt configuration, and graceful error handling.
"""
CLI Chatbot with streaming and conversation history.
Usage: python chatbot.py
"""
import os
import sys
from openai import OpenAI, APIError
# ─── Configuration ───────────────────────────────────────────
MODEL = "gpt-4o"
SYSTEM_PROMPT = """You are a helpful AI assistant. You are concise but thorough.
When showing code, always include brief comments explaining key parts."""
MAX_HISTORY = 20 # Keep last N messages to manage context window
def create_client() -> OpenAI:
"""Create OpenAI client with API key from environment."""
api_key = os.environ.get("OPENAI_API_KEY")
if not api_key:
print("Error: OPENAI_API_KEY environment variable not set.")
print("Set it with: export OPENAI_API_KEY='sk-...'")
sys.exit(1)
return OpenAI(api_key=api_key)
def stream_response(client: OpenAI, messages: list) -> str:
"""Stream a response and return the full text."""
try:
stream = client.chat.completions.create(
model=MODEL,
messages=messages,
stream=True,
temperature=0.7,
)
full_response = []
print("\n\033[36mAssistant:\033[0m ", end="")
for chunk in stream:
delta = chunk.choices[0].delta
if delta.content:
print(delta.content, end="", flush=True)
full_response.append(delta.content)
print("\n")
return "".join(full_response)
except APIError as e:
print(f"\n\033[31mAPI Error: {e.message}\033[0m\n")
return ""
def main():
client = create_client()
history = []
print("=" * 50)
print(" CLI Chatbot (type 'quit' to exit, 'clear' to reset)")
print("=" * 50)
print(f" Model: {MODEL}")
print(f" System: {SYSTEM_PROMPT[:60]}...")
print("=" * 50 + "\n")
while True:
try:
user_input = input("\033[33mYou:\033[0m ").strip()
except (KeyboardInterrupt, EOFError):
print("\n\nGoodbye!")
break
if not user_input:
continue
if user_input.lower() == "quit":
print("Goodbye!")
break
if user_input.lower() == "clear":
history.clear()
print("History cleared.\n")
continue
if user_input.lower() == "history":
print(f"Messages in history: {len(history)}")
for msg in history:
role = msg["role"]
content = msg["content"][:80]
print(f" [{role}] {content}...")
print()
continue
# Add user message to history
history.append({"role": "user", "content": user_input})
# Trim history if too long
if len(history) > MAX_HISTORY:
history = history[-MAX_HISTORY:]
# Build full message list
messages = [{"role": "system", "content": SYSTEM_PROMPT}] + history
# Stream response
response_text = stream_response(client, messages)
if response_text:
history.append({"role": "assistant", "content": response_text})
if __name__ == "__main__":
main()
Running It
# Install dependency
pip install openai
# Set your API key
export OPENAI_API_KEY="sk-..."
# Run the chatbot
python chatbot.py
Enhancements to Try
- Add a
/modelcommand to switch models mid-conversation - Add token counting to show cost per message (use
tiktokenfrom Lesson 2) - Save/load conversation history to a JSON file
- Add a
/systemcommand to change the system prompt on the fly - Implement the same chatbot using the Anthropic SDK for comparison
Key Takeaways
- LLM APIs are stateless — you must send the full conversation history each time
- Message roles (system/user/assistant) structure the conversation for the model
- Always stream responses in user-facing applications for better UX
- Implement retry logic with exponential backoff for production reliability
- Never hardcode API keys — use environment variables or secrets managers
- Use async + semaphores for batch processing with bounded concurrency