🎯 What You'll Learn
- Articulate what a managed ML platform adds on top of a self-managed FastAPI + Docker deployment, and when that trade is worth it
- Launch a managed training job with the SageMaker Python SDK's
Estimator/PyTorchclasses, including spot instances and distributed training - Decide between bringing your own Docker container ("BYOC") and using a built-in framework container
- Register, version, and approve models with the SageMaker Model Registry
- Choose between real-time endpoints, batch transform jobs, and serverless inference based on traffic shape
- Deploy an auto-scaling real-time endpoint with
Model.deploy()and configure scaling policies - Orchestrate a full train → evaluate → register → deploy workflow as a DAG with SageMaker Pipelines
- Reason about instance selection (CPU vs GPU, spot vs on-demand) and avoid the most common cost pitfalls
In Lesson 66 you packaged a model into a Docker container, wrote a FastAPI app around it, and deployed it yourself — you own the EC2 instance (or Kubernetes cluster), the load balancer, the autoscaling rules, the health checks, and the 3am pager when a node runs out of disk. That's completely viable, and plenty of production systems run exactly that way forever. SageMaker (and platforms like it — Vertex AI, Azure ML) sells you the same end state — a containerized model behind an endpoint — but takes over the undifferentiated heavy lifting: provisioning training compute, orchestrating distributed jobs, scaling endpoints up and down, versioning models, and wiring up monitoring. You trade direct control and a flat EC2 bill for a managed control plane, a different (often higher, but more predictable) cost structure, and AWS-specific APIs you can't easily port elsewhere. The skill this lesson teaches isn't "SageMaker is better" — it's recognizing which problem you actually have, because the two approaches solve different ones.
1 Why Move Off Self-Managed Docker?
The FastAPI + Docker stack from Lesson 66 is minimal and transparent: you control every layer, there's no vendor abstraction between you and the kernel, and you pay only for the EC2 instances (or equivalent) you provision. It starts to strain in a few recognizable ways as a team or workload grows:
- Training infrastructure churn. A single GPU box is fine for one experiment. A team running 30 training jobs a week, some needing 8x A100s for a few hours and most needing nothing, ends up either over-provisioning idle GPU boxes or hand-rolling a job queue and autoscaler — which is its own infrastructure project.
- Endpoint scaling under variable load. Hand-rolling autoscaling for a FastAPI container means wiring CloudWatch alarms, an Application Load Balancer, target tracking policies, and health checks yourself — solvable, but it's now infrastructure code you own and must maintain.
- Model lineage and governance. "Which container image, which weights file, and which training data produced the model currently serving production traffic?" is easy to lose track of with ad hoc S3 paths and Docker tags, especially across a team.
- Operational burden distributed across many small projects. A single ML team might own a dozen models. Re-deriving health checks, logging, scaling policies, and rollback procedures a dozen times is real, recurring cost — even if each instance is "simple."
SageMaker addresses each of these with a managed service: Training Jobs for ephemeral, auto-provisioned training compute; Model Registry for lineage and versioning; Endpoints with built-in auto-scaling and monitoring; and Pipelines to wire the stages together. None of this is magic — under the hood it's still EC2 instances running containers — but AWS operates the orchestration layer so your team doesn't have to.
What You Give Up
The honest tradeoffs, not just the marketing version:
- Vendor lock-in. Code written against the
sagemakerSDK, Pipelines DAGs, and SageMaker-specific container conventions (e.g./opt/ml/model,SM_MODEL_DIRenvironment variables) does not move to GCP or on-prem without rework. - Cost opacity and floors. Real-time endpoints bill per-hour for the underlying instance whether or not it's serving traffic, on top of SageMaker's per-hour markup over raw EC2. A forgotten endpoint is a forgotten EC2 bill that doesn't stop until someone deletes it.
- Slower local iteration. Debugging a training job that fails after 20 minutes of provisioning + data download is slower than debugging the same script run locally in your container.
- Abstraction leakage. When something goes wrong inside the managed layer (a container that won't start, a networking misconfiguration in VPC mode), you're debugging AWS's orchestration as much as your own code.
If you have one or two models, predictable traffic, and a team comfortable owning infrastructure, the Lesson 66 approach is often cheaper and simpler — don't adopt SageMaker just because it's "the enterprise way." Reach for SageMaker when you have many models or experiments competing for compute, spiky/unpredictable training needs that benefit from spot instances and elastic provisioning, a compliance requirement for auditable model lineage, or an ML platform team whose job is specifically to amortize this orchestration cost across many product teams.
2 SageMaker Studio and the Development Loop
SageMaker Studio is a managed JupyterLab-based IDE: notebooks run on managed compute instances (selectable per-notebook, including GPU instances), with direct access to S3, IAM-scoped permissions, and the rest of the SageMaker APIs already configured. The practical benefit over a local notebook is that the heavy compute (loading a large dataset, doing exploratory training on a GPU) happens on a rented instance you can resize or shut down independently of your laptop, and the notebook environment is reproducible across a team.
For day-to-day development, most teams don't write training logic inside a Studio notebook — they use the notebook as a control surface that submits jobs to the managed training and endpoint APIs, and write the actual model code as a standalone script that can run identically inside a SageMaker training container or on a laptop. That separation matters: it keeps your model code portable and testable outside of AWS, with only a thin "launch" layer depending on the sagemaker SDK.
import sagemaker
import boto3
# A SageMaker "session" wraps boto3 clients with SageMaker-aware helpers
# (S3 upload conventions, default bucket naming, region resolution, etc.)
session = sagemaker.Session()
# The execution role: an IAM role SageMaker assumes to read training data
# from S3, pull container images from ECR, and write model artifacts back to S3.
# Inside Studio this is auto-detected; outside Studio you pass an explicit ARN.
role = sagemaker.get_execution_role() # or: "arn:aws:iam::123456789012:role/SageMakerExecutionRole"
bucket = session.default_bucket()
region = session.boto_region_name
print(f"SageMaker session ready")
print(f" Execution role: {role}")
print(f" Default bucket: {bucket}")
print(f" Region: {region}")Everything that follows — launching a training job, registering a model, deploying an endpoint — is driven through this session and role, whether you call it from Studio, a local terminal with AWS credentials configured, or a CI/CD pipeline.
3 Managed Training: The Estimator API
The core abstraction for training is the Estimator: you describe what to run (a script, a container, hyperparameters), what compute to run it on (instance type, count, spot vs on-demand), and where the data and output artifacts live (S3 paths). Calling .fit() provisions ephemeral compute, runs your training script inside a container, uploads the resulting model artifact to S3, and tears the compute back down — you never SSH into a long-lived training box.
Built-In Framework Containers vs Bring Your Own Container (BYOC)
SageMaker maintains pre-built, AWS-optimized Docker images for the major frameworks (PyTorch, TensorFlow, Hugging Face, XGBoost, scikit-learn). Using sagemaker.pytorch.PyTorch means you supply only your training script and a requirements.txt — AWS handles the CUDA drivers, framework installation, and distributed-training plumbing. This is the path of least resistance and what most teams should default to.
Bringing your own container (the same Dockerfile pattern from Lesson 66, with a few SageMaker conventions added — reading hyperparameters from SM_HP_* environment variables, writing the model to /opt/ml/model, accepting an SM_CHANNEL_TRAIN data path) is the right call when your training stack has unusual system dependencies the built-in containers don't support (a custom compiled library, a non-standard CUDA version, a non-Python toolchain step) — essentially the same "do I need a custom container" judgment call you made in Lesson 66, just applied to training instead of serving.
from sagemaker.pytorch import PyTorch
# Using a built-in PyTorch framework container — no Dockerfile needed.
estimator = PyTorch(
entry_point="train.py", # your training script (runs inside the container)
source_dir="src", # local dir containing train.py + requirements.txt
role=role,
framework_version="2.3",
py_version="py311",
instance_type="ml.g5.xlarge", # 1x NVIDIA A10G GPU
instance_count=1,
hyperparameters={
"epochs": 10,
"batch-size": 64,
"learning-rate": 3e-4,
},
output_path=f"s3://{bucket}/churn-model/output",
base_job_name="churn-classifier",
# Spot instances: up to ~70% cheaper than on-demand, at the cost of
# possible interruption. SageMaker automatically resumes from the
# last checkpoint if the spot instance is reclaimed.
use_spot_instances=True,
max_run=3600, # hard ceiling: 1 hour of compute time
max_wait=7200, # allow up to 2 hours of *wall clock* time waiting for spot capacity
checkpoint_s3_uri=f"s3://{bucket}/churn-model/checkpoints",
)
# Data channels: named S3 locations SageMaker mounts into the container
# as local directories (e.g. /opt/ml/input/data/train).
estimator.fit({
"train": f"s3://{bucket}/churn-data/train/",
"validation": f"s3://{bucket}/churn-data/validation/",
})Distributed Training
For models too large or datasets too slow to train on one machine, the same Estimator scales out by increasing instance_count and specifying a distribution strategy — SageMaker handles inter-node networking (placement groups, EFA where available) and launches the appropriate distributed backend on each node.
from sagemaker.pytorch import PyTorch
distributed_estimator = PyTorch(
entry_point="train_ddp.py",
source_dir="src",
role=role,
framework_version="2.3",
py_version="py311",
instance_type="ml.p4d.24xlarge", # 8x A100 per node
instance_count=4, # 4 nodes = 32 GPUs total
hyperparameters={"epochs": 5, "batch-size": 32},
output_path=f"s3://{bucket}/large-model/output",
distribution={"torch_distributed": {"enabled": True}}, # PyTorch DDP across nodes
)
distributed_estimator.fit({"train": f"s3://{bucket}/large-dataset/train/"})It's easy to reflexively reach for ml.p4d.24xlarge ($30+/hour on-demand) for a model that trains fine on a single ml.g5.xlarge ($1.4/hour). Multi-GPU and multi-node instances only help if your training loop and data pipeline can actually saturate them — otherwise you're paying for idle GPUs while your dataloader is the bottleneck. Start with the smallest instance that fits the model and batch size in memory, profile GPU utilization (SageMaker emits this to CloudWatch automatically), and scale up only when utilization is consistently near 100% and wall-clock time is the binding constraint. Always set max_run as a safety ceiling — a training job stuck in an infinite loop with no time limit will happily bill you for days.
4 SageMaker Model Registry
Once estimator.fit() finishes, the resulting model artifact (weights + any preprocessing objects) lives at an S3 path under estimator.model_data. On its own that's just a file — the Model Registry turns it into a versioned, governed entity: a Model Package Group represents a logical model (e.g. "churn-classifier"), and each training run that produces a candidate is registered as a numbered Model Package version within that group, carrying metadata (metrics, the training job that produced it, an approval status).
This plays the same role MLflow's Model Registry played in Lesson 65 — the difference is that SageMaker's registry is wired directly into the deployment APIs: a model package's approval status can directly gate whether a downstream Pipeline step is allowed to deploy it.
import boto3
sm_client = boto3.client("sagemaker")
# Create (once) a Model Package Group — the logical "model" the versions belong to
sm_client.create_model_package_group(
ModelPackageGroupName="churn-classifier",
ModelPackageGroupDescription="Customer churn classifier, retrained weekly",
)
# Register this training run's artifact as a new version in that group
model_package = estimator.register(
content_types=["application/json"],
response_types=["application/json"],
inference_instances=["ml.m5.large", "ml.m5.xlarge"],
transform_instances=["ml.m5.large"],
model_package_group_name="churn-classifier",
approval_status="PendingManualApproval", # require a human (or automated gate) to approve
model_metrics=None, # in practice: attach a sagemaker.model_metrics.ModelMetrics object
)
print(f"Registered: {model_package.model_package_arn}")
# Later — after a reviewer checks validation metrics — approve it for deployment
sm_client.update_model_package(
ModelPackageArn=model_package.model_package_arn,
ModelApprovalStatus="Approved",
)The real value of the registry shows up in automation: a CI/CD pipeline (or a SageMaker Pipeline, covered in Section 7) can be configured to only deploy model packages with ModelApprovalStatus = Approved. That turns "did someone review this model before it went live" from a process you hope people follow into a condition the deployment code enforces — directly analogous to requiring a passing test suite before a merge.
Quick Check
5 Three Ways to Serve Predictions
This is the section that maps most directly onto Lesson 66. There, you built one thing: a FastAPI process behind a REST endpoint, always running, serving requests synchronously. SageMaker splits "serving predictions" into three distinct products because real workloads don't all have the same traffic shape — using the wrong one is a common source of either overspending or excess latency.
The three SageMaker serving options trade latency, cost model, and operational shape against each other rather than one strictly dominating: a real-time endpoint pays for always-on capacity to get the lowest latency, batch transform trades synchronous access for near-zero idle cost on large offline jobs, and serverless inference scales to zero between calls at the cost of cold-start latency on the first request after idle.
5.1 Real-Time Inference Endpoints
The closest analog to Lesson 66's FastAPI deployment: a persistent HTTPS endpoint backed by one or more always-on instances, for synchronous, low-latency, unpredictable-arrival-time requests. You get this from Model.deploy(), and — unlike your hand-rolled FastAPI container — auto-scaling, multi-model hosting, and request/response logging for monitoring are configuration, not code you write.
from sagemaker.model import Model
from sagemaker.predictor import Predictor
from sagemaker.serializers import JSONSerializer
from sagemaker.deserializers import JSONDeserializer
model = Model(
model_data=estimator.model_data, # S3 path to the trained artifact
image_uri=estimator.training_image_uri(), # matching inference container
role=role,
sagemaker_session=session,
)
predictor = model.deploy(
initial_instance_count=2,
instance_type="ml.m5.large",
endpoint_name="churn-classifier-rt",
serializer=JSONSerializer(),
deserializer=JSONDeserializer(),
)
# Synchronous prediction — just like calling your FastAPI endpoint in Lesson 66
result = predictor.predict({"features": [0.42, 1.0, 0.0, 5.3, 12]})
print(result)
# Configure auto-scaling: target 70% average CPU utilization,
# scale out within minutes of sustained load, scale back in during quiet periods.
import boto3
autoscaling = boto3.client("application-autoscaling")
resource_id = f"endpoint/{predictor.endpoint_name}/variant/AllTraffic"
autoscaling.register_scalable_target(
ServiceNamespace="sagemaker",
ResourceId=resource_id,
ScalableDimension="sagemaker:variant:DesiredInstanceCount",
MinCapacity=2,
MaxCapacity=10,
)
autoscaling.put_scaling_policy(
PolicyName="churn-endpoint-cpu-target",
ServiceNamespace="sagemaker",
ResourceId=resource_id,
ScalableDimension="sagemaker:variant:DesiredInstanceCount",
PolicyType="TargetTrackingScaling",
TargetTrackingScalingPolicyConfiguration={
"TargetValue": 70.0,
"PredefinedMetricSpecification": {
"PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"
},
"ScaleInCooldown": 300,
"ScaleOutCooldown": 60,
},
)5.2 Batch Transform
When you need predictions for a large, known set of records all at once — no live endpoint, no per-request latency requirement — spinning up a persistent endpoint is wasted spend. Batch Transform provisions compute, reads input from S3, runs inference over the whole dataset, writes predictions back to S3, and tears the compute down. This is the natural fit for nightly or weekly scoring jobs.
transformer = model.transformer(
instance_count=4,
instance_type="ml.m5.xlarge",
output_path=f"s3://{bucket}/churn-scores/{run_date}/",
strategy="MultiRecord", # batch multiple records per inference call
max_payload=6, # MB per mini-batch sent to the container
)
transformer.transform(
data=f"s3://{bucket}/churn-data/full-customer-base/{run_date}/",
content_type="text/csv",
split_type="Line",
)
transformer.wait()
print(f"Batch scoring complete: {transformer.output_path}")5.3 Serverless Inference
For endpoints with spiky, intermittent, or hard-to-predict traffic — a few requests an hour, then a burst — paying for always-on instances (real-time) is wasteful, but you still want a managed HTTPS endpoint rather than building a batch job. Serverless Inference scales to zero between requests and charges per invocation plus compute duration, at the cost of cold-start latency on the first request after idle.
from sagemaker.serverless import ServerlessInferenceConfig
serverless_config = ServerlessInferenceConfig(
memory_size_in_mb=4096, # 1024–6144 MB in 1024 MB increments
max_concurrency=10, # max simultaneous invocations before throttling
)
serverless_predictor = model.deploy(
serverless_inference_config=serverless_config,
endpoint_name="recsys-serverless",
serializer=JSONSerializer(),
deserializer=JSONDeserializer(),
)
result = serverless_predictor.predict({"user_id": 48213, "context": "homepage"})
print(result)| Option | Traffic shape | Cost model | Latency |
|---|---|---|---|
| Real-time endpoint | Steady, latency-sensitive | Per-hour, always-on | Lowest, no cold start |
| Batch transform | Large known dataset, offline | Per-job compute time only | N/A (not synchronous) |
| Serverless inference | Spiky / intermittent | Per-invocation + duration | Cold-start penalty after idle |
6 Instance Selection and Cost Control
Every job and endpoint in SageMaker runs on an explicit EC2 instance type, and choosing well is most of what determines your bill. A few practical guidelines:
- CPU instances (the
ml.m5,ml.c5families) are the default for classical ML (XGBoost, scikit-learn, small tabular neural nets) and for inference on small-to-medium deep learning models where latency tolerance is moderate. They are dramatically cheaper per hour than GPU instances. - GPU instances (
ml.g5for cost-effective inference and light training,ml.p4d/ml.p5for serious large-model training) are worth the premium when a CPU instance would make training intractably slow or push real-time inference latency past your SLA — not by default. - Spot instances for training can cut costs 60–90%, with the caveat that the instance can be reclaimed with a two-minute warning. Always pair
use_spot_instances=Truewith checkpointing (as in Section 3) so an interruption loses minutes, not hours, of progress. - Inference Recommender (a SageMaker tool, not covered in depth here) will load-test your model artifact across candidate instance types and report cost/latency tradeoffs automatically — worth running before committing to an instance type for a production endpoint.
Illustrative cost / latency / throughput tradeoff across serving options — not official AWS pricing. Bubble size represents typical throughput for that option. Compute-optimized real-time instances and batch transform sit toward the cheaper, higher-throughput end; serverless inference's per-invocation pricing carries a premium at sustained volume, even though — as in Section 6's monthly cost comparison below — it is far cheaper than an always-on endpoint at low, spiky traffic. Batch transform's "latency" reflects typical per-record turnaround inside an offline job rather than a live response time, plotted on a log scale purely to make the comparison visible.
# Rough mental model for back-of-envelope cost comparisons (illustrative
# on-demand us-east-1 pricing — always check current AWS pricing pages,
# rates change and vary by region).
hourly_rates_usd = {
"ml.t3.medium": 0.0582, # CPU, dev/test only
"ml.m5.xlarge": 0.230, # CPU, general purpose
"ml.c5.2xlarge": 0.408, # CPU, compute-optimized
"ml.g5.xlarge": 1.408, # 1x A10G GPU
"ml.p4d.24xlarge": 37.688, # 8x A100 GPU
}
def estimate_realtime_endpoint_cost(instance_type, instance_count, hours_per_month=730):
rate = hourly_rates_usd[instance_type]
return rate * instance_count * hours_per_month
# A 2-instance ml.m5.large real-time endpoint, running 24/7 all month:
monthly = estimate_realtime_endpoint_cost("ml.m5.xlarge", instance_count=2)
print(f"Always-on 2x ml.m5.xlarge endpoint: ${monthly:,.2f}/month")
# The same endpoint serverless, assuming light traffic (~50 invocations/day,
# 2s avg duration, 4GB memory) — pay-per-use instead of pay-per-hour:
invocations_per_month = 50 * 30
serverless_cost = invocations_per_month * 0.0000166667 * 4 * 2 # approx GB-second pricing
print(f"Equivalent low-traffic serverless endpoint: ${serverless_cost:,.2f}/month")Unlike a training job, which has a natural end, a real-time endpoint runs — and bills — forever until you explicitly delete it. The single most common SageMaker cost surprise is a forgotten endpoint from a demo, a hyperparameter sweep, or an old project still running months later at $300–$1,000+/month. Treat predictor.delete_endpoint() (or aws sagemaker delete-endpoint) as a required cleanup step, not an optional one, and set up a CloudWatch billing alarm or a scheduled script that lists and flags endpoints with zero invocations over the trailing 14 days. The same applies to leftover SageMaker Studio notebook instances left running outside of business hours.
# Always clean up real-time endpoints when you're done with them.
predictor.delete_endpoint()
serverless_predictor.delete_endpoint()
# Or, to audit for orphaned endpoints across an account:
import boto3
from datetime import datetime, timezone
sm = boto3.client("sagemaker")
for ep in sm.list_endpoints()["Endpoints"]:
age_days = (datetime.now(timezone.utc) - ep["CreationTime"]).days
print(f" {ep['EndpointName']:30s} status={ep['EndpointStatus']:10s} age={age_days}d")7 SageMaker Pipelines: Orchestrating the Full Workflow
Sections 3–4 showed training and registration as separate, manually-triggered steps. In production you want this wired into a single, repeatable, versioned workflow that runs on a schedule or a trigger (new data lands, a retraining cadence fires) without a human manually re-running notebook cells. SageMaker Pipelines defines this as a DAG of steps — train, evaluate, conditionally register, conditionally deploy — expressed in Python and executed by the managed Pipelines service.
The full SageMaker Pipelines DAG from this section: raw data in S3 is processed, trained (via a built-in container or BYOC), and registered as a versioned, approval-gated model package — which can then be deployed down any of the three serving paths from Section 5, depending on the traffic shape of the consuming application.
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.steps import TrainingStep, ProcessingStep
from sagemaker.workflow.condition_step import ConditionStep
from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo
from sagemaker.workflow.functions import JsonGet
from sagemaker.workflow.model_step import ModelStep
from sagemaker.workflow.parameters import ParameterString
from sagemaker.processing import ScriptProcessor, ProcessingInput, ProcessingOutput
from sagemaker.workflow.pipeline_context import PipelineSession
pipeline_session = PipelineSession()
# Parameterize so the same pipeline definition can be reused across runs
training_data_uri = ParameterString(
name="TrainingDataUri", default_value=f"s3://{bucket}/churn-data/train/"
)
# Step 1: Train (reuses the same Estimator config from Section 3)
estimator.sagemaker_session = pipeline_session
step_train = TrainingStep(
name="TrainChurnModel",
estimator=estimator,
inputs={"train": training_data_uri},
)
# Step 2: Evaluate the trained model against a held-out test set
eval_processor = ScriptProcessor(
image_uri=estimator.training_image_uri(),
command=["python3"],
role=role,
instance_type="ml.m5.large",
instance_count=1,
sagemaker_session=pipeline_session,
)
step_evaluate = ProcessingStep(
name="EvaluateChurnModel",
processor=eval_processor,
code="src/evaluate.py",
inputs=[
ProcessingInput(
source=step_train.properties.ModelArtifacts.S3ModelArtifacts,
destination="/opt/ml/processing/model",
),
],
outputs=[ProcessingOutput(output_name="evaluation", source="/opt/ml/processing/evaluation")],
)
# Step 3: Register the model — but only if it clears a quality bar
step_register = ModelStep(
name="RegisterChurnModel",
step_args=estimator.register(
content_types=["application/json"],
response_types=["application/json"],
model_package_group_name="churn-classifier",
approval_status="PendingManualApproval",
),
)
step_conditional_register = ConditionStep(
name="CheckAUCThreshold",
conditions=[
ConditionGreaterThanOrEqualTo(
left=JsonGet(
step_name=step_evaluate.name,
property_file="evaluation_report",
json_path="metrics.auc.value",
),
right=0.85, # only register models that beat this AUC bar
)
],
if_steps=[step_register],
else_steps=[],
)
pipeline = Pipeline(
name="churn-train-eval-register-pipeline",
parameters=[training_data_uri],
steps=[step_train, step_evaluate, step_conditional_register],
sagemaker_session=pipeline_session,
)
pipeline.upsert(role_arn=role)
execution = pipeline.start()
execution.wait()This is the SageMaker analog of the CI/CD pipeline you might build around the Lesson 66 Docker image (lint → test → build → push → deploy), except the "compute" for each stage is itself managed cloud infrastructure rather than a CI runner, and the steps are ML-specific (training jobs, evaluation against held-out data, conditional registration based on a metric threshold) rather than generic build steps.
The ConditionStep in this pipeline is doing real governance work: a retraining run that produces a worse model than the current production version is automatically prevented from reaching the registry's "pending approval" queue at all, rather than relying on a human noticing a metrics regression in a dashboard. Teams running weekly or daily automated retraining lean heavily on this pattern — the pipeline can run unattended, and a human is only pulled in when a candidate model clears the automated bar and needs a final sign-off before taking production traffic.
Real-World Spotlight: Batch Forecasting and Serverless Recommendations
Retail: Nightly Demand Forecasting Across 50,000 SKUs
A mid-size retailer forecasts next-day demand for roughly 50,000 SKUs across its store and warehouse network, feeding replenishment and staffing decisions. The workload is the textbook case for batch transform: every prediction is needed by 5am the next morning, none of them are needed synchronously in response to a user action, and the full input (yesterday's sales, inventory levels, upcoming promotions, weather forecasts) is known and assembled into S3 well before the job runs. Running a persistent real-time endpoint for this would mean paying for idle GPU or CPU capacity 23 hours a day for a workload that only needs to execute once.
from sagemaker.model import Model
forecast_model = Model(
model_data="s3://retailer-ml/demand-forecast/model-v14/model.tar.gz",
image_uri=estimator.training_image_uri(),
role=role,
sagemaker_session=session,
)
# Scheduled nightly (e.g. via EventBridge -> Lambda -> this script,
# or as the final step of a SageMaker Pipeline triggered on a cron schedule)
transformer = forecast_model.transformer(
instance_count=8, # parallelize across the 50k-SKU input
instance_type="ml.c5.2xlarge", # CPU is sufficient for this gradient-boosted model
output_path="s3://retailer-ml/demand-forecast/scores/2026-07-02/",
strategy="MultiRecord",
max_concurrent_transforms=4,
)
transformer.transform(
data="s3://retailer-ml/demand-forecast/inputs/2026-07-02/",
content_type="text/csv",
split_type="Line",
)
transformer.wait()
print(f"Scored 50,000 SKUs in batch job: {transformer.latest_transform_job.job_name}")
print(f"Output: {transformer.output_path}")
print("Job billed for ~22 minutes of 8x ml.c5.2xlarge — no idle endpoint cost overnight")Startup: Serverless Inference for a Spiky Recommendation API
A small startup serves product recommendations on its marketing site. Traffic is extremely uneven: near zero overnight, modest during the day, and bursts 10–20x during a flash sale or a newsletter send. Provisioning a real-time endpoint sized for the burst would leave expensive capacity idle the rest of the time; sizing for the average would cause timeouts during bursts. Serverless Inference fits this shape directly — it scales transparently with the burst (up to the configured concurrency) and costs nothing during the long idle stretches.
from sagemaker.model import Model
from sagemaker.serverless import ServerlessInferenceConfig
from sagemaker.serializers import JSONSerializer
from sagemaker.deserializers import JSONDeserializer
recsys_model = Model(
model_data="s3://startup-ml/recsys/model-v3/model.tar.gz",
image_uri=estimator.training_image_uri(),
role=role,
sagemaker_session=session,
)
serverless_config = ServerlessInferenceConfig(
memory_size_in_mb=3072,
max_concurrency=20, # headroom for newsletter-send traffic bursts
)
recsys_predictor = recsys_model.deploy(
serverless_inference_config=serverless_config,
endpoint_name="recsys-prod-serverless",
serializer=JSONSerializer(),
deserializer=JSONDeserializer(),
)
response = recsys_predictor.predict({
"user_id": "u-88421",
"page": "product/sku-4471",
"k": 5,
})
print(response)Both spotlights share the same underlying lesson: matching the serving option to the traffic shape, not defaulting to "stand up an endpoint," is where most of the cost difference between a thoughtful SageMaker deployment and a wasteful one comes from.
✍️ Practice Exercises
- Using the SageMaker Python SDK (or a written-out plan if you don't have an AWS account handy), configure a
PyTorchEstimatorto train a small classifier withuse_spot_instances=True, a checkpoint S3 URI, and amax_run/max_waitpair. Explain what happens if the spot instance is reclaimed 80% of the way through training. - You have a model that needs to serve 200 requests/second at p99 latency under 100ms, 24/7, with traffic that doesn't vary much by time of day. Decide which of the three serving options (real-time endpoint, batch transform, serverless) fits, and justify rejecting the other two for this specific traffic shape.
- Sketch a SageMaker Pipeline (in words or code) for a fraud-detection model that must never be deployed if its precision on a fixed validation set drops below 0.95. Identify which step would contain that check and what should happen to the pipeline execution if the check fails.
- Write a boto3 script (or describe the API calls) that lists every SageMaker real-time endpoint in an account, checks each one's CloudWatch
Invocationsmetric over the trailing 14 days, and prints a warning for any endpoint with zero invocations — the audit script referenced in Section 6's cost-pitfall callout.
▶ Show Solution (Exercise 4 — Orphaned Endpoint Audit Script)
import boto3
from datetime import datetime, timedelta, timezone
sm = boto3.client("sagemaker")
cw = boto3.client("cloudwatch")
def find_idle_endpoints(idle_days=14, min_invocations=1):
"""
List SageMaker endpoints with near-zero traffic over the trailing
`idle_days` days — candidates for deletion to avoid silent cost leakage.
"""
end_time = datetime.now(timezone.utc)
start_time = end_time - timedelta(days=idle_days)
idle_endpoints = []
paginator = sm.get_paginator("list_endpoints")
for page in paginator.paginate():
for ep in page["Endpoints"]:
name = ep["EndpointName"]
if ep["EndpointStatus"] != "InService":
continue
# Sum invocations across the window from CloudWatch
metrics = cw.get_metric_statistics(
Namespace="AWS/SageMaker",
MetricName="Invocations",
Dimensions=[{"Name": "EndpointName", "Value": name}],
StartTime=start_time,
EndTime=end_time,
Period=86400,
Statistics=["Sum"],
)
total_invocations = sum(dp["Sum"] for dp in metrics["Datapoints"])
if total_invocations < min_invocations:
age_days = (end_time - ep["CreationTime"]).days
idle_endpoints.append({
"name": name,
"age_days": age_days,
"invocations_last_14d": total_invocations,
})
return idle_endpoints
idle = find_idle_endpoints()
if idle:
print(f"⚠ Found {len(idle)} idle endpoint(s) — review for deletion:")
for ep in idle:
print(f" {ep['name']:30s} age={ep['age_days']}d invocations(14d)={ep['invocations_last_14d']:.0f}")
else:
print("No idle endpoints found.")📚 Primary Source for This Lesson
AWS SageMaker Developer Guide
The official reference for the Estimator API, Model Registry, endpoint types, and SageMaker Pipelines covered throughout this lesson, including detailed instance-type and pricing guidance for the cost-control discussion in Section 6.