🔧 Custom REST Handler — Storage Layer

Unlike CRDs (which store in etcd automatically), an extension API server implements its own storage. You can return data from any source — in-memory, a database, or computed on the fly:

// pkg/apis/mymetrics/v1alpha1/podmetricssummary_rest.go
type PodMetricsSummaryREST struct {
    metricsClient MetricsClient
}

// Get implements the Get verb for /apis/mymetrics/v1alpha1/namespaces/{ns}/podmetricssummaries/{name}
func (r *PodMetricsSummaryREST) Get(
    ctx context.Context,
    name string,
    opts *metav1.GetOptions,
) (runtime.Object, error) {
    ns := request.NamespaceValue(ctx)

    // Fetch live data from Prometheus (not etcd!)
    metrics, err := r.metricsClient.QueryPodMetrics(ctx, ns, name)
    if err != nil {
        return nil, errors.NewInternalError(err)
    }

    return &v1alpha1.PodMetricsSummary{
        ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: ns},
        CPU:    metrics.CPU,
        Memory: metrics.Memory,
        Requests: metrics.Requests,
        Errors:   metrics.Errors,
    }, nil
}

// List implements GET /apis/mymetrics/v1alpha1/namespaces/{ns}/podmetricssummaries
func (r *PodMetricsSummaryREST) List(
    ctx context.Context,
    opts *internalversion.ListOptions,
) (runtime.Object, error) {
    // Return a list of metrics for all pods in the namespace
    ns := request.NamespaceValue(ctx)
    return r.metricsClient.QueryAllPodMetrics(ctx, ns)
}

// Implement NamespacedScopeStrategy to scope resources to namespaces
func (r *PodMetricsSummaryREST) NamespaceScoped() bool { return true }
func (r *PodMetricsSummaryREST) New() runtime.Object   { return &v1alpha1.PodMetricsSummary{} }
func (r *PodMetricsSummaryREST) NewList() runtime.Object { return &v1alpha1.PodMetricsSummaryList{} }

⚖️ API Aggregation vs CRDs — Decision Guide

Availability impact — APIService status

# If the extension server is down, the APIService shows Unavailable
kubectl get apiservice v1beta1.metrics.k8s.io
# NAME                     SERVICE                      AVAILABLE   AGE
# v1beta1.metrics.k8s.io   kube-system/metrics-server   False       10m

# Describe for the error message
kubectl describe apiservice v1beta1.metrics.k8s.io
# Status:
#   Conditions:
#     Message: failing or missing response from https://10.96.x.x:443/apis/metrics.k8s.io/v1beta1
#     Reason:  ServiceNotFound
#     Status:  False
#     Type:    Available

# Impact: kubectl top nodes/pods will fail when metrics-server is down
kubectl top nodes
# Error from server (ServiceUnavailable): the server is currently unable to handle the request

📋 The APIService Object

An APIService is a cluster-scoped Kubernetes object that registers a new API group with the aggregation layer. Its name follows the convention version.group:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io    # version.group
spec:
  group:                metrics.k8s.io
  version:              v1beta1
  groupPriorityMinimum: 100       # higher = preferred when multiple groups match
  versionPriority:      100       # higher = preferred version within the group
  insecureSkipTLSVerify: true     # dev only — use caBundle in production
  service:
    name:      metrics-server
    namespace: kube-system
    port:      443

# Production: provide a CA bundle so the API server verifies the extension server's TLS cert
spec:
  caBundle: <base64-encoded-CA-cert>
  insecureSkipTLSVerify: false

Verify registration

# List all registered APIServices
kubectl get apiservices

# Check if a specific one is Available
kubectl get apiservice v1beta1.metrics.k8s.io -o yaml
# status.conditions:
# - type: Available
#   status: "True"
#   reason: Passed
#   message: all checks passed

# Query through the aggregated API
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes | jq .

Building an extension API server with apiserver-builder

The apiserver-builder library scaffolds an extension API server that handles auth delegation, discovery, and storage. It mirrors the Kubebuilder experience for aggregated APIs:

# Scaffold a new extension API server project
go install sigs.k8s.io/apiserver-builder-alpha/cmd/apiserver-boot@latest

apiserver-boot init repo --domain myorg.example.com
apiserver-boot create group version resource \
  --group mymetrics \
  --version v1alpha1 \
  --kind PodMetricsSummary

# Generated structure:
# pkg/apis/mymetrics/v1alpha1/podmetricssummary_types.go   ← type definition
# pkg/apis/mymetrics/v1alpha1/podmetricssummary_rest.go    ← REST handler
# main.go                                                   ← server entrypoint

Authentication delegation — the key mechanism

The extension API server doesn't authenticate requests itself. It delegates to the main API server using TokenReview and SubjectAccessReview:

// The extension server's authentication flow (handled by apiserver-builder):
// 1. Client sends Bearer token to extension server
// 2. Extension server calls kube-apiserver TokenReview API
//    POST /apis/authentication.k8s.io/v1/tokenreviews
// 3. If valid, extension server calls SubjectAccessReview
//    POST /apis/authorization.k8s.io/v1/subjectaccessreviews
// 4. If authorized, handle the request

# RBAC: the extension server's ServiceAccount needs permission to call these APIs
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: extension-server-auth-delegator
rules:
- apiGroups: ["authentication.k8s.io"]
  resources: ["tokenreviews"]
  verbs:     ["create"]
- apiGroups: ["authorization.k8s.io"]
  resources: ["subjectaccessreviews"]
  verbs:     ["create"]

🧠 Knowledge Check