🕸️ What Istio Actually Does

Istio injects an Envoy sidecar proxy into every pod. All inbound and outbound traffic is transparently redirected through this proxy using iptables rules. The proxy implements mTLS, retries, circuit breaking, load balancing, and telemetry — without any application code changes.

Istiod (Control Plane) Pilot · Citadel · Galley — unified since 1.5 Pod: checkout-api App Envoy sidecar Pod: payment-svc Envoy sidecar App xDS config xDS config mTLS (Envoy ↔ Envoy)

Installing Istio (production-grade)

# Install istioctl
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.20.0 sh -
export PATH=$PATH:./istio-1.20.0/bin

# Production install with IstioOperator profile
istioctl install --set profile=default -y

# Or via Helm (recommended for GitOps)
helm repo add istio https://istio-release.storage.googleapis.com/charts
helm install istio-base    istio/base    -n istio-system --create-namespace
helm install istiod        istio/istiod  -n istio-system --wait
helm install istio-ingress istio/gateway -n istio-ingress --create-namespace

# Enable sidecar injection for a namespace
kubectl label namespace production istio-injection=enabled

# Verify injection is working
kubectl get pods -n production
# NAME                    READY   STATUS    — 2/2 means app + envoy
# checkout-api-7d9f-xkp   2/2     Running

Istio's Core CRDs

VirtualService

Defines routing rules: which traffic goes where, with retries, timeouts, fault injection, and traffic splitting. The "router".

DestinationRule

Defines policies applied to traffic after routing: load balancing, connection pools, circuit breaking, mTLS mode per destination.

Gateway

Configures an Envoy proxy at the edge (ingress/egress). Works with VirtualService to bind external traffic to internal services.

PeerAuthentication

Sets the mTLS mode for a namespace or workload: STRICT (no plaintext), PERMISSIVE (both allowed), DISABLE.

AuthorizationPolicy

L7 access control: allow/deny by source principal (SPIFFE), namespace, HTTP method, path, headers. Zero-trust authorization.

ServiceEntry

Registers external services (outside the mesh) so Istio can apply policies to egress traffic. Enables traffic control to external APIs.

🔐 Security — mTLS & Authorization

Enforcing strict mTLS namespace-wide

# Step 1: PERMISSIVE — allow both mTLS and plaintext (migration phase)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: PERMISSIVE

# Step 2: Once all sidecars are injected → STRICT
spec:
  mtls:
    mode: STRICT   # reject any non-mTLS connection — zero-trust enforced

AuthorizationPolicy — L7 zero-trust access control

AuthorizationPolicy enforces access at L7 based on cryptographic SPIFFE identity — not IP addresses. This is the Istio feature that replaces per-service authentication middleware:

# Only checkout-svc SA can POST to /api/v1/charge on payment-svc
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payment-svc-authz
  namespace: production
spec:
  selector:
    matchLabels: { app: payment-svc }
  action: ALLOW
  rules:
  - from:
    - source:
        principals:
        - "cluster.local/ns/production/sa/checkout-svc"
    to:
    - operation:
        methods: ["POST"]
        paths:   ["/api/v1/charge", "/api/v1/refund"]
  - from:
    - source:
        namespaces: ["monitoring"]    # allow Prometheus scraping
    to:
    - operation:
        paths:   ["/metrics"]
        methods: ["GET"]

---
# Default-deny for the entire namespace (implicit deny when no ALLOW rule matches)
# Just create an empty AuthorizationPolicy — no rules = deny all
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: production
spec:
  {}   # empty spec = deny all traffic into the namespace
⚠️ AuthorizationPolicy evaluation order Istio evaluates policies as: DENY overrides → ALLOW matches → default deny. Create the default-deny policy last, after all ALLOW policies are in place, to avoid locking yourself out. Test with istioctl x authz check <pod> before enforcing.

📊 Observability — The Killer Feature

Istio's sidecars emit rich telemetry automatically — no application instrumentation needed. The kiali/Grafana/Jaeger suite turns this into actionable visibility:

# Install Kiali, Prometheus, Grafana, Jaeger (telemetry addons)
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/prometheus.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/grafana.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/jaeger.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/kiali.yaml

# Access Kiali dashboard (service topology + mTLS status)
istioctl dashboard kiali

# Key Istio metrics exposed by each Envoy sidecar:
# istio_requests_total{source_workload, destination_workload, response_code}
# istio_request_duration_milliseconds_bucket
# istio_tcp_connections_opened_total

Telemetry API — customise what Envoy collects

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: production-telemetry
  namespace: production
spec:
  tracing:
  - providers:
    - name: jaeger
    randomSamplingPercentage: 1.0    # 1% sampling for high-traffic
  metrics:
  - providers:
    - name: prometheus
  accessLogging:
  - providers:
    - name: envoy
    disabled: false
💡 Production Istio resource overhead Each Envoy sidecar uses ~50 MB memory and ~0.1 CPU at idle. At 100 pods that's 5 GB RAM dedicated to the mesh. Tune concurrency (Envoy worker threads) and set proper resources.requests on the sidecar via ProxyConfig. Use Ambient Mesh (Istio 1.21+) to eliminate sidecars entirely for lower overhead.

🚦 Traffic Management in Practice

Canary deployment with traffic splitting

# Deploy v2 alongside v1 — same Service selector, different labels
# Deployment v1: labels: app: checkout, version: v1
# Deployment v2: labels: app: checkout, version: v2

# DestinationRule: define subsets
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: checkout-api
  namespace: production
spec:
  host: checkout-api
  subsets:
  - name: v1
    labels: { version: v1 }
  - name: v2
    labels: { version: v2 }
  trafficPolicy:
    connectionPool:
      http:
        http2MaxRequests:   1000
        h2UpgradePolicy:    UPGRADE
    outlierDetection:
      consecutiveErrors:  5
      interval:           10s
      baseEjectionTime:   30s   # circuit breaker
# VirtualService: 90% → v1, 10% → v2 canary
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: checkout-api
  namespace: production
spec:
  hosts:
  - checkout-api
  http:
  - match:
    - headers:
        x-canary:
          exact: "true"     # QA team always gets v2
    route:
    - destination:
        host:   checkout-api
        subset: v2
  - route:                   # everyone else: 90/10 split
    - destination:
        host:   checkout-api
        subset: v1
      weight: 90
    - destination:
        host:   checkout-api
        subset: v2
      weight: 10
    retries:
      attempts:            3
      perTryTimeout:       2s
      retryOn:             gateway-error,connect-failure,retriable-4xx
    timeout: 10s

Fault injection — test resilience in production-like environments

# Inject a 5s delay for 10% of requests to test timeout handling
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payment-svc-fault
spec:
  hosts: [payment-svc]
  http:
  - fault:
      delay:
        percentage: { value: 10 }
        fixedDelay: 5s
      abort:
        percentage: { value: 2 }
        httpStatus: 503   # 2% of requests return 503
    route:
    - destination:
        host: payment-svc

Ingress Gateway — replacing nginx Ingress

# Istio Gateway + VirtualService for external traffic
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: main-gateway
spec:
  selector:
    istio: ingress
  servers:
  - port: { number: 443, name: https, protocol: HTTPS }
    tls:
      mode:              SIMPLE
      credentialName:    api-tls-cert    # Secret name
    hosts:             [api.example.com]
  - port: { number: 80, name: http, protocol: HTTP }
    tls:
      httpsRedirect: true   # redirect all HTTP → HTTPS
    hosts:             [api.example.com]

---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api-external
spec:
  hosts:    [api.example.com]
  gateways: [main-gateway]
  http:
  - route:
    - destination:
        host: checkout-api.production.svc.cluster.local
        port: { number: 8080 }

🧠 Knowledge Check

Q1. You have two Deployments for checkout-api with labels version: v1 and version: v2. Both match the same Service selector. How do you route 10% of traffic to v2 without changing the Service?

A) Create two Services, one per version, and use Kubernetes round-robin
B) Scale v2 to 10% of v1's replica count — Kubernetes will naturally route proportionally
C) Create a DestinationRule with v1/v2 subsets and a VirtualService with weight: 90/10
D> Set a canary annotation on the Deployment — Istio reads Kubernetes annotations

Q2. An AuthorizationPolicy with an empty spec (spec: {}) is created in a namespace. What is the effect?

A) No effect — an empty spec is ignored by Istio
B) All traffic into the namespace is denied — default-deny is established
C) All traffic is allowed — empty spec means unrestricted
D> Only external traffic is denied; pod-to-pod within the namespace is unaffected

Q3. What is the difference between a VirtualService and a DestinationRule?

A) VirtualService is for external traffic; DestinationRule is for internal traffic
B) They are interchangeable — both configure Envoy proxy rules
C) VirtualService = WHERE traffic goes (routing); DestinationRule = HOW it behaves at the destination (circuit breaker, LB, mTLS)
D> VirtualService handles TLS; DestinationRule handles HTTP routing

Q4. Your cluster has 200 pods. Istio sidecars are adding ~50 MB RAM each — 10 GB total overhead. What is the best way to reduce this while keeping mesh features?

A) Remove mTLS — it is the largest contributor to sidecar memory
B) Switch to Linkerd — it has lighter proxies than Envoy
C) Use Istio Ambient Mesh — eliminates per-pod sidecars using a shared per-node ztunnel
D> Reduce Envoy log verbosity — logging is the main memory consumer