🗺️ Why Cloud-Native Storage?

Cloud-managed disks (EBS, GCP PD) work well in a single cloud but limit portability and cost control. Cloud-native storage systems run inside the cluster — using your nodes' disks — and expose Kubernetes-native interfaces (StorageClass, PVC, CSI).

Two dominant open-source options: Rook/Ceph (enterprise-grade distributed storage, highly scalable) and Longhorn (simpler, lightweight distributed block storage designed for ease of use).

🟠 Rook / Ceph Deep Dive

Key Components

Component	Role
Rook Operator	Kubernetes controller that manages the entire Ceph lifecycle via CRDs
MON (Monitor)	Maintains the Ceph cluster map; requires a quorum (3 or 5 pods)
MGR (Manager)	Provides metrics, dashboard, and orchestration modules
OSD (Object Storage Daemon)	One per disk; stores actual data with replication
MDS (Metadata Server)	Required for CephFS (shared filesystem / RWX volumes)
RGW (RADOS Gateway)	S3/Swift-compatible object storage endpoint

Install via Helm

# Add Rook Helm chart repo
helm repo add rook-release https://charts.rook.io/release
helm repo update

# Install Rook operator
helm install rook-ceph rook-release/rook-ceph \
  --namespace rook-ceph \
  --create-namespace \
  --version v1.13.0

# Deploy a CephCluster (uses raw disks on each node)
kubectl apply -f - <<EOF
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v18
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
  storage:
    useAllNodes: true
    useAllDevices: false
    deviceFilter: "^sd[b-z]"   # use all non-boot disks
  dashboard:
    enabled: true
EOF

Block Storage StorageClass (RBD)

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  replicated:
    size: 3          # 3-way replication
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: replicapool
  imageFormat: "2"
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
reclaimPolicy: Delete
allowVolumeExpansion: true

Shared Filesystem (CephFS / RWX)

apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: myfs
  namespace: rook-ceph
spec:
  metadataPool:
    replicated: { size: 3 }
  dataPools:
    - name: replicated
      replicated: { size: 3 }
  metadataServer:
    activeCount: 1
    activeStandby: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-cephfs
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  clusterID: rook-ceph
  fsName: myfs
  pool: myfs-replicated
reclaimPolicy: Delete
allowVolumeExpansion: true

ℹ️ RWX with CephFS CephFS StorageClass supports ReadWriteMany — multiple pods across nodes can mount the same volume simultaneously. Essential for shared config files, media uploads, or ML training datasets.