Kubernetes Storage¶

Overview¶

Storage in the cluster is primarily provided by Longhorn, a cloud-native distributed block storage solution.

Longhorn Architecture¶

graph TB
    subgraph Kubernetes
        PVC[PersistentVolumeClaim]
        PV[PersistentVolume]
        SC[StorageClass]
    end

    subgraph Longhorn
        LM[Longhorn Manager]
        LE[Longhorn Engine]

        subgraph Replicas
            R1[Replica 1]
            R2[Replica 2]
            R3[Replica 3]
        end
    end

    PVC --> SC
    SC --> LM
    LM --> LE
    LE --> R1
    LE --> R2
    LE --> R3

Storage Classes¶

Available Classes¶

Class	Provisioner	Replicas	Reclaim
longhorn	driver.longhorn.io	3	Delete
longhorn-retain	driver.longhorn.io	3	Retain
local-path	rancher.io/local-path	1	Delete

Default Storage Class¶

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "2880"
  fromBackup: ""

Persistent Volume Claims¶

Creating a PVC¶

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-data
  namespace: my-app
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 10Gi

Access Modes¶

Mode	Description	Longhorn Support
ReadWriteOnce (RWO)	Single node read-write	Yes
ReadOnlyMany (ROX)	Multi-node read-only	Yes
ReadWriteMany (RWX)	Multi-node read-write	Yes (NFS)

Current Storage Usage¶

By Namespace¶

Namespace	Volumes	Total Size
monitoring	3	100Gi
databases	4	200Gi
hub	1	10Gi
streamsets	1	50Gi

Volume Health¶

Monitor in Longhorn UI or Grafana:

Volume state (attached/detached)
Replica health
Used vs available space
I/O performance

Backup and Recovery¶

Longhorn Backups¶

Longhorn supports S3-compatible backup targets:

apiVersion: longhorn.io/v1beta1
kind: Setting
metadata:
  name: backup-target
  namespace: longhorn-system
value: "s3://bucket-name@region/"

Snapshot Schedule¶

apiVersion: longhorn.io/v1beta1
kind: RecurringJob
metadata:
  name: daily-backup
  namespace: longhorn-system
spec:
  cron: "0 2 * * *"
  task: backup
  groups:
    - default
  retain: 7

Volume Expansion¶

Expanding a PVC¶

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-data
spec:
  resources:
    requests:
      storage: 20Gi  # Increased from 10Gi

Note

Volume expansion requires allowVolumeExpansion: true in StorageClass

Troubleshooting¶

Common Issues¶

Issue	Cause	Resolution
Volume degraded	Replica failure	Wait for rebuild or delete bad replica
Attach failed	Node issue	Check node health, restart kubelet
Out of space	Disk full	Expand volume or add storage

Useful Commands¶

# List all PVCs
kubectl get pvc --all-namespaces

# Describe volume issue
kubectl describe pvc <name> -n <namespace>

# Check Longhorn volumes
kubectl get volumes.longhorn.io -n longhorn-system

Best Practices¶

Set resource requests: Define appropriate storage sizes
Use appropriate reclaim policy: Retain for production data
Monitor capacity: Alert before running out of space
Regular backups: Schedule automated backups
Test restores: Verify backup integrity periodically