云原生环境中的存储管理从PV到StorageClass的全面解析 硬核开场各位技术大佬们今天咱们来聊聊云原生环境中的存储管理。别以为存储就是简单的挂载磁盘在云原生环境中存储管理涉及PV、PVC、StorageClass、StatefulSet等多个组件复杂得很今天susu就带你们深入解析云原生环境中的存储管理最佳实践从存储类型选择到备份策略从StatefulSet部署到存储性能优化全给你整明白 核心内容1. 云原生存储的挑战动态性容器的快速创建和销毁持久化需要数据在容器重启后保持可扩展性存储需要随应用规模扩展性能满足不同应用的性能需求可靠性确保数据安全和高可用2. 存储类型2.1 临时存储apiVersion: v1 kind: Pod metadata: name: temp-storage-pod spec: containers: - name: app image: nginx volumeMounts: - name: temp-volume mountPath: /tmp volumes: - name: temp-volume emptyDir: {}2.2 持久卷PV和持久卷声明PVC# 定义PV apiVersion: v1 kind: PersistentVolume metadata: name: my-pv spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: standard hostPath: path: /mnt/data --- # 定义PVC apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: standard --- # 使用PVC apiVersion: v1 kind: Pod metadata: name: pv-pod spec: containers: - name: app image: nginx volumeMounts: - name: persistent-storage mountPath: /usr/share/nginx/html volumes: - name: persistent-storage persistentVolumeClaim: claimName: my-pvc2.3 存储类StorageClassapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast provisioner: kubernetes.io/aws-ebs parameters: type: gp3 iopsPerGB: 10000 encrypted: true reclaimPolicy: Delete allowVolumeExpansion: true volumeBindingMode: WaitForFirstConsumer --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: slow provisioner: kubernetes.io/aws-ebs parameters: type: gp2 encrypted: true reclaimPolicy: Retain allowVolumeExpansion: true volumeBindingMode: Immediate3. 云提供商存储3.1 AWS EBSapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: aws-ebs-gp3 defaultClass: true provisioner: kubernetes.io/aws-ebs parameters: type: gp3 iopsPerGB: 10000 encrypted: true reclaimPolicy: Delete allowVolumeExpansion: true volumeBindingMode: WaitForFirstConsumer3.2 GCP Persistent DiskapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gcp-pd-ssd provisioner: kubernetes.io/gce-pd parameters: type: pd-ssd replication-type: none reclaimPolicy: Delete allowVolumeExpansion: true volumeBindingMode: WaitForFirstConsumer3.3 Azure DiskapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: azure-disk-premium provisioner: kubernetes.io/azure-disk parameters: storageaccounttype: Premium_LRS kind: Managed reclaimPolicy: Delete allowVolumeExpansion: true volumeBindingMode: WaitForFirstConsumer4. StatefulSet与存储4.1 部署StatefulSetapiVersion: apps/v1 kind: StatefulSet metadata: name: mysql-statefulset namespace: default spec: serviceName: mysql replicas: 3 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql:8.0 env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-secret key: password ports: - containerPort: 3306 volumeMounts: - name: mysql-data mountPath: /var/lib/mysql volumeClaimTemplates: - metadata: name: mysql-data spec: accessModes: [ReadWriteOnce] storageClassName: fast resources: requests: storage: 20Gi4.2 服务发现apiVersion: v1 kind: Service metadata: name: mysql namespace: default spec: selector: app: mysql ports: - port: 3306 targetPort: 3306 clusterIP: None5. 存储性能优化5.1 配置存储QoSapiVersion: v1 kind: PersistentVolumeClaim metadata: name: high-performance-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 50Gi storageClassName: fast volumeMode: Filesystem5.2 使用本地存储apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: local-storage provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer --- apiVersion: v1 kind: PersistentVolume metadata: name: local-pv spec: capacity: storage: 100Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /mnt/disks/ssd1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node15.3 存储分片apiVersion: apps/v1 kind: StatefulSet metadata: name: sharded-app namespace: default spec: serviceName: sharded-app replicas: 4 selector: matchLabels: app: sharded-app template: metadata: labels: app: sharded-app spec: containers: - name: app image: my-app:latest volumeMounts: - name: data mountPath: /data volumeClaimTemplates: - metadata: name: data spec: accessModes: [ReadWriteOnce] storageClassName: fast resources: requests: storage: 100Gi6. 存储备份与恢复6.1 使用Velero# 安装Velero wget https://github.com/vmware-tanzu/velero/releases/download/v1.9.0/velero-v1.9.0-linux-amd64.tar.gz tar -xzf velero-v1.9.0-linux-amd64.tar.gz mv velero-v1.9.0-linux-amd64/velero /usr/local/bin/ # 配置Velero velero install \ --provider aws \ --plugins velero/velero-plugin-for-aws:v1.5.0 \ --bucket velero-backups \ --secret-file ./credentials-velero \ --backup-location-config regionus-east-1 \ --snapshot-location-config regionus-east-1 # 创建备份 velero backup create mysql-backup --include-namespaces default # 查看备份 velero backup get # 恢复备份 velero restore create --from-backup mysql-backup6.2 数据库备份# 创建备份脚本 cat EOF backup-mysql.sh #!/bin/bash POD_NAME$(kubectl get pods -l appmysql -o jsonpath{.items[0].metadata.name}) kubectl exec $POD_NAME -- mysqldump -u root -p$MYSQL_ROOT_PASSWORD --all-databases mysql-backup-$(date %Y%m%d).sql EOF chmod x backup-mysql.sh # 执行备份 ./backup-mysql.sh # 恢复备份 POD_NAME$(kubectl get pods -l appmysql -o jsonpath{.items[0].metadata.name}) kubectl cp mysql-backup-20230401.sql $POD_NAME:/tmp/ kubectl exec $POD_NAME -- mysql -u root -p$MYSQL_ROOT_PASSWORD /tmp/mysql-backup-20230401.sql7. 存储安全7.1 加密存储apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: encrypted-storage provisioner: kubernetes.io/aws-ebs parameters: type: gp3 encrypted: true kmsKeyId: arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012 reclaimPolicy: Delete allowVolumeExpansion: true volumeBindingMode: WaitForFirstConsumer7.2 访问控制apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: storage-admin namespace: default rules: - apiGroups: [storage.k8s.io] resources: [storageclasses, persistentvolumes, persistentvolumeclaims] verbs: [get, list, create, update, delete] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: storage-admin-binding namespace: default subjects: - kind: User name: admin apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: storage-admin apiGroup: rbac.authorization.k8s.io8. 存储监控8.1 Prometheus监控apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: kubernetes-storage namespace: monitoring spec: selector: matchLabels: k8s-app: kubelet endpoints: - port: http-metrics interval: 15s path: /metrics honorLabels: true metricRelabelings: - sourceLabels: [__name__] regex: kubelet_volume_stats_.* action: keep8.2 Grafana Dashboard{ annotations: { list: [ { builtIn: 1, datasource: -- Grafana --, enable: true, hide: true, iconColor: rgba(0, 211, 255, 1), name: Annotations Alerts, type: dashboard } ] }, editable: true, gnetId: null, graphTooltip: 0, id: 1, links: [], panels: [ { aliasColors: {}, bars: false, dashLength: 10, dashes: false, datasource: Prometheus, fieldConfig: { defaults: { custom: {} }, overrides: [] }, fill: 1, fillGradient: 0, gridPos: { h: 8, w: 12, x: 0, y: 0 }, hiddenSeries: false, id: 2, legend: { avg: false, current: false, max: false, min: false, show: true, total: false, values: false }, lines: true, linewidth: 1, nullPointMode: null, options: { alertThreshold: true }, percentage: false, pluginVersion: 7.3.6, pointradius: 2, points: false, renderer: flot, seriesOverrides: [], spaceLength: 10, stack: false, steppedLine: false, targets: [ { expr: sum(kubelet_volume_stats_available_bytes{namespace\default\}) by (persistentvolumeclaim), interval: , legendFormat: {{persistentvolumeclaim}}, refId: A } ], thresholds: [], timeFrom: null, timeRegions: [], timeShift: null, title: 存储可用空间, tooltip: { shared: true, sort: 0, value_type: individual }, type: graph, xaxis: { buckets: null, mode: time, name: null, show: true, values: [] }, yaxes: [ { format: bytes, label: null, logBase: 1, max: null, min: 0, show: true }, { format: short, label: null, logBase: 1, max: null, min: null, show: true } ], yaxis: { align: false, alignLevel: null } } ], refresh: 5s, schemaVersion: 26, style: dark, tags: [], templating: { list: [] }, time: { from: now-6h, to: now }, timepicker: {}, timezone: , title: 存储监控, uid: storage-monitoring, version: 1 }9. 高级存储配置9.1 本地持久卷apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: local-storage provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer --- apiVersion: v1 kind: PersistentVolume metadata: name: local-pv-1 spec: capacity: storage: 1Ti accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /mnt/disks/ssd1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node1 --- apiVersion: v1 kind: PersistentVolume metadata: name: local-pv-2 spec: capacity: storage: 1Ti accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /mnt/disks/ssd2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node29.2 存储快照apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-aws-vsc driver: kubernetes.io/aws-ebs deletionPolicy: Delete --- apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: mysql-snapshot spec: volumeSnapshotClassName: csi-aws-vsc source: persistentVolumeClaimName: mysql-data --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-restore spec: storageClassName: fast dataSource: name: mysql-snapshot kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: - ReadWriteOnce resources: requests: storage: 20Gi9.3 存储池管理# 安装OpenEBS helm repo add openebs https://openebs.github.io/charts helm repo update helm install openebs openebs/openebs --namespace openebs --create-namespace # 验证安装 kubectl get pods -n openebs # 创建存储池 kubectl apply -f - EOF apiVersion: openebs.io/v1alpha1 kind: StoragePoolClaim metadata: name: openebs-pool namespace: openebs spec: name: openebs-pool type: hostpath poolSpec: poolType: striped hostpath: path: /mnt/openebs EOF # 创建存储类 kubectl apply -f - EOF apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: openebs-standard provisioner: openebs.io/local parameters: storagePool: openebs-pool fsType: ext4 reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer EOF10. 存储管理工具kubectlKubernetes命令行工具# 查看PV kubectl get pv # 查看PVC kubectl get pvc # 查看StorageClass kubectl get storageclassVelero备份和恢复工具# 查看备份 velero backup get # 查看恢复 velero restore get # 查看备份位置 velero backup-location getOpenEBS容器原生存储# 查看存储池 kubectl get spc -n openebs # 查看卷 kubectl get cstorvolume -n openebsRook云原生存储编排# 查看Ceph集群 kubectl get cephcluster -n rook-ceph # 查看存储池 kubectl get cephblockpool -n rook-ceph # 查看存储类 kubectl get storageclass | grep rook️ 最佳实践选择合适的存储类型根据应用需求选择存储类型数据库等有状态应用使用持久存储临时数据使用emptyDir配置文件使用ConfigMapStorageClass配置创建多个StorageClass满足不同性能需求配置合适的reclaimPolicy启用volume expansion使用WaitForFirstConsumer绑定模式StatefulSet部署使用volumeClaimTemplates自动创建PVC配置Headless Service用于服务发现合理设置副本数考虑使用StatefulSet的有序部署和删除性能优化使用本地存储提高性能配置合适的存储QoS考虑存储分片监控存储性能指标备份与恢复定期备份重要数据使用Velero进行集群级备份测试恢复流程存储备份到外部存储安全配置启用存储加密配置适当的访问控制限制存储资源使用定期审计存储配置监控与告警监控存储使用情况设置存储容量告警监控存储I/O性能建立存储健康检查自动化管理使用Helm管理存储组件实施存储配置自动化建立存储配置版本控制自动化存储备份文档和培训建立存储架构文档培训团队存储知识记录存储配置变更分享存储最佳实践持续改进定期评估存储性能优化存储配置学习和应用新的存储技术适应业务需求变化 总结云原生环境中的存储管理是应用稳定运行的重要组成部分涉及多个组件和配置。通过本文的实践你应该已经掌握了存储类型的选择和使用PV、PVC和StorageClass的配置云提供商存储的使用StatefulSet与存储的集成存储性能优化策略存储备份与恢复存储安全配置存储监控与告警高级存储配置存储管理工具的使用记住存储配置是一个持续优化的过程需要根据业务需求和应用特点不断调整。在实际生产环境中要结合具体情况选择合适的存储方案确保数据的安全和高可用。susu碎碎念存储是数据的基础一定要重视选择存储类型时要考虑性能和成本定期备份数据避免数据丢失监控存储使用情况及时扩容存储安全不能忽视要加密敏感数据文档很重要要记录存储架构和配置持续学习存储新技术保持技术先进性觉得有用点个赞再走咱们下期见