Kubernetes可观测性体系构建全面监控与故障排查指南一、可观测性概述可观测性Observability是指通过系统产生的数据来理解系统内部状态的能力。在Kubernetes中可观测性体系包含三个核心维度指标Metrics、日志Logs和追踪Tracing。1.1 可观测性三要素要素用途工具指标Metrics实时监控、告警、趋势分析Prometheus、Grafana日志Logs问题排查、审计、合规ELK Stack、Loki追踪Tracing分布式链路追踪、性能分析Jaeger、Zipkin1.2 可观测性架构应用层 ↓ Metrics → Prometheus → Grafana ↓ Logs → Fluentd/Loki → Grafana ↓ Tracing → Jaeger → Grafana ↓ 告警 → Alertmanager → PagerDuty/Email二、指标监控体系2.1 Prometheus部署apiVersion: v1 kind: ServiceAccount metadata: name: prometheus namespace: monitoring --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus rules: - apiGroups: [] resources: [nodes, services, endpoints, pods] verbs: [get, list, watch] - apiGroups: [] resources: [nodes/metrics] verbs: [get] --- apiVersion: apps/v1 kind: Deployment metadata: name: prometheus namespace: monitoring spec: replicas: 1 selector: matchLabels: app: prometheus template: metadata: labels: app: prometheus spec: serviceAccountName: prometheus containers: - name: prometheus image: prom/prometheus:v2.45.0 ports: - containerPort: 9090 volumeMounts: - name: config mountPath: /etc/prometheus - name: data mountPath: /prometheus volumes: - name: config configMap: name: prometheus-config - name: data persistentVolumeClaim: claimName: prometheus-pvc2.2 Prometheus配置apiVersion: v1 kind: ConfigMap metadata: name: prometheus-config namespace: monitoring data: prometheus.yml: | global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: kubernetes-apiservers kubernetes_sd_configs: - role: endpoints scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https - job_name: kubernetes-nodes kubernetes_sd_configs: - role: node scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.) - job_name: kubernetes-pods kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.)2.3 自定义指标暴露from flask import Flask from prometheus_client import Counter, Histogram, generate_latest app Flask(__name__) REQUEST_COUNT Counter(app_requests_total, Total requests) REQUEST_LATENCY Histogram(app_request_duration_seconds, Request duration) app.route(/) def hello(): REQUEST_COUNT.inc() with REQUEST_LATENCY.time(): return Hello World app.route(/metrics) def metrics(): return generate_latest(), 200, {Content-Type: text/plain; version0.0.4; charsetutf-8} if __name__ __main__: app.run(host0.0.0.0, port8080)三、日志管理体系3.1 Loki部署apiVersion: apps/v1 kind: StatefulSet metadata: name: loki namespace: monitoring spec: serviceName: loki replicas: 1 selector: matchLabels: app: loki template: metadata: labels: app: loki spec: containers: - name: loki image: grafana/loki:2.8.0 ports: - containerPort: 3100 volumeMounts: - name: data mountPath: /loki args: - -config.file/etc/loki/config.yaml volumeClaimTemplates: - metadata: name: data spec: accessModes: [ReadWriteOnce] resources: requests: storage: 100Gi3.2 Fluentd配置apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd namespace: monitoring spec: selector: matchLabels: app: fluentd template: metadata: labels: app: fluentd spec: containers: - name: fluentd image: fluent/fluentd-kubernetes-daemonset:v1.15.3 env: - name: LOKI_URL value: http://loki:3100 volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers3.3 日志查询示例# 查询指定Pod的日志 kubectl logs pod-name # 查询指定命名空间的日志 kubectl logs -n namespace pod-name # 流式日志 kubectl logs -f pod-name # 使用Loki查询 {namespacedefault, appmy-app} | error | tail 10四、分布式追踪体系4.1 Jaeger部署apiVersion: jaegertracing.io/v1 kind: Jaeger metadata: name: jaeger namespace: monitoring spec: strategy: production storage: type: elasticsearch options: es: server-urls: http://elasticsearch:92004.2 追踪集成代码from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.jaeger.thrift import JaegerExporter trace.set_tracer_provider(TracerProvider()) tracer trace.get_tracer(__name__) jaeger_exporter JaegerExporter( collector_endpointhttp://jaeger-collector:14268/api/traces ) trace.get_tracer_provider().add_span_processor( BatchSpanProcessor(jaeger_exporter) ) tracer.start_as_current_span(my-operation) def my_function(): with tracer.start_as_current_span(inner-operation): print(Inside inner operation)4.3 追踪查询# 查看Jaeger UI kubectl port-forward -n monitoring svc/jaeger-query 16686:16686 # 通过trace ID查询 curl http://localhost:16686/api/traces/trace-id五、告警与通知体系5.1 Alertmanager配置apiVersion: v1 kind: ConfigMap metadata: name: alertmanager-config namespace: monitoring data: config.yml: | global: resolve_timeout: 5m route: group_by: [alertname] group_wait: 10s group_interval: 10s repeat_interval: 1h receiver: email receivers: - name: email email_configs: - to: adminexample.com send_resolved: true inhibit_rules: - source_match: severity: critical target_match: severity: warning equal: [alertname, dev, instance]5.2 告警规则配置apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: example-rules namespace: monitoring spec: groups: - name: node.rules rules: - alert: HighCPUUsage expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{modeidle}[1m])) * 100) 90 for: 5m labels: severity: critical annotations: summary: High CPU usage on {{ $labels.instance }} description: CPU usage is above 90% for 5 minutes - alert: HighMemoryUsage expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 85 for: 5m labels: severity: warning annotations: summary: High memory usage on {{ $labels.instance }} description: Memory usage is above 85% for 5 minutes六、Grafana可视化6.1 Grafana部署apiVersion: apps/v1 kind: Deployment metadata: name: grafana namespace: monitoring spec: replicas: 1 selector: matchLabels: app: grafana template: metadata: labels: app: grafana spec: containers: - name: grafana image: grafana/grafana:10.1.0 ports: - containerPort: 3000 env: - name: GF_SECURITY_ADMIN_PASSWORD valueFrom: secretKeyRef: name: grafana-secret key: admin-password volumeMounts: - name: data mountPath: /var/lib/grafana volumes: - name: data persistentVolumeClaim: claimName: grafana-pvc6.2 仪表盘配置apiVersion: grafana.integreatly.org/v1beta1 kind: GrafanaDashboard metadata: name: kubernetes-dashboard namespace: monitoring spec: configMapRef: name: kubernetes-dashboard-config datasources: - inputName: DS_PROMETHEUS datasourceName: Prometheus七、可观测性最佳实践7.1 指标命名规范# 指标命名格式 # namespace_component_metric_unit # 示例 api_http_requests_total # 总请求数 api_request_duration_seconds # 请求持续时间 database_connection_pool_size # 数据库连接池大小7.2 日志结构化{ timestamp: 2024-01-15T10:30:00Z, level: INFO, service: order-service, trace_id: abc-123, span_id: def-456, message: Order created successfully, data: { order_id: ORD-001, customer_id: CUS-123 } }7.3 采样策略apiVersion: v1 kind: ConfigMap metadata: name: tracing-config data: tracing.yaml: | sampling: rate: 0.1 # 10%采样率 max_samples_per_second: 100八、可观测性监控8.1 监控指标# 查看Prometheus状态 kubectl get pods -n monitoring -l appprometheus # 检查告警状态 kubectl get alerts -n monitoring # 查看Grafana状态 kubectl get pods -n monitoring -l appgrafana8.2 健康检查apiVersion: v1 kind: Pod metadata: name: observability-check spec: containers: - name: health-check image: busybox:1.28 command: - /bin/sh - -c - | curl -f http://prometheus:9090/-/ready || exit 1 curl -f http://loki:3100/ready || exit 1 curl -f http://jaeger-query:16686/ || exit 1九、总结构建完善的Kubernetes可观测性体系需要指标监控使用Prometheus收集关键指标日志管理使用Loki和Fluentd聚合日志分布式追踪使用Jaeger追踪请求链路可视化使用Grafana展示数据告警通知配置Alertmanager发送告警建议根据业务需求选择合适的工具组合并持续优化监控策略。参考资料Prometheus官方文档Grafana官方文档Jaeger官方文档Loki官方文档