DeOldify服务监控方案PrometheusGrafana实时跟踪GPU利用率与QPS1. 监控方案概述在实际的AI服务部署中仅仅能够运行服务是不够的。我们需要实时了解服务的运行状态、资源使用情况以及性能指标。对于DeOldify这样的深度学习图像上色服务GPU利用率和每秒查询数QPS是两个最关键的性能指标。传统的日志查看方式无法提供实时的可视化监控而PrometheusGrafana组合能够为我们提供完整的监控解决方案。这个方案可以帮助我们实时监控GPU使用情况避免资源浪费或瓶颈跟踪服务处理能力了解QPS变化趋势设置告警阈值及时发现异常情况历史数据回溯分析性能优化效果2. 环境准备与组件安装2.1 安装PrometheusPrometheus是一个开源的系统监控和警报工具包专门用于收集和存储时间序列数据。# 下载Prometheus wget https://github.com/prometheus/prometheus/releases/download/v2.47.2/prometheus-2.47.2.linux-amd64.tar.gz # 解压 tar xvfz prometheus-2.47.2.linux-amd64.tar.gz cd prometheus-2.47.2.linux-amd64 # 创建配置文件 cat prometheus.yml EOF global: scrape_interval: 15s scrape_configs: - job_name: prometheus static_configs: - targets: [localhost:9090] - job_name: node static_configs: - targets: [localhost:9100] - job_name: gpu static_configs: - targets: [localhost:9435] - job_name: deoldify static_configs: - targets: [localhost:8000] EOF # 启动Prometheus ./prometheus --config.fileprometheus.yml 2.2 安装Node ExporterNode Exporter用于收集硬件和操作系统指标。# 下载Node Exporter wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz # 解压并启动 tar xvfz node_exporter-1.6.1.linux-amd64.tar.gz cd node_exporter-1.6.1.linux-amd64 ./node_exporter 2.3 安装GPU Exporter对于GPU监控我们需要专门的GPU exporter。# 使用DCGM Exporter需要先安装NVIDIA DCGM docker run -d --rm \ --gpus all \ -p 9400:9400 \ nvcr.io/nvidia/k8s/dcgm-exporter:3.2.6-3.2.2-ubuntu20.04 # 或者使用简单的nvidia-smi exporter git clone https://github.com/utkuozdemir/nvidia_gpu_exporter cd nvidia_gpu_exporter docker build -t nvidia_gpu_exporter . docker run -d --rm \ --gpus all \ -p 9835:9835 \ nvidia_gpu_exporter2.4 安装GrafanaGrafana用于可视化监控数据。# 安装Grafana wget https://dl.grafana.com/oss/release/grafana-10.2.0.linux-amd64.tar.gz tar xvfz grafana-10.2.0.linux-amd64.tar.gz cd grafana-10.2.0 # 启动Grafana ./bin/grafana-server web 3. DeOldify服务指标暴露要让Prometheus能够收集DeOldify服务的指标我们需要在服务中添加指标暴露功能。3.1 添加Prometheus客户端库首先在DeOldify服务中添加Prometheus客户端依赖# requirements.txt 添加 prometheus-client0.19.03.2 创建指标收集中间件为Flask应用添加Prometheus指标收集from prometheus_client import Counter, Gauge, Histogram, generate_latest, REGISTRY from flask import Response, request import time # 定义指标 REQUEST_COUNT Counter( deoldify_requests_total, Total number of requests, [method, endpoint, status] ) REQUEST_LATENCY Histogram( deoldify_request_latency_seconds, Request latency in seconds, [method, endpoint] ) GPU_UTILIZATION Gauge( deoldify_gpu_utilization_percent, GPU utilization percentage, [gpu_index] ) GPU_MEMORY Gauge( deoldify_gpu_memory_usage_mb, GPU memory usage in MB, [gpu_index] ) QPS Gauge( deoldify_qps, Queries per second ) # 添加到Flask应用 def setup_metrics(app): app.before_request def before_request(): request.start_time time.time() app.after_request def after_request(response): # 计算请求耗时 latency time.time() - request.start_time REQUEST_LATENCY.labels( methodrequest.method, endpointrequest.path ).observe(latency) # 统计请求次数 REQUEST_COUNT.labels( methodrequest.method, endpointrequest.path, statusresponse.status_code ).inc() return response # 添加metrics端点 app.route(/metrics) def metrics(): return Response(generate_latest(REGISTRY), mimetypetext/plain)3.3 集成GPU监控添加GPU使用情况监控import subprocess import re import threading def get_gpu_utilization(): 获取GPU使用情况 try: result subprocess.check_output([ nvidia-smi, --query-gpuindex,utilization.gpu,memory.used, --formatcsv,noheader,nounits ]).decode(utf-8) for line in result.strip().split(\n): if line: gpu_index, utilization, memory_used line.split(, ) GPU_UTILIZATION.labels(gpu_indexgpu_index).set(float(utilization)) GPU_MEMORY.labels(gpu_indexgpu_index).set(float(memory_used)) except Exception as e: print(f获取GPU信息失败: {e}) def start_gpu_monitor(): 启动GPU监控线程 def monitor_loop(): while True: get_gpu_utilization() time.sleep(5) # 每5秒更新一次 thread threading.Thread(targetmonitor_loop, daemonTrue) thread.start()3.4 集成QPS计算添加QPS计算功能from collections import deque import time class QPSCalculator: def __init__(self, window_size10): self.window_size window_size self.request_times deque() self.lock threading.Lock() def add_request(self): with self.lock: current_time time.time() self.request_times.append(current_time) # 移除超过时间窗口的请求记录 while self.request_times and self.request_times[0] current_time - self.window_size: self.request_times.popleft() def get_qps(self): with self.lock: if not self.request_times: return 0.0 time_window self.window_size if len(self.request_times) 1: time_window min(self.window_size, self.request_times[-1] - self.request_times[0]) return len(self.request_times) / time_window if time_window 0 else 0.0 # 全局QPS计算器 qps_calculator QPSCalculator() def update_qps_metrics(): 更新QPS指标 while True: qps qps_calculator.get_qps() QPS.set(qps) time.sleep(1) # 在请求处理中更新QPS app.after_request def after_request(response): # ... 其他代码 ... qps_calculator.add_request() return response4. Grafana仪表板配置4.1 添加数据源首先在Grafana中添加Prometheus数据源访问Grafana界面通常为http://localhost:3000默认用户名/密码admin/admin进入Configuration Data Sources添加Prometheus数据源URL填写http://localhost:90904.2 创建监控仪表板创建DeOldify服务监控仪表板包含以下关键面板GPU利用率面板{ title: GPU利用率, type: gauge, targets: [ { expr: deoldify_gpu_utilization_percent, legendFormat: GPU {{gpu_index}} } ], maxDataPoints: 100, interval: 10s }QPS监控面板{ title: 每秒查询数 (QPS), type: graph, targets: [ { expr: rate(deoldify_requests_total[1m]), legendFormat: 总QPS } ], refresh: 5s }请求延迟面板{ title: 请求延迟分布, type: heatmap, targets: [ { expr: histogram_quantile(0.95, sum(rate(deoldify_request_latency_seconds_bucket[5m])) by (le)), legendFormat: P95延迟 } ] }GPU内存使用面板{ title: GPU内存使用, type: stat, targets: [ { expr: deoldify_gpu_memory_usage_mb, legendFormat: GPU {{gpu_index}} } ], colorMode: value, graphMode: area }4.3 设置告警规则在Prometheus中配置告警规则# alert.rules.yml groups: - name: deoldify-alerts rules: - alert: HighGPUUsage expr: deoldify_gpu_utilization_percent 90 for: 5m labels: severity: warning annotations: summary: GPU使用率过高 description: GPU {{ $labels.gpu_index }} 使用率持续5分钟超过90% - alert: LowQPS expr: deoldify_qps 1 for: 10m labels: severity: warning annotations: summary: QPS过低 description: 服务QPS持续10分钟低于1 - alert: HighLatency expr: histogram_quantile(0.95, deoldify_request_latency_seconds_bucket[5m]) 10 for: 5m labels: severity: critical annotations: summary: 请求延迟过高 description: P95请求延迟持续5分钟超过10秒5. 实战部署与验证5.1 完整部署脚本创建一键部署脚本#!/bin/bash # deploy_monitoring.sh echo 开始部署DeOldify监控系统... # 创建监控目录 mkdir -p /opt/monitoring cd /opt/monitoring # 安装Prometheus echo 安装Prometheus... wget -q https://github.com/prometheus/prometheus/releases/download/v2.47.2/prometheus-2.47.2.linux-amd64.tar.gz tar xfz prometheus-2.47.2.linux-amd64.tar.gz cd prometheus-2.47.2.linux-amd64 # 配置Prometheus cat prometheus.yml EOF global: scrape_interval: 15s scrape_configs: - job_name: prometheus static_configs: - targets: [localhost:9090] - job_name: node static_configs: - targets: [localhost:9100] - job_name: gpu static_configs: - targets: [localhost:9435] - job_name: deoldify static_configs: - targets: [localhost:7860] metrics_path: /metrics EOF # 启动Prometheus nohup ./prometheus --config.fileprometheus.yml prometheus.log 21 # 安装Node Exporter echo 安装Node Exporter... cd /opt/monitoring wget -q https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz tar xfz node_exporter-1.6.1.linux-amd64.tar.gz cd node_exporter-1.6.1.linux-amd64 nohup ./node_exporter node_exporter.log 21 # 安装Grafana echo 安装Grafana... cd /opt/monitoring wget -q https://dl.grafana.com/oss/release/grafana-10.2.0.linux-amd64.tar.gz tar xfz grafana-10.2.0.linux-amd64.tar.gz cd grafana-10.2.0 nohup ./bin/grafana-server web grafana.log 21 echo 监控系统部署完成 echo Prometheus: http://localhost:9090 echo Grafana: http://localhost:3000 echo 默认用户名/密码: admin/admin5.2 验证监控系统使用以下脚本验证监控系统是否正常工作# test_monitoring.py import requests import time import threading def test_metrics_endpoint(): 测试metrics端点 try: response requests.get(http://localhost:7860/metrics, timeout5) if response.status_code 200: print(✓ Metrics端点正常) return True else: print(f✗ Metrics端点异常: {response.status_code}) return False except Exception as e: print(f✗ Metrics端点访问失败: {e}) return False def test_prometheus(): 测试Prometheus try: response requests.get(http://localhost:9090/api/v1/query?queryup, timeout5) if response.status_code 200: data response.json() if data[data][result]: print(✓ Prometheus正常运行) return True print(✗ Prometheus异常) return False except Exception as e: print(f✗ Prometheus访问失败: {e}) return False def test_grafana(): 测试Grafana try: response requests.get(http://localhost:3000/api/health, timeout5) if response.status_code 200: print(✓ Grafana正常运行) return True print(✗ Grafana异常) return False except Exception as e: print(f✗ Grafana访问失败: {e}) return False def generate_test_traffic(): 生成测试流量 import random endpoints [/colorize, /health, /ui] for i in range(20): endpoint random.choice(endpoints) try: if endpoint /colorize: # 模拟图片上传请求 files {image: (test.jpg, open(test.jpg, rb))} requests.post(fhttp://localhost:7860{endpoint}, filesfiles, timeout10) else: requests.get(fhttp://localhost:7860{endpoint}, timeout5) print(f✓ 测试请求 {endpoint} 成功) except Exception as e: print(f✗ 测试请求 {endpoint} 失败: {e}) time.sleep(random.uniform(0.1, 1.0)) if __name__ __main__: print(开始验证监控系统...) # 测试各组件 metrics_ok test_metrics_endpoint() prometheus_ok test_prometheus() grafana_ok test_grafana() if all([metrics_ok, prometheus_ok, grafana_ok]): print(\n✓ 监控系统基础功能正常) print(开始生成测试流量...) generate_test_traffic() print(\n测试完成请查看Grafana仪表板确认监控数据) else: print(\n✗ 监控系统存在异常请检查相关服务)6. 监控方案优化建议6.1 性能优化对于高并发场景监控系统本身也需要优化# 优化指标收集性能 from prometheus_client import CollectorRegistry, multiprocess, generate_latest def create_app(): app Flask(__name__) # 使用多进程注册表 registry CollectorRegistry() multiprocess.MultiProcessCollector(registry) app.route(/metrics) def metrics(): return Response(generate_latest(registry), mimetypetext/plain) return app6.2 数据保留策略调整Prometheus数据保留策略# prometheus.yml global: scrape_interval: 15s evaluation_interval: 15s # 数据保留15天 retention: 15d # 使用TSDB压缩 storage: tsdb: retention: 15d min-block-duration: 2h max-block-duration: 24h6.3 监控数据备份设置监控数据备份策略# backup_monitoring_data.sh #!/bin/bash BACKUP_DIR/backup/monitoring DATE$(date %Y%m%d_%H%M%S) # 备份Prometheus数据 tar -czf $BACKUP_DIR/prometheus_$DATE.tar.gz /opt/monitoring/prometheus-2.47.2.linux-amd64/data/ # 备份Grafana配置 tar -czf $BACKUP_DIR/grafana_$DATE.tar.gz /opt/monitoring/grafana-10.2.0/conf/ # 保留最近7天的备份 find $BACKUP_DIR -name *.tar.gz -mtime 7 -delete7. 总结通过PrometheusGrafana监控方案我们为DeOldify图像上色服务建立了一套完整的监控体系。这个方案能够实时监控GPU利用率确保GPU资源得到合理利用避免成为性能瓶颈跟踪QPS变化了解服务处理能力为扩容和优化提供数据支持可视化监控数据通过Grafana仪表板直观展示关键指标设置智能告警及时发现异常情况确保服务稳定性历史数据分析回溯性能数据支持容量规划和优化决策这套监控方案不仅适用于DeOldify服务也可以很容易地适配到其他AI推理服务中。通过实时监控和告警我们能够确保服务始终处于最佳运行状态为用户提供稳定可靠的图像上色服务。获取更多AI镜像想探索更多AI镜像和应用场景访问 CSDN星图镜像广场提供丰富的预置镜像覆盖大模型推理、图像生成、视频生成、模型微调等多个领域支持一键部署。