合集 - LLM应用实战(17)1.LLM应用实战当KBQA集成LLM(二)2024-04-252.LLM应用实战当KBQA集成LLM2024-04-113.LLM实战LLM微调加速神器-Unsloth LLama32024-05-144.LLM实战LLM微调加速神器-Unsloth Qwen1.52024-05-165.LLM应用实战当图谱问答(KBQA)集成大模型(三)2024-06-126.LLM应用实战: 产业治理多标签分类2024-08-207.LLM应用实战: 文档问答系统Kotaemon-1. 简介及部署实践2024-09-188.LLM应用实战: OpenAI多代理新作-Swarm2024-10-229.LLM应用实战: AI资讯的自动聚合及报告生成2024-11-0610.LLM应用实战: 给个公司简称输出公司全称2024-11-1811.LLM应用剖析: 舆情分析多智能体-微舆BettaFish2025-11-1412.LLM应用剖析: AI对冲基金2025-11-1913.LLM应用剖析: 热点新闻助手TrendRadar2025-11-2814.LLM应用剖析: 小红书AI图文生成器-红墨2025-12-0815.LLM应用剖析: 手机智能助理Phone Agent2025-12-1116.LLM应用实践: NoteBookLM初次使用2025-12-1517.Qwen3-Embedding国产化部署03-06收起1. 背景最近一直在做ToG的项目其中用到了语义检索研发环境使用A40和vllm即可轻松部署Qwen3-Embedding-8B但客户环境要求国产化环境因此探索Qwen3-Embedding-8B如何在国产化环境部署。实不相瞒本来以为仅需半天搞定的事情我居然耗费了3天半的时间(汗颜....)究其原因主要是包含如下几个方面:研发环境没有国产化系统及显卡也没有相关的部署经验网上针对特定的300I Duo部署Qwen3-Embedding-8B的教程几乎没找到多的是910系列。但客户只有Altas 300I Duo的环境。最初两天一直在探索vllm-ascend进行部署结果现阶段版本仍不支持。最后一天转向MIS-TEI才终于看到曙光。下面教程就是部署成功的完整流程各位看官如有同样环境无脑借鉴即可省心省力。2. 系统环境操作系统: 麒麟Kylin V10 (Sword)显卡型号: 8卡Altlas 300I Duo驱动版本: Ascend-hdk-310p-npu-driver_25.5.1_linux-aarch64.run固件版本: Ascend-hdk-310p-npu-firmware_7.8.0.6.201.runCANN版本: Ascend-cann_8.5.0_linux-aarch64.run优化算子版本: Ascend-cann-310p-ops_8.5.0_linux-aarch64.run3. 驱动安装# 1. 创建用户及用户组sudo groupadd HwHiAiUsersudo useradd -g HwHiAiUser -d /home/HwHiAiUser -m -s /bin/bash HwHiAiUser# 2. 安装驱动(等待几分钟)sudo ./Ascend-hdk-310p-npu-driver_25.5.1_linux-aarch64.run --full --install-for-all --forcereboot # 重启npu-smi info # 驱动验证# 3. 安装固件sudo ./Ascend-hdk-310p-npu-firmware_7.8.0.6.201.run --fullreboot# 4. CANN驱动安装sudo ./Ascend-cann_8.5.0_linux-aarch64.run --install# 5. 安装优化算子驱动sudo ./Ascend-cann-310p-ops_8.5.0_linux-aarch64.run --install# 6. 设置驱动环境变量source /usr/local/Ascend/ascend-toolkit/set_env.shecho source /usr/local/Ascend/ascend-toolkit/set_env.sh ~/.bashrc# 7. 验证驱动是否成功npu-smi infopython3 -c import acl; print(ACL Load OK)4. Docker部署及TEI镜像拉取# 联网环境进行拉取docker pull swr.cn-south-1.myhuaweicloud.com/ascendhub/mis-tei:7.3.0-300I-Duo-aarch64# 镜像打包及传输docker save swr.cn-south-1.myhuaweicloud.com/ascendhub/mis-tei:7.3.0-300I-Duo-aarch64 -o mis-tei.tar# 客户环境加载docker load -i mis-tei.tar# 模型下载Qwen3-Embedding-8B直接通过ModelScope进行下载此处省略其过程5. 容器启动这块耗费的时间比较长因为存在各种报错最终才形成稳定版本# 1. 编写/data/models/start_tei.sh的脚本#!/bin/bashsource /usr/local/Ascend/ascend-toolkit/set_env.shexport LD_PRELOAD/usr/local/lib/python3.11/site-packages/scikit_learn.libs/libgomp-d22c30c5.so.1.0.0:/usr/lib/aarch64-linux-gnu/libgomp.so.1export PYTHONPATH/usr/local/Ascend/mxRag-7.3.0/ops/transformer_adapter:$PYTHONPATHexport RAG_SDK_HOME/usr/local/Ascend/mxRag-7.3.0export LD_LIBRARY_PATH/usr/local/Ascend/nnal/atb/8.5.0/atb/cxx_abi_0/lib:/usr/local/Ascend/mxRag-7.3.0/ops/lib:$LD_LIBRARY_PATHPORT${PORT:-8080}UDS_PATH${UDS_PATH:-/tmp/text-embeddings-inference-server}DEVICE_ID${ASCEND_VISIBLE_DEVICES:-0}export ASCEND_VISIBLE_DEVICES$DEVICE_IDexport NPU_VISIBLE_DEVICES$DEVICE_IDexport RANK0export LOCAL_RANK0exec /home/HwHiAiUser/.cargo/bin/text-embeddings-router \--model-id /home/HwHiAiUser/model/ \--hostname 0.0.0.0 \--port $PORT \--uds-path $UDS_PATH \--auto-truncate \--dtype float16 \--max-batch-tokens 16384# 2. 编写启动容器的脚本此处8张卡上默认都启动了两个进程总计16个进程(显卡资源需监测利用率)for i in $(seq 0 15); dodocker stop tei-$i 2/dev/nulldocker rm tei-$i 2/dev/nulldonefor i in $(seq 0 15); doDEVICE$((i % 8))PORT$((8080 i))UDS/tmp/tei-server-$idocker run -u root -d \-e ENABLE_BOOSTTrue \-e TEI_NPU_DEVICE$DEVICE \-e PORT$PORT \-e UDS_PATH$UDS \--nametei-$i \--nethost \--privileged \--device/dev/davinci_manager \--device/dev/hisi_hdc \--device/dev/devmm_svm \--device/dev/davinci$DEVICE \-v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \-v /usr/local/sbin:/usr/local/sbin:ro \-v /data/models/Qwen3-Embedding-8B:/home/HwHiAiUser/model \-v /data/models/start_tei.sh:/start_tei.sh \-v /etc/hostname:/etc/hostname:ro \-v /etc/hosts:/etc/hosts:ro \--entrypoint /bin/bash \swr.cn-south-1.myhuaweicloud.com/ascendhub/mis-tei:7.3.0-300I-Duo-aarch64 \/start_tei.shecho Started tei-$i on Device$DEVICE PORT$PORTsleep 5done6. nginx负载均衡转发# 1. 镜像拉取docker pull docker.1ms.run/nginx:latest# 2. nginx配置文件mkdir -p /data/nginx/logscat /data/nginx/tei.conf EOFupstream tei_cluster {least_conn;server 10.100.122.3:8080;server 10.100.122.3:8081;server 10.100.122.3:8082;server 10.100.122.3:8083;server 10.100.122.3:8084;server 10.100.122.3:8085;server 10.100.122.3:8086;server 10.100.122.3:8087;server 10.100.122.3:8088;server 10.100.122.3:8089;server 10.100.122.3:8090;server 10.100.122.3:8091;server 10.100.122.3:8092;server 10.100.122.3:8093;server 10.100.122.3:8094;server 10.100.122.3:8095;keepalive 32;}server {listen 8000;location / {proxy_pass http://tei_cluster;proxy_http_version 1.1;proxy_set_header Connection ;proxy_set_header Host $host;proxy_set_header X-Real-IP $remote_addr;proxy_read_timeout 120s;proxy_connect_timeout 5s;proxy_send_timeout 120s;}location /nginx_status {stub_status on;allow 127.0.0.1;deny all;}}EOF# 3. nginx容器启动docker run -d \--namenginx-tei \--nethost \-v /data/nginx/tei.conf:/etc/nginx/conf.d/tei.conf:ro \-v /data/nginx/logs:/var/log/nginx \docker.1ms.run/nginx:latest7. 效果测试耐朔轮吵