本次ubuntu22系统# **Kubernetes 高可用集群 极速安装与排障实战手册 (v1.35.2)**## 第一章基础安装篇### 1.1 所有节点基础环境bash# 关闭 swapswapoff -a sed -i /swap/s/^/#/ /etc/fstab# 加载内核模块cat /etc/modules-load.d/k8s.conf EOFoverlaybr_netfilternf_conntrackEOFmodprobe overlay br_netfilter nf_conntrack# 配置系统参数cat /etc/sysctl.d/99-kubernetes.conf EOFnet.bridge.bridge-nf-call-iptables 1net.bridge.bridge-nf-call-ip6tables 1net.ipv4.ip_forward 1kernel.panic 10kernel.panic_on_oops 1vm.overcommit_memory 1vm.swappiness 0net.netfilter.nf_conntrack_max 1048576net.core.somaxconn 32768net.ipv4.tcp_tw_reuse 1net.ipv4.ip_local_port_range 10240 60999fs.inotify.max_user_watches 524288fs.file-max 1048576EOFsysctl -p /etc/sysctl.d/99-kubernetes.conf### 1.2 配置 containerd关键镜像加速bash# 安装 containerdapt-get update apt-get install -y containerd# 生成配置并修改mkdir -p /etc/containerdcontainerd config default | tee /etc/containerd/config.tomlsed -i s/SystemdCgroup false/SystemdCgroup true/ /etc/containerd/config.tomlsed -i s#sandbox_image registry.k8s.io/pause:3.8#sandbox_image registry.aliyuncs.com/google_containers/pause:3.10# /etc/containerd/config.toml# 能下载就不替换添加镜像加速器【注意下载不下镜像就配置下面的加速都是这样换源加速】cat /etc/containerd/config.toml EOF[plugins.io.containerd.grpc.v1.cri.registry.mirrors][plugins.io.containerd.grpc.v1.cri.registry.mirrors.docker.io]endpoint [https://docker.m.daocloud.io, https://docker.xuanyuan.me][plugins.io.containerd.grpc.v1.cri.registry.mirrors.quay.io]endpoint [https://quay.m.daocloud.io][plugins.io.containerd.grpc.v1.cri.registry.mirrors.registry.k8s.io]endpoint [https://k8s.m.daocloud.io]EOF# 重启并启用systemctl restart containerd systemctl enable containerd### 1.3 安装 kubeadm/kubelet/kubectlbash# 配置源curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.35/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpgecho deb [signed-by/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.35/deb/ / | tee /etc/apt/sources.list.d/kubernetes.listapt-get update# 安装并锁定版本apt-get install -y kubelet1.35.2-1.1 kubeadm1.35.2-1.1 kubectl1.35.2-1.1apt-mark hold kubelet kubeadm kubectl### 1.4 Master 节点初始化bashkubeadm init \--apiserver-advertise-address192.168.1.38 \--image-repository registry.aliyuncs.com/google_containers \--pod-network-cidr10.244.0.0/16 \--kubernetes-version v1.35.2# 配置 kubectlmkdir -p $HOME/.kubecp -i /etc/kubernetes/admin.conf $HOME/.kube/configchown $(id -u):$(id -g) $HOME/.kube/config# 保存 join 命令kubeadm token create --print-join-command /root/kubeadm-join-command.txt---## 第二章网络插件Calico安装与排障### 2.1 下载并修改 Calico 配置bash# 下载wget https://raw.githubusercontent.com/projectcalico/calico/v3.31.0/manifests/calico.yaml# 修改 Pod CIDR必须和初始化一致sed -i /CALICO_IPV4POOL_CIDR/,1 s/# // calico.yamlsed -i s/192.168.0.0\/16/10.244.0.0\/16/ calico.yaml# 修改镜像地址为国内源sed -i s?docker.io/calico?quay.io/calico?g calico.yaml# 注意quay.io 也需要加速已在 containerd 中配置# 应用kubectl apply -f calico.yaml### 2.2 Calico 常见问题修复bash# 问题1install-cni 容器一直失败# 解决在 Worker 节点清理 CNI 目录rm -rf /etc/cni/net.d/*rm -rf /opt/cni/bin/*systemctl restart containerd# 问题2token 缺失导致 Unauthorizedkubectl delete secret -n kube-system calico-cni-plugin-token 2/dev/nullcat EOF | kubectl apply -f -apiVersion: v1kind: Secretmetadata:name: calico-cni-plugin-tokennamespace: kube-systemannotations:kubernetes.io/service-account.name: calico-cni-plugintype: kubernetes.io/service-account-tokenEOF# 问题3BGP not established对端节点 Calico 没起来kubectl delete pod -n kube-system -l k8s-appcalico-node---## 第三章Worker 节点加入### 3.1 Worker 节点初始化脚本bash# 在 Worker 节点执行cat worker-init.sh EOF#!/bin/bash# 基础环境同 Masterswapoff -a sed -i /swap/s/^/#/ /etc/fstabmodprobe overlay br_netfilter nf_conntrackcat /etc/sysctl.d/99-kubernetes.conf EOL# ... 复制上面的 sysctl 配置 ...EOLsysctl -p /etc/sysctl.d/99-kubernetes.conf# 安装 containerd 并配置加速器同 Masterapt-get update apt-get install -y containerdmkdir -p /etc/containerdcontainerd config default | tee /etc/containerd/config.tomlsed -i s/SystemdCgroup false/SystemdCgroup true/ /etc/containerd/config.tomlsed -i s#sandbox_image registry.k8s.io/pause:3.8#sandbox_image registry.aliyuncs.com/google_containers/pause:3.10# /etc/containerd/config.toml# 添加 mirrors 配置同上systemctl restart containerd systemctl enable containerd# 安装 kubelet/kubeadmcurl -fsSL https://pkgs.k8s.io/core:/stable:/v1.35/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpgecho deb [signed-by/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.35/deb/ / | tee /etc/apt/sources.list.d/kubernetes.listapt-get updateapt-get install -y kubelet1.35.2-1.1 kubeadm1.35.2-1.1apt-mark hold kubelet kubeadmEOFchmod x worker-init.sh./worker-init.sh# 加入集群kubeadm join 192.168.1.38:6443 --token your-token --discovery-token-ca-cert-hash sha256:your-hash---## 第四章控制平面组件排障重启后的噩梦解决### 4.1 核心问题probe-port 变量未替换这是重启后 etcd、apiserver、controller-manager、scheduler 卡在 0/1 的根本原因。bash# 检查是否有 probe-port 变量grep -r probe-port /etc/kubernetes/manifests/# 一键替换所有 probe-port 为实际端口sed -i s/port: probe-port/port: 2381/g /etc/kubernetes/manifests/etcd.yaml # etcd 端口 2381sed -i s/port: probe-port/port: 10257/g /etc/kubernetes/manifests/kube-controller-manager.yaml # controller-manager 端口 10257sed -i s/port: probe-port/port: 10259/g /etc/kubernetes/manifests/kube-scheduler.yaml # scheduler 端口 10259# apiserver 通常没有 probe-port 问题如有则改 10259# 修改后无需重启kubelet 自动生效### 4.2 etcd 健康检查bash# 手动测试 etcd 健康端口curl http://127.0.0.1:2381/readyz# 测试 etcd 业务端口ETCDCTL_API3 etcdctl --endpointshttps://127.0.0.1:2379 \--cacert/etc/kubernetes/pki/etcd/ca.crt \--cert/etc/kubernetes/pki/apiserver-etcd-client.crt \--key/etc/kubernetes/pki/apiserver-etcd-client.key \endpoint health### 4.3 容器名称冲突CreateContainerErrorbash# 清理残留容器crictl ps -a | grep -E kube-controller-manager|kube-scheduler | grep Exited | awk {print $1} | xargs -r crictl rm# 重启 kubeletsystemctl restart kubelet### 4.4 一键重启所有控制平面组件bash# 移动清单文件触发重建for component in etcd kube-apiserver kube-controller-manager kube-scheduler; domv /etc/kubernetes/manifests/${component}.yaml /tmp/sleep 5mv /tmp/${component}.yaml /etc/kubernetes/manifests/donesystemctl restart kubelet---## 第五章重启后一键恢复脚本重启后先重启worker后master 只有 节点一个pod运行不起来kubectl delete pod -n kube-system calico-node-thfbj 例如这个是节点pod 直接删除kubectl get pods -n kube-system -w | grep calico-node 再观察创建 /root/k8s-recover.sh包含所有经验bash#!/bin/bashecho echo Kubernetes 重启后一键恢复脚本echo echo 1. 等待 60 秒让系统启动...sleep 60echo 2. 检查 probe-port 变量if grep -r probe-port /etc/kubernetes/manifests/; thenecho 发现 probe-port 变量正在修复...sed -i s/port: probe-port/port: 2381/g /etc/kubernetes/manifests/etcd.yamlsed -i s/port: probe-port/port: 10257/g /etc/kubernetes/manifests/kube-controller-manager.yamlsed -i s/port: probe-port/port: 10259/g /etc/kubernetes/manifests/kube-scheduler.yamlfiecho 3. 清理残留容器crictl ps -a | grep -E kube-controller-manager|kube-scheduler | grep Exited | awk {print $1} | xargs -r crictl rmecho 4. 重启 kubeletsystemctl restart kubeletsleep 30echo 5. 检查 etcd 健康curl -s http://127.0.0.1:2381/readyz echo etcd 健康 || echo etcd 异常echo 6. 重启 Calico 和 kube-proxykubectl delete pod -n kube-system -l k8s-appcalico-node 2/dev/nullkubectl delete pod -n kube-system -l k8s-appkube-proxy 2/dev/nullkubectl delete pod -n kube-system -l k8s-appkube-dns 2/dev/nullecho 7. 等待 60 秒...sleep 60echo 8. 最终状态kubectl get pods -Akubectl get nodesecho echo 恢复脚本执行完毕bashchmod x /root/k8s-recover.sh---## 第六章日常运维与验证### 6.1 快速验证集群bash# 测试 DNSkubectl run test-dns \--imagedocker.m.daocloud.io/busybox:latest \--rm -it \--restartNever \-- nslookup kubernetes.default.svc.cluster.local# 测试调度kubectl run test-nginx --imagenginx --restartNever --rm -it -- echo ok# 查看所有 Podkubectl get pods -A -o wide### 6.2 节点维护bash# 正确重启顺序先 Worker后 Master# 在 Worker 节点reboot# 等待 3 分钟后在 Master 节点reboot# 重启后执行恢复脚本/root/k8s-recover.sh### 6.3 备份重要数据bash# 备份证书cp -r /etc/kubernetes/pki /root/k8s-backup-$(date %Y%m%d)# 备份静态 Pod 清单cp -r /etc/kubernetes/manifests /root/manifests-backup-$(date %Y%m%d)---## 第七章问题速查表| 现象 | 错误日志关键词 | 快速解决 ||:---|:---|:---|| etcd 0/1 | probe-port | sed -i s/port: probe-port/port: 2381/g /etc/kubernetes/manifests/etcd.yaml || controller-manager 0/1 | probe-port | sed -i s/port: probe-port/port: 10257/g /etc/kubernetes/manifests/kube-controller-manager.yaml || scheduler 0/1 | probe-port | sed -i s/port: probe-port/port: 10259/g /etc/kubernetes/manifests/kube-scheduler.yaml || Calico Unauthorized | Unauthorized | 重新创建 token见 2.2 || Calico BGP not established | BGP not established with | 删除对端节点 Calico Pod || CreateContainerError | name is reserved for | crictl rm 清理残留容器 || 镜像拉取超时 | i/o timeout | 检查 containerd 加速器配置 |---这份手册包含了我们这次所有实战经验你只需保存为 .md 文件以后无论是新装集群还是处理重启故障都能快速解决。