yaml 格式Pod 管理yaml 格式yaml格式只使用空格缩进对于空格的数量没有强制要求正常使用2个空格。基本规则• 同一级别的元素使用相同的缩进。• 对于子项目使用比父项目更多的缩进。• 增加空白行提高可读性。yaml 示例vim设置为粘贴模式set paste排版不会错乱。playbook.yaml 内容如下# yaml格式起始行一般不省略 --- # Playbook中第一个play # play具有属性namehostsbecometasks缩进一致 # name属性用于简要描述play - name: debploy WebSite # hosts属性用于定义要在哪个受管理节点执行 hosts: webs # tasks属性用于描述play中任务属性是列表格式 tasks: # 第一个任务 # 任务具有属性涵name和模块名等。 # name属性用于简要描述任务 - name: latest version of httpd and firewalld installed # 指明模块名也就是要执行的任务 yum: # 指定要操作的rpm包名称 name: # rpm包名称是-开头的列表格式或者逗号分隔的列表格式 - httpd - firewalld # 定义软件包的状态lastet代表升级为最新版本 state: latest # 第二个任务 - name: prepare index.html # copy 模块用于将content属性值写入到目标文件 copy: content: Welcome to Laoma WebSite!\n dest: /var/www/html/index.html # 第三个任务 - name: enable and start httpd # service模块用于启用并启动httpd服务 service: name: httpd enabled: true state: started # 第四个任务 - name: enable and start firewalld # service模块用于启用并启动firewalld服务 service: name: firewalld enabled: true state: started # 第五个任务 - name: firewalld permits access to httpd service # firewalld用于放行http服务 firewalld: service: http permanent: true state: enabled immediate: yes # Playbook中第二个play-开头表示列表 - name: Test WebSite hosts: localhost become: no tasks: - name: connect to intranet web server # uri模块用于测试网站是否可以访问 uri: url: http://{{item}} loop: - node1 - node2 # yaml格式结束行一般省略 ...YAML 注释在 YAML中 编号或井号符号(#)右侧的所有内容都是注释。如果注释的左侧有内容 请在该编号符号的前面加一个空格。注释可用于提高可读性。示例# This is YAML comment Some data # This is also a YAML commentYAML 单行字符串YAML中的字符串通常不需要放在引号里即使字符串中包含空格。字符串也可以用双引号或单引号括起。this is a string this is another string this is yet another a stringYAML 多行字符串可以使用竖线(I)字符表示保留字符串中的换行字符示例yaml • name: test string hosts: node1 tasks: ◦ name: test string # 用户显示变量值或者字符串 debug: msg: | Example Company 123 Main Street Atlanta, GA 30303 •也可以使用大于号()字符表示换行字符。执行时换行符使用空格代替并且行内的引导空白将被删除。yaml • name: test string hosts: node1 tasks: ◦ name: test string # 用户显示变量值或者字符串 debug: msg: This is an example of a long string, that will become a single sentence once folded. 这种方法通常用于将很长的字符串在空格字符处断行使它们跨占多行来提高可读性。YAML 字典一组键值对的集合又称为映射mapping和哈希hashes。以缩进块的形式编写键值对集合如下方所示user属性是字典格式是多个键值对集合user: name: laoma uid: 1088 state: absent字典也可以使用以花括号括起的内联块格式编写如下方所示user: {name: laoma, uid: 1088, state: absent}大多数情形中应避免内联块格式其可读性较差。不过当playbook中包含角色列表时使用这种语法更加容易区分play中包含的角色和传递给角色的变量。某些 playbook 可能使用较旧的简写shorthand格式通过将模块的键值对放在与模块名称相同的行上来定义任务。- name: shorhand form user: namelaoma uid1088 stateabsent普通格式- name: shorhand form user: name: laoma uid: 1088 state: absent两者格式总结通常您应避免简写格式而使用普通格式。• 普通格式的行数较多更容易操作。任务的关键字垂直堆叠更容易区分。 阅读play时您的眼睛直接向下扫视左右运动较少。• 普通格式是原生的YAML现代文本编辑器中的语法突出显示工具可以识别简写形式则不支持。• 可能会在文档和他人提供的旧playbook中看到这种语法而且这种语法仍然可以发挥作用。YAML 列表一组按次序排列的值又称为序列sequence和数组array。以缩进块的形式编写的键值对集合如下方所示- name: latest version of httpd and firewalld installed yum: name: - httpd - firewalld state: latest - name: test html page is installed copy: content: Welcome to the example.com intranet!\n dest: /var/www/html/index.htmlk8s中pod定义示例apiVersion: v1 kind: Pod metadata: labels: run: wordpress name: wordpress namespace: pods spec: containers: - image: wordpress imagePullPolicy: Always name: wordpress nodeName: worker31.laoma.cloud restartPolicy: Always schedulerName: default-scheduler serviceAccount: default serviceAccountName: default volumes: - name: kube-api-access projected: defaultMode: 420 sources:Kubernetes Pod多个容器指定网络 --network host容器跟宿主机公用网络。nginx使用宿主机80端口mysql使用宿主机3306端口多个容器指定 -v /data:/data容器跟宿主机公用网络。环境准备[rootmaster30 ~]# kubectl create ns pods [rootmaster30 ~]# kubectl config set-context --current --namespace podskubectl api-resources[rootmaster30 ~]# kubectl api-resources |egrep NAMESPACED|namespace|pod|nodes NAME SHORTNAMES APIVERSION NAMESPACED KIND namespaces ns v1 false Namespace nodes no v1 false Node pods po v1 true Pod podtemplates v1 true PodTemplate horizontalpodautoscalers hpa autoscaling/v2 true HorizontalPodAutoscaler caliconodestatuses crd.projectcalico.org/v1 false CalicoNodeStatus poddisruptionbudgets pdb policy/v1 true PodDisruptionBudget csinodes storage.k8s.io/v1 false CSINode多容器 pod示例文件blog.yaml[rootmaster30 ~]# vim pod-blog.yaml apiVersion: v1 kind: Pod metadata: name: bbs labels: run: bbs spec: containers: - image: mysql:latest imagePullPolicy: IfNotPresent name: mysql env: - name: MYSQL_ROOT_PASSWORD value: 123 - name: MYSQL_USER value: tom - name: MYSQL_PASSWORD value: 123 - name: MYSQL_DATABASE value: bbs ports: - containerPort: 3306 name: mysql protocol: TCP - image: wordpress:latest imagePullPolicy: IfNotPresent name: wordpress env: - name: WORDPRESS_DB_USER value: tom - name: WORDPRESS_DB_PASSWORD value: 123 - name: WORDPRESS_DB_NAME value: bbs - name: WORDPRESS_DB_HOST value: 127.0.0.1 ports: - containerPort: 80 name: wordpress protocol: TCP hostPort: 80[rootmaster30 ~]# kubectl apply -f pod-blog.yaml [rootmaster30 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES bbs 2/2 Running 0 10m 10.224.225.68 worker32.laoma.cloud none none多容器pod中执行命令通过-c指定容器[rootmaster30 ~]# kubectl exec bbs -c wordpress -- hostname bbs [rootmaster30 ~]# kubectl cp /etc/hosts bbs:/new-hosts -c wordpress [rootmaster30 ~]# kubectl exec bbs -c wordpress -- ls /new-hosts /new-hosts创建 svc# 创建 svc [rootmaster30 ~]# kubectl expose pod bbs --type NodePort [rootmaster30 ~]# kubectl get svc bbs NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE bbs NodePort 10.98.5.216 none 3306:32590/TCP,80:30576/TCP 118s数据库访问测试[rootmaster30 ~]# kubectl exec -it bbs -c mysql -- mysql -utom -p123 Type help; or \h for help. Type \c to clear the current input statement. mysqlpod 关键属性[rootmaster30 ~]# kubectl explain pod | grep ^ [a-zA-Z] apiVersion string kind string metadata ObjectMeta spec PodSpec status PodStatuspod.metadata[rootmaster30 ~]# kubectl explain pod.metadata | grep ^ [a-zA-Z] annotations map[string]string clusterName string creationTimestamp string deletionGracePeriodSeconds integer deletionTimestamp string finalizers []string generateName string generation integer labels map[string]string managedFields []Object name string namespace string ownerReferences []Object resourceVersion string selfLink string uid string需要关注的属性labelsnamenamespace。pod.spec[rootmaster30 ~]# kubectl explain pod.spec | grep ^ [a-zA-Z] activeDeadlineSeconds integer affinity Object automountServiceAccountToken boolean containers []Object -required- dnsConfig Object dnsPolicy string enableServiceLinks boolean ephemeralContainers []Object hostAliases []Object hostIPC boolean hostNetwork boolean hostPID boolean hostname string imagePullSecrets []Object initContainers []Object nodeName string nodeSelector map[string]string overhead map[string]string preemptionPolicy string priority integer priorityClassName string readinessGates []Object restartPolicy string runtimeClassName string schedulerName string securityContext Object serviceAccount string serviceAccountName string setHostnameAsFQDN boolean shareProcessNamespace boolean subdomain string terminationGracePeriodSeconds integer tolerations []Object topologySpreadConstraints []Object volumes []Object重点关注containers、nodeName、volumes等。pod.spec.containers[rootmaster30 ~]# kubectl explain pod.spec.containers | grep ^ [a-zA-Z] args []string command []string env []Object envFrom []Object image string imagePullPolicy string lifecycle Object livenessProbe Object name string -required- ports []Object readinessProbe Object resources Object securityContext Object startupProbe Object stdin boolean stdinOnce boolean terminationMessagePath string terminationMessagePolicy string tty boolean volumeDevices []Object volumeMounts []Object workingDir string重点关注command、env、image、imagePullPolicy、volumeMounts等。pod.spec.containers.ImagePullPolicyAlways总是从仓库下载镜像。• **Never**只使用本地镜像不下载。•IfNotPresent优先使用本地镜像如果没有才从仓库下载镜像。pod lifecycle容器的运行状态取决于容器中的进程。pod的运行状态取决于pod中所有容器的状态。Pod and Container statusContainerCreating 正在创建 • Running 正在运行 • Completed 运行完成 • Error 运行错误 • CrashLoopBackOff 重新创建 • ErrImagePull 获取镜像错误 • ImagePullBackOff 重新获取镜像container statesWaiting: 等待某个条件满足变成Running状态例如下载镜像更新secrets等。 通过describe pod查看message和reason详细信息。 • Running: 容器正在运行没有问题。同时记录Running开始时间。 • Terminated: 容器运行完成也有可能是运行失败终止。实验1pod中包含1个容器验证pod状态监控命令 watch -n 1 kubectl get podapiVersion: v1 kind: Pod metadata: name: busybox labels: app: busybox spec: restartPolicy: Never containers: - name: busybox image: busybox imagePullPolicy: IfNotPresent command: [sh, -c, echo Hello Kubernetes! sleep 10]command命令可以改写如下command: - sh - -c - echo OK! sleep 10或者args: - sh - -c - echo OK! sleep 10观察pod状态为ContainerCreating–Running–Completed示例2apiVersion: v1 kind: Pod metadata: name: busybox labels: app: busybox spec: restartPolicy: Never containers: - name: busybox image: busybox imagePullPolicy: IfNotPresent command: [sh, -c, echoxxx Hello Kubernetes! sleep 5]实验2pod中包含2个容器验证pod状态apiVersion: v1 kind: Pod metadata: name: busybox labels: app: busybox spec: restartPolicy: Never containers: - name: busybox1 image: busybox imagePullPolicy: IfNotPresent command: [sh, -c, echo Hello Kubernetes! sleep 5] - name: busybox2 image: busybox imagePullPolicy: IfNotPresent command: [sh, -c, echo Hello Kubernetes! sleep 10]观察pod状态为ContainerCreating–Running–NotReady–CompletedapiVersion: v1 kind: Pod metadata: name: busybox labels: app: busybox spec: restartPolicy: Never containers: - name: busybox1 image: busybox imagePullPolicy: IfNotPresent command: [sh, -c, echoxx Hello Kubernetes! sleep 5] - name: busybox2 image: busybox imagePullPolicy: IfNotPresent command: [sh, -c, echo Hello Kubernetes! sleep 20]观察pod状态为ContainerCreating–Errorpod.spec.restartPolicyrestartPolicy: Never containers: - name: myapp-container image: busybox command: [sh, -c, echo The app is running! sleep 5]restartPolicy针对pod中所有容器生效。Always除了 Running 状态其他状态总是重启默认值。 • OnFailure失败了才重启。 • Never从不重启静态 pod问题正常情况下pod 由 master 节点统一管理。那么 master 上的kube-apiserver、kube-scheduler、kube-controller-manager等pod又是由谁来管理的呢答案kubelet 服务。master节点上组件以静态Pod方式运行由本地kubelet进行管理。它们不能通过API Server进行管理无法与ReplicationController、Deployment或DaemonSet进行关联并且kubelet也无法对其健康检查。我们来分析下 kubelet.service 配置。[rootmaster30 ~]# systemctl status kubelet.service |cat ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: enabled) Drop-In: /usr/lib/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since Sun 2024-07-14 09:52:08 UTC; 1h 7min ago Docs: https://kubernetes.io/docs/ Main PID: 7489 (kubelet) Tasks: 11 (limit: 4557) Memory: 39.0M (peak: 39.9M) CPU: 1min 45.913s CGroup: /system.slice/kubelet.service └─7489 /usr/bin/kubelet --bootstrap-kubeconfig/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig/etc/kubernetes/kubelet.conf --config/var/lib/kubelet/config.yaml --container-runtime-endpointunix:///var/run/containerd/containerd.sock --pod-infra-container-imageregistry.k8s.io/pause:3.9 ...... [rootmaster30 ~]# cat /usr/lib/systemd/system/kubelet.service [Unit] Descriptionkubelet: The Kubernetes Node Agent Documentationhttps://kubernetes.io/docs/ Wantsnetwork-online.target Afternetwork-online.target [Service] ExecStart/usr/bin/kubelet Restartalways StartLimitInterval0 RestartSec10 [Install] WantedBymulti-user.target # kubelet.service服务启动参数通过 Drop-In 文件 10-kubeadm.conf 配置 [rootmaster30 ~]# cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf # Note: This dropin only works with kubeadm and kubelet v1.11 [Service] EnvironmentKUBELET_KUBECONFIG_ARGS--bootstrap-kubeconfig/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig/etc/kubernetes/kubelet.conf EnvironmentKUBELET_CONFIG_ARGS--config/var/lib/kubelet/config.yaml # This is a file that kubeadm init and kubeadm join generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically EnvironmentFile-/var/lib/kubelet/kubeadm-flags.env # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file. EnvironmentFile-/etc/default/kubelet ExecStart ExecStart/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS注意不要修改 staticPodPath 参数配置否则需要将集群组件4个yaml文件也要复制过去。我们还可以通过参数 --pod-manifest-path设置该目录下所有 pod 也是静态 pod。[rootmaster30 ~]# kubelet --help|grep pod-manifest-path --pod-manifest-path string Path to the directory containing static pod files to run, or the path to a single static pod file. Files starting with dots will be ignored. (DEPRECATED: This parameter should be set via the config file specified by the Kubelets --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)思考kubernetes 中 pod 和 容器区别答容器是运行时概念真正跑应用的进程环境。kubernetes 中 Pod是 K8s 独有的概念是 K8s 中最小调度、管理、自愈单元是容器的“外壳 运行环境”。可以为pod额外配置健康检查、重启策略、资源配额等。• pod 有 独立 IP内部容器共享IP。• pod 有 独立存储内部容器共享存储。• 一个 Pod 里可以有 1 个或多个容器pod中所有容器• 同住一个房间共用网络、存储。• 一起被调度到同一台机器一起销毁。kubernetes 为什么直接管理 pod 而不是容器1.容器太“原子”不适合直接调度。K8s 需要一个能直接被调度、能独立运行、有完整身份的对象–Pod。•容器只是一个进程/运行环境Docker/containerd。•Pod是容器的封装 网络/存储/配置/生命周期的统一抽象。K8s 调度、扩缩容、自愈、服务发现、监控……全都以 Pod 为单位。2.Pod 支持“多容器协同”最关键设计很多场景必须多个容器一起跑、共享资源业务容器 Sidecar日志、监控、代理• 业务容器 InitContainer初始化• 业务容器 网络/安全代理容器它们需要共享 Network Namespace同一个 IP、端口空间• 共享 Volume• 一起调度到同一台机器3.Pod 提供统一的生命周期与自愈K8s 要做• 重启失败容器• 替换崩溃节点上的实例• 滚动更新、回滚• 扩缩容4.解耦底层容器运行时Pod 屏蔽了底层实现• Docker• containerd• cri-o• 其他 CRI 运行时K8s 只跟 Pod/CRI 打交道不绑定某一种容器技术。解释 pod 中 pause 容器作用在 Kubernetes 里每个 Pod 都会自动创建一个 pause 容器也叫 infra 容器它是 Pod 里第一个启动、最后退出的容器作用非常关键pause 容器是干嘛的pause 容器就是为了“占坑”把 Pod 的网络和命名空间先 hold 住。它的 3 个核心作用创建并持有 Pod 的 Linux NamespacePod 里所有容器要共享Network namespace同一个 IP、端口• PID namespace可选• IPC namespace这些共享空间必须由一个“永远不死”的容器来持有否则共享空间会消失。这个容器就是 pause。让 Pod 生命周期独立于业务容器业务容器挂了 → 重启• pause 容器不挂 → Pod 就不会消失• 网络、IP、存储挂载都能保持不变实现多容器共享网络nginx、业务容器、sidecar 都加入 pause 的 network namespace共享同一个 IP• 可以用 127.0.0.1 互相访问• 端口不能冲突