doris operator部署Doris集群教程
- 前言
- 部署流程
前言
kubernetes Operator是遵循kubernetes API和控制器模式，它主要用来封装运维业务逻辑的软件。它利用kubernetes的自定义资源定义（CRD）扩展API，并通过控制器模式监听资源对象，实现应用的部署和生命周期。大多数人使用kubernetes的时候是使用原生资源（pod、deployment、Service等），当然也可以使用operator来特定需求的新业务逻辑。
kubernetes Operator组成主要包括以下几个部分：
1.自定义资源定义（CRD）：CRD 是对kubernetes API的扩展，允许用户定义自己的资源类型和属性。Operator使用CRD来定义自己管理的应用资源对象。
2.控制器（Controller）：控制器监视自定义资源对象的状态，并根据定义的业务逻辑对资源对象进行协调，让状态达到预期。
3.管理器（manager）:它本质上就是一个管理器框架。主要负责启动所有控制器，使多个控制器可以协同工作。而且它还提供共享缓存和客户端，不同控制器可以共享这些资源。实现了控制器逻辑的并发和同步。
Doris 官网上有 Kubernetes 部署的文档，无奈根据官网的文档，构建完镜像无法成功启动。故参考官网做了一些改动，成功启动 FE、BE 节点。此探索为临时部署方案，还需再完善、优化。目前好消息就是，Doris提供的Operator可以方便部署Doris集群。
部署流程
部署非常简单，首先在github下载Doris的operator项目，链接在doris operator。然后，根据operator部署Doris cluster，在本篇文章中，例子有三个节点,具体如下所示。
| 地址 | node |
|---|---|
| 172.16.43.71 | fe 节点 |
| 172.16.43.72 | be节点 |
| 172.16.43.73 | be节点 |
doris cluster的yaml文件如下
# this yaml is the simplest mode to deploy doris with fe and be.
# the log and be storage use `EmptyDir` mode as disk. when the pod restart the log and storage will lost.
# This yaml only for experience deploy and usability on k8s about doris.
apiVersion: doris.selectdb.com/v1
kind: DorisCluster
metadata:
labels:
app.kubernetes.io/name: doriscluster
app.kubernetes.io/instance: doriscluster-sample
app.kubernetes.io/part-of: doris-operator
name: doriscluster
namespace: doris
spec:
feSpec:
replicas: 1
image: selectdb/doris.fe-ubuntu:2.0.2
envVars:
- name: TZ
value: "Asia/Shanghai"
configMapInfo:
# use kubectl create configmap fe-configmap --from-file=fe.conf
configMapName: fe-configmap
resolveKey: fe.conf
service:
type: NodePort
servicePorts:
- nodePort: 31030
targetPort: 9030
- nodePort: 31031
targetPort: 8030
requests:
memory: 1Gi
limits:
memory: 2Gi
persistentVolumes:
- mountPath: /opt/apache-doris/fe/doris-meta
name: fe-data
persistentVolumeClaimSpec:
# when use specific storageclass, the storageClassName should reConfig, example as annotation.
storageClassName: doris-fe-storage
accessModes:
- ReadWriteOnce
resources:
# notice: if the storage size less 5G, fe will not start normal.
requests:
storage: 5Gi
beSpec:
replicas: 2
image: selectdb/doris.be-ubuntu:2.0.2
envVars:
- name: TZ
value: "Asia/Shanghai"
systemInitialization:
command: ["/sbin/sysctl", "-w", "vm.max_map_count=2000000"]
configMapInfo:
# use kubectl create configmap be-configmap --from-file=be.conf
configMapName: be-configmap
resolveKey: be.conf
persistentVolumes:
- mountPath: /opt/apache-doris/be/storage
name: be-data
persistentVolumeClaimSpec:
# when use specific storageclass, the storageClassName should reConfig, example as annotation.
storageClassName: doris-be-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 6Gi
operator的yaml文件定义如下
# permissions to do leader election.
apiVersion: v1
kind: Namespace
metadata:
labels:
control-plane: doris-operator
app.kubernetes.io/name: namespace
app.kubernetes.io/instance: doris
app.kubernetes.io/component: doris-operator
app.kubernetes.io/part-of: doris-operator
name: doris
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/name: role
app.kubernetes.io/instance: leader-election-role
app.kubernetes.io/component: rbac
app.kubernetes.io/created-by: doris-operator
app.kubernetes.io/part-of: doris-operator
app.kubernetes.io/managed-by: kustomize
name: leader-election-role
namespace: doris
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/name: rolebinding
app.kubernetes.io/instance: leader-election-rolebinding
app.kubernetes.io/component: rbac
app.kubernetes.io/created-by: doris-operator
app.kubernetes.io/part-of: doris-operator
app.kubernetes.io/managed-by: kustomize
name: leader-election-rolebinding
namespace: doris
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: leader-election-role
subjects:
- kind: ServiceAccount
name: doris-operator
namespace: doris
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: null
name: doris-operator
rules:
- apiGroups:
- apps
resources:
- statefulsets
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- apps
resources:
- statefulsets/status
verbs:
- get
- apiGroups:
- autoscaling
resources:
- horizontalpodautoscalers
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- serviceaccounts
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- services
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- doris.selectdb.com
resources:
- dorisclusters
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- doris.selectdb.com
resources:
- dorisclusters/finalizers
verbs:
- update
- apiGroups:
- doris.selectdb.com
resources:
- dorisclusters/status
verbs:
- get
- patch
- update
- apiGroups:
- rbac.authorization.k8s.io
resources:
- clusterrolebindings
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- rbac.authorization.k8s.io
resources:
- rolebindings
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/name: clusterrolebinding
app.kubernetes.io/instance: doris-operator-rolebinding
app.kubernetes.io/component: rbac
app.kubernetes.io/created-by: doris-operator
app.kubernetes.io/part-of: doris-operator
app.kubernetes.io/managed-by: kustomize
name: doris-operator-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: doris-operator
subjects:
- kind: ServiceAccount
name: doris-operator
namespace: doris
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/name: serviceaccount
app.kubernetes.io/instance: controller-doris-operator-sa
app.kubernetes.io/component: rbac
app.kubernetes.io/created-by: doris-operator
app.kubernetes.io/part-of: doris-operator
app.kubernetes.io/managed-by: kustomize
name: doris-operator
namespace: doris
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: doris-operator
namespace: doris
labels:
control-plane: doris-operator
app.kubernetes.io/name: deployment
app.kubernetes.io/instance: doris-operator
app.kubernetes.io/component: doris-operator
app.kubernetes.io/created-by: doris-operator
app.kubernetes.io/part-of: doris-operator
spec:
selector:
matchLabels:
control-plane: doris-operator
replicas: 1
template:
metadata:
annotations:
kubectl.kubernetes.io/default-container: doris-operator
labels:
control-plane: doris-operator
spec:
# TODO(user): Uncomment the following code to configure the nodeAffinity expression
# according to the platforms which are supported by your solution.
# It is considered best practice to support multiple architectures. You can
# build your manager image using the makefile target docker-buildx.
# affinity:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: kubernetes.io/arch
# operator: In
# values:
# - amd64
# - arm64
# - ppc64le
# - s390x
# - key: kubernetes.io/os
# operator: In
# values:
# - linux
securityContext:
runAsNonRoot: true
# TODO(user): For common cases that do not require escalating privileges
# it is recommended to ensure that all your Pods/Containers are restrictive.
# More info: https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted
# Please uncomment the following code if your project does NOT have to work on old Kubernetes
# versions < 1.19 or on vendors versions which do NOT support this field by default (i.e. Openshift < 4.11 ).
# seccompProfile:
# type: RuntimeDefault
containers:
- command:
- /dorisoperator
args:
- --leader-elect
image: selectdb/doris.k8s-operator:latest
name: dorisoperator
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- "ALL"
livenessProbe:
httpGet:
path: /healthz
port: 8081
initialDelaySeconds: 15
periodSeconds: 20
readinessProbe:
httpGet:
path: /readyz
port: 8081
initialDelaySeconds: 5
periodSeconds: 10
# TODO(user): Configure the resources accordingly based on the project requirements.
# More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
resources:
limits:
cpu: "2"
memory: 4Gi
requests:
cpu: "1"
memory: 2Gi
serviceAccountName: doris-operator
terminationGracePeriodSeconds: 10
fe的storageclass的yaml文件如下
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: doris-fe-storage
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
fe的pv定义yaml文件如下
apiVersion: v1
kind: PersistentVolume
metadata:
name: fe-node-71
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: doris-fe-storage
local:
path: /data/doris/fe
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- 172.16.43.71
be的storageclass定义yaml文件如下
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: doris-be-storage
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
be的pv定义yaml文件如下
apiVersion: v1
kind: PersistentVolume
metadata:
name: be-node-72
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: doris-be-storage
local:
path: /data/doris/be/storage
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- 172.16.43.72
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: be-node-73
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: doris-be-storage
local:
path: /data/doris/be/storage
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- 172.16.43.73
执行如下构建命令构建
kubectl apply -f doris-fe-st.yaml
kubectl apply -f doris-fe-pv.yaml
kubectl apply -f doris-be-st.yaml
kubectl apply -f doris-be-pv.yaml
kubectl apply -f operator.yaml -ndoris
kubectl apply -f doriscluster.yaml -ndoris
查看pvc确定是否绑定成功
查看pod 是否启动成功
执行show frontends命令查看fe的状态，结果如下
执行show backends命令查看be的状态，具体如下图
