Elasticsearch知识汇总之ElasticSearch部署

五 ElasticSearch部署

部署Elasticsearch，可以在任何 Linux、MacOS 或 Windows 机器上运行 Elasticsearch。在Docker 容器中运行 Elasticsearch 。使用Elastic Cloud on Kubernetes 设置和管理 Elasticsearch、Kibana、Elastic Agent 以及 Kubernetes 上的 Elastic Stack 。

Elasticsearch 安装包，以下包格式提供：

Zip在 Windows 上安装 Elasticsearch。

tar.gz可用于安装在任何 Linux 发行版和 MacOS 上。

Deb软件包适用于 Debian、Ubuntu 和其他基于 Debian 的系统。Debian 软件包可以从 Elasticsearch 网站或Debian 存储库下载。

rpm软件包适合安装在 Red Hat、Centos、SLES、OpenSuSE 和其他基于 RPM 的系统上。RPM 可以从 Elasticsearch 网站或我们的 RPM 存储库下载。

docker映像可用于将 Elasticsearch 作为 Docker 容器运行。它们可以从 Elastic Docker Registry 下载。

5.1 裸金属部署

本部署方案采用三台服务器部署master节点和data节点

服务器信息表

IP地址	配置	系统版本	角色
1xx.1xx.1xx.60	4C 8G 100G硬盘（SSD）	CentOS 7.9 64位	Master/data
1xx.1xx.1xx.61	4C 8G 100G硬盘（SSD）	CentOS 7.9 64位	Master/data
1xx.1xx.1xx.62	4C 8G 100G硬盘（SSD）	CentOS 7.9 64位	Master/data

修改系统参数

Linux默认配置的File descriptors（文件描述符）不能够满足elasticsearch高吞吐量的要求

vi /etc/security/limits.conf

# 在最后加入，修改完成后，重启系统生效。

* soft nofile 131072

* hard nofile 131072

部署流程

所有节点

→ 安装jdk环境

jdk1.8安装

将jdk-8u271-linux-x64.tar.gz上传到服务器，并解压

tar -zvxf jdk-8u271-linux-x64.tar.gz -C

mv jdk1.8.0_271 /usr/local/jdk

在 /etc/profile 添加以下环境变量

export JAVA_HOME=/usr/local/jdk

export CLASSPATH=.:%JAVA_HOME/lib/dt.jar:%JAVA_HOME/lib/tools.jar

export PATH=$PATH:$JAVA_HOME/bin

如下图:

重新加载环境变量

source /etc/profile

→ 创建目录(所有节点)

useradd elasticsearch

mkdir -pv /home/esdata

mkdir -pv /home/logs

chown -R elasticsearch.elasticsearch /home/esdata /home/logs

chown -R elasticsearch.elasticsearch /usr/local/elasticsearch/

（2）elasticsearch安装

将elasticsearch-7.9.0-linux-x86_64.tar.gz安装包上传到服务器并解压

tar xf elasticsearch-7.9.0-linux-x86_64.tar.gz

mv elasticsearch-7.9.0-linux /usr/local/elasticsearch

添加elasticsearch用户并根据系统配置修改elasticsearch配置文件

修改es配置文件

编辑 elasticsearch.yml

cluster.name: logdata

node.name:elkmaster
node.roles: [data,master]

path.data: /home/esdata

path.logs: /home/eslogs

bootstrap.memory_lock: true

network.host: 1xx.1xx.1xx.60

http.port: 9200

transport.tcp.port: 9300

http.cors.enabled: true

http.cors.allow-origin: "*"

编辑 jvm.options

-Xms4g

-Xmx4g

备注: 一般设置为服务器内存的50%-60%

启动es服务

su - elasticsearch -c "/usr/local/elasticsearch/bin/elasticsearch &"

kibana安装（非必要组件）

将kibana-7.9.0.tar.gz安装包上传到服务器并解压

tar xf kibana-7.9.0.tar.gz

mv kibana-7.9.0 /usr/local/kibana

修改kibana.yml 连接es地址

server.port: 5601

server.host: "1xx.1xx.1xx.60"

elasticsearch.hosts: ["http://1xx.1xx.1xx.60:9200","http://1xx.1xx.1xx.61:9200","http://1xx.1xx.1xx.62:9200"]

启动kibana

./kibana --allow-root &

开放端口（若防火墙开启则需要执行以下命令，若防火墙关闭则不需要执行）

firewall-cmd --add-port=9200/tcp --permanent

firewall-cmd --add-port=9300/tcp --permanent

firewall-cmd --reload

firewall-cmd --list-all

→ 检查是否启动成功

jps查看或者直接查看elasticsearch进程

5.2 Docker-Compose部署ElasticSearch+Kibana

部署说明

用docker-compose快速部署es集群+kibana，这个集群是带安全检查的（自签证书+账号密码）

找个干净目录，新建名为.env的文件，内容如下，

# elastic账号的密码 (至少六个字符)

ELASTIC_PASSWORD=************

# kibana_system账号的密码 (至少六个字符)，该账号仅用于一些kibana的内部设置，不能用来查询es

KIBANA_PASSWORD=************

# es和kibana的版本

STACK_VERSION=8.2.2

# 集群名字

CLUSTER_NAME=docker-cluster

# x-pack安全设置，这里选择basic，基础设置，如果选择了trail，则会在30天后到期

LICENSE=basic

#LICENSE=trial

# es映射到宿主机的的端口

ES_PORT=9200

# kibana映射到宿主机的的端口

KIBANA_PORT=5601

# es容器的内存大小，请根据自己硬件情况调整在此验证给与1G配置

MEM_LIMIT=1073741824

# 命名空间，会体现在容器名的前缀上

COMPOSE_PROJECT_NAME=demo

然后是docker-compose.yaml文件，这里面会用到刚才创建的.env文件，一共创建了五个容器：启动操作、三个es组成集群，一个kibana

version: "2.2"

services:

setup:

image: elasticsearch:${STACK_VERSION}

volumes:

- certs:/usr/share/elasticsearch/config/certs

user: "0"

command: >

bash -c '

if [ x${ELASTIC_PASSWORD} == x ]; then

echo "Set the ELASTIC_PASSWORD environment variable in the .env file";

exit 1;

elif [ x${KIBANA_PASSWORD} == x ]; then

echo "Set the KIBANA_PASSWORD environment variable in the .env file";

exit 1;

fi;

if [ ! -f config/certs/ca.zip ]; then

echo "Creating CA";

bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;

unzip config/certs/ca.zip -d config/certs;

fi;

if [ ! -f config/certs/certs.zip ]; then

echo "Creating certs";

echo -ne \

"instances:\n"\

" - name: es01\n"\

" dns:\n"\

" - es01\n"\

" - localhost\n"\

" ip:\n"\

" - 127.0.0.1\n"\

" - name: es02\n"\

" dns:\n"\

" - es02\n"\

" - localhost\n"\

" ip:\n"\

" - 127.0.0.1\n"\

" - name: es03\n"\

" dns:\n"\

" - es03\n"\

" - localhost\n"\

" ip:\n"\

" - 127.0.0.1\n"\

> config/certs/instances.yml;

bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;

unzip config/certs/certs.zip -d config/certs;

fi;

echo "Setting file permissions"

chown -R root:root config/certs;

find . -type d -exec chmod 750 \{\} \;;

find . -type f -exec chmod 640 \{\} \;;

echo "Waiting for Elasticsearch availability";

until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;

echo "Setting kibana_system password";

until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;

echo "All done!";

healthcheck:

test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]

interval: 1s

timeout: 5s

retries: 120

es01:

depends_on:

setup:

condition: service_healthy

image: elasticsearch:${STACK_VERSION}

volumes:

- certs:/usr/share/elasticsearch/config/certs

- esdata01:/usr/share/elasticsearch/data

ports:

- ${ES_PORT}:9200

environment:

- node.name=es01

- cluster.name=${CLUSTER_NAME}

- cluster.initial_master_nodes=es01,es02,es03

- discovery.seed_hosts=es02,es03

- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}

- bootstrap.memory_lock=true

- xpack.security.enabled=true

- xpack.security.http.ssl.enabled=true

- xpack.security.http.ssl.key=certs/es01/es01.key

- xpack.security.http.ssl.certificate=certs/es01/es01.crt

- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt

- xpack.security.http.ssl.verification_mode=certificate

- xpack.security.transport.ssl.enabled=true

- xpack.security.transport.ssl.key=certs/es01/es01.key

- xpack.security.transport.ssl.certificate=certs/es01/es01.crt

- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt

- xpack.security.transport.ssl.verification_mode=certificate

- xpack.license.self_generated.type=${LICENSE}

mem_limit: ${MEM_LIMIT}

ulimits:

memlock:

soft: -1

hard: -1

healthcheck:

test:

[

"CMD-SHELL",

"curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",

]

interval: 10s

timeout: 10s

retries: 120

es02:

depends_on:

- es01

image: elasticsearch:${STACK_VERSION}

volumes:

- certs:/usr/share/elasticsearch/config/certs

- esdata02:/usr/share/elasticsearch/data

environment:

- node.name=es02

- cluster.name=${CLUSTER_NAME}

- cluster.initial_master_nodes=es01,es02,es03

- discovery.seed_hosts=es01,es03

- bootstrap.memory_lock=true

- xpack.security.enabled=true

- xpack.security.http.ssl.enabled=true

- xpack.security.http.ssl.key=certs/es02/es02.key

- xpack.security.http.ssl.certificate=certs/es02/es02.crt

- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt

- xpack.security.http.ssl.verification_mode=certificate

- xpack.security.transport.ssl.enabled=true

- xpack.security.transport.ssl.key=certs/es02/es02.key

- xpack.security.transport.ssl.certificate=certs/es02/es02.crt

- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt

- xpack.security.transport.ssl.verification_mode=certificate

- xpack.license.self_generated.type=${LICENSE}

mem_limit: ${MEM_LIMIT}

ulimits:

memlock:

soft: -1

hard: -1

healthcheck:

test:

[

"CMD-SHELL",

"curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",

]

interval: 10s

timeout: 10s

retries: 120

es03:

depends_on:

- es02

image: elasticsearch:${STACK_VERSION}

volumes:

- certs:/usr/share/elasticsearch/config/certs

- esdata03:/usr/share/elasticsearch/data

environment:

- node.name=es03

- cluster.name=${CLUSTER_NAME}

- cluster.initial_master_nodes=es01,es02,es03

- discovery.seed_hosts=es01,es02

- bootstrap.memory_lock=true

- xpack.security.enabled=true

- xpack.security.http.ssl.enabled=true

- xpack.security.http.ssl.key=certs/es03/es03.key

- xpack.security.http.ssl.certificate=certs/es03/es03.crt

- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt

- xpack.security.http.ssl.verification_mode=certificate

- xpack.security.transport.ssl.enabled=true

- xpack.security.transport.ssl.key=certs/es03/es03.key

- xpack.security.transport.ssl.certificate=certs/es03/es03.crt

- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt

- xpack.security.transport.ssl.verification_mode=certificate

- xpack.license.self_generated.type=${LICENSE}

mem_limit: ${MEM_LIMIT}

ulimits:

memlock:

soft: -1

hard: -1

healthcheck:

test:

[

"CMD-SHELL",

"curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",

]

interval: 10s

timeout: 10s

retries: 120

kibana:

depends_on:

es01:

condition: service_healthy

es02:

condition: service_healthy

es03:

condition: service_healthy

image: kibana:${STACK_VERSION}

volumes:

- certs:/usr/share/kibana/config/certs

- kibanadata:/usr/share/kibana/data

ports:

- ${KIBANA_PORT}:5601

environment:

- SERVERNAME=kibana

- ELASTICSEARCH_HOSTS=https://es01:9200

- ELASTICSEARCH_USERNAME=kibana_system

- ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}

- ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt

mem_limit: ${MEM_LIMIT}

healthcheck:

test:

[

"CMD-SHELL",

"curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",

]

interval: 10s

timeout: 10s

retries: 120

volumes:

certs:

driver: local

esdata01:

driver: local

esdata02:

driver: local

esdata03:

driver: local

kibanadata:

driver: local

注意：.env和docker-compose.yaml两个文件在同一目录下

启动应用

在docker-compose.yaml文件所在目录，执行命令docker-compose up -d启动所有容器

[root@node212 es]# docker-compose up -d

Creating network "demo_default" with the default driver

Creating volume "demo_certs" with local driver

Creating volume "demo_esdata01" with local driver

Creating volume "demo_esdata02" with local driver

Creating volume "demo_esdata03" with local driver

Creating volume "demo_kibanadata" with local driver

Pulling setup (elasticsearch:8.2.2)...

8.2.2: Pulling from library/elasticsearch

d5fd17ec1767: Pull complete

960bdea67557: Pull complete

87e8a9ab5eb5: Pull complete

d1a41a1f6148: Pull complete

2f30a84c2b73: Pull complete

2c111419937d: Pull complete

a098105ec516: Pull complete

4c72f9050453: Pull complete

77b3d5560f6a: Pull complete

Digest: sha256:8c666cb1e76650306655b67644a01663f9c7a5422b2c51dd570524267f11ce3d

Status: Downloaded newer image for elasticsearch:8.2.2

Pulling kibana (kibana:8.2.2)...

8.2.2: Pulling from library/kibana

d5fd17ec1767: Already exists

0e13695e6282: Pull complete

f4c86adffcb8: Pull complete

37df8a7a2f1c: Pull complete

605b30158b0c: Pull complete

4f4fb700ef54: Pull complete

8789a463d8bc: Pull complete

6c1b4670a98a: Pull complete

787921eb6497: Pull complete

7833e8f6b5e0: Pull complete

60937e7413ca: Pull complete

a04fb33dd003: Pull complete

5fcdf8cb4a0b: Pull complete

929af379dbc3: Pull complete

Digest: sha256:cf34801f36a2e79c834b3cdeb0a3463ff34b8d8588c3ccdd47212c4e0753f8a5

Status: Downloaded newer image for kibana:8.2.2

Creating demo_setup_1 ... done

Creating demo_es01_1 ... done

Creating demo_es02_1 ... done

Creating demo_es03_1 ... done

Creating demo_kibana_1 ... done

查看容器状态，负责启动的demo_setup_1已退出，其他的正常运行

最终部署成功示例如下：

查看demo_setup_1的日志，提示启动顺利

[root@node212 ~]# docker logs demo_setup_1

Setting file permissions

Waiting for Elasticsearch availability

Setting kibana_system password

All done!

如果要使用curl命令向ES发请求，需要提前将crt文件从容器中复制出来

docker cp demo_es01_1:/usr/share/elasticsearch/config/certs/es01/es01.crt .

验证

现在来验证es集群和kibana能不能正常工作

浏览器访问https://localhost:9200/，注意是https，会看到以下警告页面

根据之前的配置账号elastic，浏览器显示如下，证明es成功响应了

如果chrome上安装了eshead插件，此时就能查看es集群情况了(注意内部的地址栏中，要用https，而非http)，如下图，一共三个节点，es02前面有五角星标志，表示其主节点的身份

目前看 es集群部署和运行都已经正常，再看kibana是否可用

访问http://localhost:5601/，账号elastic，密码*********

清理

如果要删除es，执行docker-compose down就会删除容器，但是，此命令不会删除数据，下次执行docker-compose up -d后，新的es集群中会出现刚才创建的test001索引，并且数据也在

这是因为docker-compose.yaml中使用了数据卷volume存储es集群的关键数据，这些输入被保存在宿主机的磁盘上

❯ docker volume ls

DRIVER VOLUME NAME

local demo_certs

local demo_esdata01

local demo_esdata02

local demo_esdata03

local demo_kibanadata

执行docker volume rm demo_certs demo_esdata01 demo_esdata02

demo_esdata03即可将它们彻底清除

以上就是快速部署es集群+kibana的整个过程了

5.3 Kubernetes部署ElasticSearch集群

由3个master节点和3个data节点组成, master采用statefulset，不进行数据持久化, data节点使用持久化存储卷采用NFS

5.3.1 部署nfs

yum install nfs-utils rpcbind

创建共享目录 cat /etc/exports

/root/es/data1 *(rw,sync,no_subtree_check,no_root_squash)

/root/es/data2 *(rw,sync,no_subtree_check,no_root_squash)

/root/es/data3 *(rw,sync,no_subtree_check,no_root_squash)

启动nfs、rpcbing

systemctl restart nfs

systemctl restart rpcbind

在其他工作节点安装nfs-utils并测试挂载

yum install nfs-utils

mount -t nfs1xx.1xx.0.4:/nfs/data /mnt

查看挂载信息

mount

5.3.2 创建PV

[root@test-1 es]# cat pv.yaml

apiVersion: v1

kind: PersistentVolume

metadata:

name: es-store-0

spec:

accessModes:

- ReadWriteOnce

volumeMode: Filesystem

capacity:

storage: 1Gi

mountOptions:

- nolock

nfs:

path: /root/es/data1

server: 1xx.1xx.0.4

---

apiVersion: v1

kind: PersistentVolume

metadata:

name: es-store-1

spec:

accessModes:

- ReadWriteOnce

volumeMode: Filesystem

capacity:

storage: 1Gi

mountOptions:

- nolock

nfs:

path: /root/es/data2

server: 1xx.1xx.0.4

---

apiVersion: v1

kind: PersistentVolume

metadata:

name: es-store-2

spec:

accessModes:

- ReadWriteOnce

volumeMode: Filesystem

capacity:

storage: 1Gi

mountOptions:

- nolock

nfs:

path: /root/es/data3

server: 1xx.1xx.0.4

kubectl apply -f svc.yaml

5.3.3 创建svc

[root@test-1 es]# cat svc.yaml

apiVersion: v1

kind: Service

metadata:

name: es-svc

namespace: test

labels:

app: es-svc

spec:

selector:

app: es

ports:

- name: http

port: 9200 # 需要暴露的集群端口(service暴露的)

protocol: TCP

targetPort: 9200 # 容器的端口(后端容器提供服务的端口）

nodePort: 30200 # type=NodePort时有效 : 映射到物理机的端口 (范围30000-32767)

type: NodePort

# clusterIP: ""

---

apiVersion: v1

kind: Service

metadata:

name: es-master-svc

namespace: test

labels:

app: es-master-svc

spec:

selector:

app: es

role: master

publishNotReadyAddresses: true

ports:

- protocol: TCP

name: transport

port: 9300 # 需要暴露的集群端口(service暴露的)

# targetPort: 9300 # 容器的端口(后端容器提供服务的端口）

# nodePort: 30300 # type=NodePort时有效 : 映射到物理机的端口 (范围30000-32767)

type: ClusterIP

clusterIP: None

kubectl apply -f svc.yaml

5.3.4 创建es-master

[root@test-1 es]# cat es-master.yaml

apiVersion: apps/v1

kind: StatefulSet

metadata:

labels:

app: es-master

name: es-master

namespace: test

spec:

replicas: 3

serviceName: es-master-svc

selector:

matchLabels:

app: es

role: master

template:

metadata:

labels:

app: es

role: master

spec:

initContainers:

- name: init-sysctl

image: busybox

command:

- sysctl

- -w

- vm.max_map_count=262144

securityContext:

privileged: true

containers:

- name: es-master

image: elasticsearch:7.16.2

imagePullPolicy: Never

ports:

- containerPort: 9200

protocol: TCP

name: http

- containerPort: 9300

protocol: TCP

name: transport

env:

- name: "node.name"

valueFrom:

fieldRef:

fieldPath: "metadata.name"

- name: "cluster.name"

value: "es-cluster"

- name: "cluster.remote.connect"

value: "false"

- name: "node.master"

value: "true"

- name: "node.data"

value: "false"

- name: "node.ingest"

value: "false"

- name: "network.host"

value: "0.0.0.0"

- name: "path.data"

value: "/usr/share/elasticsearch/data"

- name: "path.logs"

value: "/usr/share/elasticsearch/logs"

- name: "bootstrap.memory_lock"

value: "false"

- name: "http.compression"

value: "true"

- name: "http.cors.enabled"

value: "true"

- name: "http.cors.allow-origin"

value: "*"

- name: "cluster.initial_master_nodes"

value: "es-master-0,es-master-1,es-master-2"

- name: "discovery.seed_hosts"

value: "es-master-svc"

- name: "xpack.ml.enabled"

value: "false"

- name: "ES_JAVA_OPTS"

value: "-Xms50m -Xmx50m"

resources:

requests:

cpu: 100m

limits:

cpu: 500m

livenessProbe:

tcpSocket:

port: transport

initialDelaySeconds: 60

periodSeconds: 10

kubectl apply -f es-master.yaml

测试：（进入其中一个pod，查看状态）

5.3.5 创建es-data

[root@test-1 es]# cat es-data.yaml

apiVersion: apps/v1

kind: StatefulSet

metadata:

labels:

app: es-data

name: es-data

namespace: test

spec:

selector:

matchLabels:

app: es

role: data

serviceName: es-svc

replicas: 1

template:

metadata:

labels:

app: es

role: data

spec:

initContainers:

- name: init-sysctl

image: busybox

command:

- sysctl

- -w

- vm.max_map_count=262144

securityContext:

privileged: true

containers:

- name: es-data

image: elasticsearch:7.16.2

imagePullPolicy: Never

env:

- name: node.name

valueFrom:

fieldRef:

fieldPath: metadata.name

- name: "node.max_local_storage_nodes"

value: "2"

- name: "cluster.name"

value: "es-cluster"

- name: "cluster.remote.connect"

value: "false"

- name: "node.master"

value: "false"

- name: "node.data"

value: "true"

- name: "node.ingest"

value: "false"

- name: "network.host"

value: "0.0.0.0"

- name: "path.data"

value: "/usr/share/elasticsearch/data"

- name: "path.logs"

value: "/usr/share/elasticsearch/logs"

- name: "bootstrap.memory_lock"

value: "false"

- name: "http.compression"

value: "true"

- name: "http.cors.enabled"

value: "true"

- name: "http.cors.allow-origin"

value: "*"

- name: "discovery.seed_hosts"

value: "es-master-svc"

- name: "xpack.ml.enabled"

value: "false"

- name: "ES_JAVA_OPTS"

value: "-Xms50m -Xmx50m"

resources:

requests:

cpu: 200m

limits:

cpu: 500m

ports:

- containerPort: 9200

protocol: TCP

name: http

- containerPort: 9300

protocol: TCP

name: transport

livenessProbe:

tcpSocket:

port: transport

initialDelaySeconds: 60

periodSeconds: 10

readinessProbe:

httpGet:

path: /_cluster/health

port: http

scheme: HTTP

initialDelaySeconds: 60

timeoutSeconds: 10

volumeMounts:

- mountPath: /usr/share/elasticsearch/data

name: storage

volumeClaimTemplates:

- metadata:

name: storage

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 1Gi

kubectl apply -f es-data.yaml