一、部署ES
1、创建网络下载镜像
docker network create elastic
docker pull elasticsearch:7.17.6
2、目录准备
mkdir /opt/ELK/elastic/{data,config} -p
chmod 777 /opt/ELK/elastic/data
cat >> /opt/ELK/elastic/config/elasticsearch.yml <<EOF
cluster.name: "ELK"
network.host: 0.0.0.0
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
EOF
3、启动单节点es
docker run -itd --name elasticsearch --network elastic -p 9200:9200 -p 9300:9300 -v /opt/ELK/elastic/data:/usr/share/elasticsearch/data -v /opt/ELK/elastic/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -e ES_JAVA_OPTS="-Xms512m -Xmx512m" -e "node.name=es1" -e "discovery.seed_hosts=es1" -e "cluster.initial_master_nodes=es1" -e "http.host=0.0.0.0" --privileged --restart=always elasticsearch:7.17.6
4、验证es正确性
http://IP:Port/_cat/health
含义说明：
- 前两项表示启动时间
- 第三项表示集群名称
- 第四项表示集群健康状态
- 后面标识ES节点，分片等数量
ES健康状态
- green：集群所有数据处在正常状态
- yellow：集群所有数据都可以访问，但一些数据的副本还没有分配
- red：集群部分数据不可访问
二、部署Logstash
logstash需要部署到所有需要收集日志的服务器上面，所以说使用docker方式部署会有一定的局限性，所以这里采用tar包部署
1、部署安装
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.17.0-linux-x86_64.tar.gz
tar zxf logstash-7.17.0-linux-x86_64.tar.gz
cd logstash-7.17.0/
2、测试可用性
./logstash -e 'input { stdin { } } output { stdout {} }'
我们输入的时候 会显示出来
解释：
@timestamp：标记时间发生的时间
host：事件发生的主机
@version：标记事件唯一类型
3、编写logstash配置文件 收集Nginx日志信息
cd /root/logstash-7.17.0/config
cp logstash-sample.conf nginxlog.conf
收集Nginx日志到终端 测试是否可以正常输出内容
input {
file {
path => "/var/log/nginx/*.log"
exclude => "error.log"
start_position => "beginning"
tags => ["web","nginx"]
type => "access"
add_field => {
"project" => "nginx-access-log"
"app" => "nginx"
}
}
}
filter {
}
output {
stdout{
}
}
如果可以正常输出Nginx日志到终端 表示配置没问题
接下来修改Nginx日志使用json格式
http {
....
log_format main '{"@timestamp":"$time_iso8601",'
'"@source":"$server_addr",'
'"hostname":"$hostname",'
'"remote_user":"$remote_user",'
'"ip":"$http_x_forwarded_for",'
'"client":"$remote_addr",'
'"request_method":"$request_method",'
'"scheme":"$scheme",'
'"domain":"$server_name",'
'"referer":"$http_referer",'
'"request":"$request_uri",'
'"requesturl":"$request",'
'"args":"$args",'
'"size":$body_bytes_sent,'
'"status": $status,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamaddr":"$upstream_addr",'
'"http_user_agent":"$http_user_agent",'
'"http_cookie":"$http_cookie",'
'"https":"$https"'
'}';
添加通过 message 字段过滤
cat nginxlog.conf
input {
file {
path => "/var/log/nginx/*.log"
exclude => "error.log"
start_position => "beginning"
tags => ["web","nginx"]
type => "access"
add_field => {
"project" => "nginx-access-log"
"app" => "nginx"
}
}
}
filter {
json {
source => "message"
skip_on_invalid_json => "true"
}
}
output {
stdout{
}
}
4、将Nginx日志输出到ES中
input {
file {
path => "/var/log/nginx/*.log"
exclude => "error.log"
start_position => "beginning"
tags => ["web","nginx"]
type => "access"
add_field => {
"project" => "nginx-access-log"
"app" => "nginx"
}
}
}
filter {
json {
source => "message"
skip_on_invalid_json => "true"
}
}
output {
elasticsearch {
hosts => ["10.0.24.5"]
index => "nginx-access-log-%{+YYYY.MM.dd}"
}
}
三、Kibana部署
1、部署
docker run -itd --name kibana --network elastic -p 5601:5601 -e "I18N_LOCALE=zh-CN" -e "ELASTICSEARCH_HOST=http://elasticsearch:9200" kibana:7.17.6
浏览器访问：http://IP:5601
2、创建Nginx索引
Discover > 创建
3、创建可视化
根据需求选择创建 饼状体、数据图等
根据索引筛选过滤信息 如：Nginx xxx -xxxx 时间段访问记录
