ELK logstash升级到2.0以及logstash-forwarder迁移到Filebeat(21st)

默北 ELK319,8566字数 4498阅读14分59秒阅读模式

将从logstash1.5版本升级到2.1版本,以及将《ELK部署指南》中使用的logstash-forwarder转移到Filebeat上。

升级步骤

  1. 停止logstash以及发送到logstash的所有管道。
  2. 更新apt或yum源或者下载新版包。
  3. 安装新版的logstash。
  4. 测试logstash配置文件是否正确。
  5. 启动logstash以及第一步停止的管道。

升级logstash和elasticsearch到2.0

升级前请先看看版本的改变信息。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

下面是elasticsearch升级到2.0后,需要执行的:文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

Mapping改变:用户自定义的模板变化,因此在默认情况下,logstash升级将抛弃这些模板。即时没有一个自定义的模板,默认情况下logstash不会覆盖已存在的模板。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

已经有一个已知的问题就是使用GeoIP过滤器需要手动更新模板。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

注意,如果有自定义模板更改,务必保持和合并这些更改。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

查看已有的模板:文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

curl -XGET localhost:9200/_template/logstash

在logstash配置文件中添加下面的配置并重启:文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

output {
        elasticsearch {
                template_overwrite => true
        }
}

有点的字段:elasticsearch2.0不允许字段名含有.字符。一些插件包括logstash-filter-metrics和 logstash-filter-elapsed已经更新弥补这一更改。这些插件更新对于logstash2.0可用。要升级这些插件可执行下面命令:文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

bin/plugin update <plugin_name>

多行过滤器:如果要在logstash配置文件中使用多行过滤器并升级到2.0,将会得到一个错误。确保filter_workers明确设置为1。如果要改变该值需通过命令行参数更改,如下所示:文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

bin/logstash `-w 1`

ELK文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

实操

  1. 关闭logstash以及输入的管道。
# /etc/init.d/topbeat stop
# /etc/init.d/packetbeat stop
# /etc/init.d/filebeat stop
# /etc/init.d/logstash-forwarder stop
# /etc/init.d/logstash stop

这节会将logstash-forwarder迁移到Filebeat上了,后续不再启动logstash-forwarder了。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

2. 升级logstash,添加yum源参见前文。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

# yum update logstash

3. 检查配置文件文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

我的配置文件是以《ELK部署指南》中的配置文件为原型的。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/01-lumberjack-input.conf 
Configuration OK
# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/10-active.conf 
Configuration OK
# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/11-nginx.conf 
Configuration OK
# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/99-lumberjack-output.conf
Error: The setting `host` in plugin `elasticsearch` is obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:port' format. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask.

更改配置文件文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

# mv 01-lumberjack-input.conf 01-beats-input.conf
input {
  beats {
    port => 5044
    host => "10.1.19.18"
    type => "logs"
  }
}

这个是把有关logstash-forwarder使用的lumberjack删除了。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

# mv 99-lumberjack-output.conf 99-beats-output.conf  
# vim 99-beats-output.conf 
output {
 if "_grokparsefailure" in [tags] {
 file { path => "/var/log/logstash/grokparsefailure-%{[type]}-%{+YYYY.MM.dd}.log" }
 }

 elasticsearch {
   hosts => ["10.162.19.184:9200"]
   sniffing => true
   manage_template => false
   template_overwrite => true
   index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
   document_type => "%{[@metadata][type]}"
 }
 #stdout { codec =>rubydebug }
}

以上是output的定义。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

# vim 11-nginx.conf 
filter {
  if [type] == "nginx" {
    grok {
      match => { "message" => "%{IPORHOST:clientip} - %{NOTSPACE:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?: %{URIPROTO:proto}/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} (?:%{NUMBER:upstime}|-) %{NUMBER:reqtime} (?:%{NUMBER:size}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{QS:reqbody} %{WORD:scheme} (?:%{IPV4:upstream}(:%{POSINT:port})?|-)" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
    }
   geoip {
        source => "clientip"
        add_tag => [ "geoip" ]
        fields => ["country_name", "country_code2","region_name", "city_name", "real_region_name", "latitude", "longitude"]
        remove_field => [ "[geoip][longitude]", "[geoip][latitude]" ]
    }
    useragent {
        source => "agent"
        target => "browser"
    }
  }
}

对类型nginx的filter。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

logstash-forwarder的配置文件文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

# vi /etc/logstash-forwarder.conf 
  "files": [
    {
        "paths": [ "/data/logs/www.ttlsa.com/active/*.log" ],
        "fields": { "type": "active" }
    },
    {
        "paths": [ "/data/logs/www.ttlsa.com/nginx/*-access.log" ],
        "fields": { "type": "nginx" }
    } 
  ]

改成Filebeat的配置文件:文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

# vim /etc/filebeat/filebeat.yml
filebeat:
  prospectors:
    -
      paths:
        - /data/logs/www.ttlsa.com/nginx/*-access.log
      document_type: nginx   //隐射为type:nginx
    -
      paths:
        - /data/logs/www.ttlsa.com/active/*.log
      document_type: active

此处document_type选项控制输出type字段,用于elasticsearch输出以确定文档类型。对于以elasticsearch输出,该值用于设置输出文档的type字段。

4. 检查配置文件是否正确

# /opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/01-beats-input.conf
Configuration OK
# /opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/11-nginx.conf 
Configuration OK
# /opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/30-beats-output.conf 
Configuration OK

5. 启动服务

# /etc/init.d/topbeat stop
# /etc/init.d/packetbeat stop
# /etc/init.d/filebeat stop
# /etc/init.d/logstash stop

以上便是升级的过程,以及将logstash-forwarder迁移到Filebeat上了。

为了避免出现下面的问题:

sun/misc/URLClassPath.java:1003:in `getResource': java.lang.InternalError: java.io.FileNotFoundException: /alidata/server/java/jre/lib/ext/localedata.jar (Too many open files)

需更改下面的配置:

# vim /etc/sysconfig/logstash  
LS_OPEN_FILES=65535

Error: Your application used more memory than the safety cap of 500M.

Specify -J-Xmx####m to increase it (#### = cap size in MB).

Specify -w for full OutOfMemoryError stack trace

需更改下面的配置:

# vim /etc/sysconfig/logstash 
LS_HEAP_SIZE="1024m"
weinxin
我的微信
微信公众号
扫一扫关注运维生存时间公众号,获取最新技术文章~
默北
  • 本文由 发表于 27/12/2015 01:55:49
  • 转载请务必保留本文链接:https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/
  • ELK
  • Filebeat
  • logstash
  • logstash-forwarder
  • Packetbeat
  • Topbeat
评论  3  访客  3
    • 匿名
      匿名 9

      filebeat经常假死怎么办

      • 小白四个圈
        小白四个圈 9

        filebeat 这货不支持rhel5 centos5的系统。

      评论已关闭!