Logtash-Forwarder 迁移到 Filebeat(19)

上文对Filebeat进行了啰嗦式的说明，下面将logstash-forwarder迁移到Filebeat上。

Filebeat带来下面的变化：文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

• 对配置文件格式进行了重组，从JSON转换为YAML。
• 对存储当前读取文件的状态的registry file被改变。
• 命令行选项被删除并移到配置文件中。
• 输出的配置选项从libbeat继承。
• Logstash必须使用一种新的输入插件。

迁移策略

logstash

Logstash 需要安装一个新的输入插件 logstash-input-beats。在Logstash 1.5.x版本和2.x版本，该插件可以与 Logstash-Forwarder 所使用的插件logstash-input-lumberjack 并行加载。文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

如果你有大量的logstash-forwarder迁移到Filebeat，建议同时加载这两个插件，将其设置为不同的端口。当所有的迁移到Filebeat，即可删除Lumberjack插件。文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

Registry File

Registry File存储了Filbeat最后一次读的位置和状态。在Logstash-Forwarder被称为.logstash-fowarder(位于/var/lib/logstash-forwarder/.logstash-forwarder)。对于Filebeat需要将其重命名为 .filebeat。文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

迁移配置文件

files部分

Logstash-Forwarder 配置文件中的"files"部分转变成Filebeat配置文件中的"prospectors"部分。如：文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

logstash-forwarder配置文件文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

  # The list of files configurations
  "files": [
    # An array of hashes. Each hash tells what paths to watch and
    # what fields to annotate on events from those paths.
    {
      "paths": [
        "/var/log/messages",
        "/var/log/*.log"
      ],

      # A dictionary of fields to annotate on each event.
      "fields": { "type": "syslog" }
    }, {
      # A path of "-" means stdin.
      "paths": [ "-" ],
      "fields": { "type": "stdin" }
    }, {
      "paths": [
        "/var/log/apache/httpd-*.log"
      ],
      "fields": { "type": "apache" }
    }
  ]

相当于Filebeat配置文件中的prospectors部分：文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -
      paths:
        - /var/log/messages
        - "/var/log/*.log"
    -
      paths:
        - "-"
      input_type: stdin
      document_type: stdin 
    -
      paths:
        - "/var/log/apache/httpd-*.log"
      document_type: apache

引入了一个新的选项document_type，如果没有类型被定义则默认为log。如果Filebeat被直接用来Elasticsearch索引，那么当在索引时document_type决定文档类型。文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

network部分

Filebeat可以于Logstash直接通信，此外，Filebeat还可以直接向elasticsearch插入日志条目。文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

logstash-forwarder配置文件：文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

 # The network section covers network configuration :)
  "network": {
    # A list of downstream servers listening for our messages.
    # logstash-forwarder will pick one at random and only switch if
    # the selected one appears to be dead or unresponsive
    "servers": [ "localhost:5043" ],

    # The path to your client ssl certificate (optional)
    "ssl certificate": "./logstash-forwarder.crt",
    # The path to your client ssl key (optional)
    "ssl key": "./logstash-forwarder.key",

    # The path to your trusted ssl CA file. This is used
    # to authenticate your downstream server.
    "ssl ca": "./logstash-forwarder.crt",

    # Network timeout in seconds. This is most important for
    # logstash-forwarder determining whether to stop waiting for an
    # acknowledgement from the downstream server. If an timeout is reached,
    # logstash-forwarder will assume the connection or server is bad and
    # will connect to a server chosen at random from the servers list.
    "timeout": 15
  }

Filebeat相当于：文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

output:
  logstash:
    enabled: true

    # The list of downstream Logstash servers. 
    hosts:
      - localhost:5043

    tls: 
      # The path to your SSL client certificate.
      certificate: ./logstash-forwarder.crt

      # The path to your SSL client certificate key.
      certificate_key: ./logstash-forwarder.key

      # The path to your trusted SSL CA file. This is used
      # to authenticate your downstream server.
      certificate_authorities:
        - ./logstash-forwarder.crt

      # Network timeout in seconds.
      timeout: 15

当定义多台主机，类似于Logstash-forwarder行为，Filebeat默认随机选择一个主机建立连接。Filebeat可用设置为负载均衡。参见：https://www.elastic.co/guide/en/beats/libbeat/1.0.0-rc1/configuration.html#loadbalance文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

更改后的配置文件选项

配置文件的重构，有些选项被删除或改名。下面是更改的条目列表：文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

完整的实例

Logstash-Forwarder配置文件文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

{
  "files": [
    {
      "paths": [
        "/var/log/*.log"
      ],
      "fields": { "type": "syslog" }
    }
  ],
  "network": {
    "servers": [ "localhost:5043" ],
  }
}

Filebeat配置文件：文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

filebeat:
  prospectors:
    -
      paths:
        - "/var/log/*.log"
      fields:
        type: syslog
output:
  elasticsearch:
    enabled: true
    hosts: ["http://localhost:5043"]

命令行选项

大部分 logstash-forwarder命令行被删除并移到配置文件中，重命名的命令行选项列表如下：文章源自运维生存时间-https://www.ttlsa.com/elk/migration-logtash-forwarder-to-filebeat/

其它的一些改变

1. 包
   一个显着的变化是registry file名称取决于包封装类型：
   .tar.gz 和 .tgz  名称为.filebeat
   DEB 和 RPM 名称为/usr/lib/filebeat/registry
   Windows zip包 名称为c:\ProgramData\filebeat\registry
2. TLS默认是关闭的
3. 日志
   Filebeat使用libbeat日志，也可以记录到轮滚的文件，而不是系统日志。