Logtash-Forwarder 迁移到 Filebeat(19)

上文对Filebeat进行了啰嗦式的说明,下面将logstash-forwarder迁移到Filebeat上。

Filebeat带来下面的变化:

  • 对配置文件格式进行了重组,从JSON转换为YAML。
  • 对存储当前读取文件的状态的registry file被改变。
  • 命令行选项被删除并移到配置文件中。
  • 输出的配置选项从libbeat继承。
  • Logstash必须使用一种新的输入插件。

迁移策略

logstash

Logstash 需要安装一个新的输入插件 logstash-input-beats。在Logstash 1.5.x版本和2.x版本,该插件可以与 Logstash-Forwarder 所使用的插件logstash-input-lumberjack 并行加载。

如果你有大量的logstash-forwarder迁移到Filebeat,建议同时加载这两个插件,将其设置为不同的端口。当所有的迁移到Filebeat,即可删除Lumberjack插件。

Registry File

Registry File存储了Filbeat最后一次读的位置和状态。在Logstash-Forwarder被称为.logstash-fowarder(位于/var/lib/logstash-forwarder/.logstash-forwarder)。对于Filebeat需要将其重命名为 .filebeat。

迁移配置文件

files部分

Logstash-Forwarder 配置文件中的"files"部分转变成Filebeat配置文件中的"prospectors"部分。如:

logstash-forwarder配置文件

相当于Filebeat配置文件中的prospectors部分:

引入了一个新的选项document_type,如果没有类型被定义则默认为log。如果Filebeat被直接用来Elasticsearch索引,那么当在索引时document_type决定文档类型。

network部分

Filebeat可以于Logstash直接通信,此外,Filebeat还可以直接向elasticsearch插入日志条目。

logstash-forwarder配置文件:

Filebeat相当于:

当定义多台主机,类似于Logstash-forwarder行为,Filebeat默认随机选择一个主机建立连接。Filebeat可用设置为负载均衡。参见:https://www.elastic.co/guide/en/beats/libbeat/1.0.0-rc1/configuration.html#loadbalance

更改后的配置文件选项

配置文件的重构,有些选项被删除或改名。下面是更改的条目列表:

Config Option Action
deadTime deadTime was renamed to ignoreOlder. In case a file is not changed for ignoreOlder, the file handler will be closed. If the file is changed again after ignoreOlder has passed, it is be reopened.
netTimeout netTimeout was removed as it is replaced by the Timeout option in libbeat.
log-to-syslog andsyslog Both options were removed as logging is part of the libbeat config.

完整的实例

Logstash-Forwarder配置文件

Filebeat配置文件:

命令行选项

大部分 logstash-forwarder命令行被删除并移到配置文件中,重命名的命令行选项列表如下:

Command Line Option Config File Option Description
-config -c The config options was split up in two part. The base and required config is linked with -c. Additional config files can be linked as part of the config file. Note: Additional config files must be in a different directory than the main config file.
-config config_dir Path to directory with additional configuration files
-idle-timeout idle_timeout idle_timeout was moved to the config file and removed as flag.
-spool-size spool_size spool_size was moved to the config file and removed as flag.
-harvester-buff-size harvester_buffer_size harvester_buffer_size was moved to the config file and removed as flag. It can now be configured specific for each harvester.
-tail tail_files tail_files was moved to the config file and removed as flag. It can now be configured specific for each prospector.
-cpuProfileFile cpuProfileFile option was removed. The profiling options of libbeat can be used instead. For more details on profiling see https://github.com/elastic/libbeat/issues/122
-quiet The quiet option was removed. Libbeat is used for logging and the libbeat configuration options have to be used.

其它的一些改变


  1. 一个显着的变化是registry file名称取决于包封装类型:
    .tar.gz 和 .tgz  名称为.filebeat
    DEB 和 RPM 名称为/usr/lib/filebeat/registry
    Windows zip包 名称为c:\ProgramData\filebeat\registry
  2. TLS默认是关闭的
  3. 日志
    Filebeat使用libbeat日志,也可以记录到轮滚的文件,而不是系统日志。

发表评论

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

目前评论:3   其中:访客  3   博主  0

  1. 匿名 5

    :wink: :roll: :idea: :idea: :idea: 1121adasd啊飒飒的

    • 匿名 5

      @匿名 俺的是多少 :arrow: 阿大多数 :cool: :mad: :roll: :wink: :wink: :wink: :idea: :idea: :neutral: :cry: :mrgreen: :mrgreen: :?: :?: :?: :?: :?: :razz: :razz: :sad:

      • 匿名 5

        @匿名 @匿名 俺的是多少 :arrow: 阿大多数 :cool: :mad: :roll: :wink: :wink: :wink: :idea: :idea: :neutral: :cry: :mrgreen: :mrgreen: :?: :?: :?: :?: :?: :razz: :razz: :sad: