- A+
所属分类：ELK
上文对Filebeat进行了啰嗦式的说明，下面将logstash-forwarder迁移到Filebeat上。
Filebeat带来下面的变化：
- 对配置文件格式进行了重组，从JSON转换为YAML。
- 对存储当前读取文件的状态的registry file被改变。
- 命令行选项被删除并移到配置文件中。
- 输出的配置选项从libbeat继承。
- Logstash必须使用一种新的输入插件。
迁移策略
logstash
Logstash 需要安装一个新的输入插件 logstash-input-beats。在Logstash 1.5.x版本和2.x版本，该插件可以与 Logstash-Forwarder 所使用的插件logstash-input-lumberjack 并行加载。
如果你有大量的logstash-forwarder迁移到Filebeat，建议同时加载这两个插件，将其设置为不同的端口。当所有的迁移到Filebeat，即可删除Lumberjack插件。
Registry File
Registry File存储了Filbeat最后一次读的位置和状态。在Logstash-Forwarder被称为.logstash-fowarder(位于/var/lib/logstash-forwarder/.logstash-forwarder)。对于Filebeat需要将其重命名为 .filebeat。
迁移配置文件
files部分
Logstash-Forwarder 配置文件中的"files"部分转变成Filebeat配置文件中的"prospectors"部分。如：
logstash-forwarder配置文件
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# The list of files configurations "files": [ # An array of hashes. Each hash tells what paths to watch and # what fields to annotate on events from those paths. { "paths": [ "/var/log/messages", "/var/log/*.log" ], # A dictionary of fields to annotate on each event. "fields": { "type": "syslog" } }, { # A path of "-" means stdin. "paths": [ "-" ], "fields": { "type": "stdin" } }, { "paths": [ "/var/log/apache/httpd-*.log" ], "fields": { "type": "apache" } } ] |
相当于Filebeat配置文件中的prospectors部分：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
filebeat: # List of prospectors to fetch data. prospectors: # Each - is a prospector. Below are the prospector specific configurations - paths: - /var/log/messages - "/var/log/*.log" - paths: - "-" input_type: stdin document_type: stdin - paths: - "/var/log/apache/httpd-*.log" document_type: apache |
引入了一个新的选项document_type，如果没有类型被定义则默认为log。如果Filebeat被直接用来Elasticsearch索引，那么当在索引时document_type决定文档类型。
network部分
Filebeat可以于Logstash直接通信，此外，Filebeat还可以直接向elasticsearch插入日志条目。
logstash-forwarder配置文件：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# The network section covers network configuration :) "network": { # A list of downstream servers listening for our messages. # logstash-forwarder will pick one at random and only switch if # the selected one appears to be dead or unresponsive "servers": [ "localhost:5043" ], # The path to your client ssl certificate (optional) "ssl certificate": "./logstash-forwarder.crt", # The path to your client ssl key (optional) "ssl key": "./logstash-forwarder.key", # The path to your trusted ssl CA file. This is used # to authenticate your downstream server. "ssl ca": "./logstash-forwarder.crt", # Network timeout in seconds. This is most important for # logstash-forwarder determining whether to stop waiting for an # acknowledgement from the downstream server. If an timeout is reached, # logstash-forwarder will assume the connection or server is bad and # will connect to a server chosen at random from the servers list. "timeout": 15 } |
Filebeat相当于：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
output: logstash: enabled: true # The list of downstream Logstash servers. hosts: - localhost:5043 tls: # The path to your SSL client certificate. certificate: ./logstash-forwarder.crt # The path to your SSL client certificate key. certificate_key: ./logstash-forwarder.key # The path to your trusted SSL CA file. This is used # to authenticate your downstream server. certificate_authorities: - ./logstash-forwarder.crt # Network timeout in seconds. timeout: 15 |
当定义多台主机，类似于Logstash-forwarder行为，Filebeat默认随机选择一个主机建立连接。Filebeat可用设置为负载均衡。参见：https://www.elastic.co/guide/en/beats/libbeat/1.0.0-rc1/configuration.html#loadbalance
更改后的配置文件选项
配置文件的重构，有些选项被删除或改名。下面是更改的条目列表：
| Config Option | Action |
|---|---|
deadTime |
deadTime was renamed to ignoreOlder. In case a file is not changed for ignoreOlder, the file handler will be closed. If the file is changed again after ignoreOlder has passed, it is be reopened. |
netTimeout |
netTimeout was removed as it is replaced by the Timeout option in libbeat. |
log-to-syslog andsyslog |
Both options were removed as logging is part of the libbeat config. |
完整的实例
Logstash-Forwarder配置文件
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
{ "files": [ { "paths": [ "/var/log/*.log" ], "fields": { "type": "syslog" } } ], "network": { "servers": [ "localhost:5043" ], } } |
Filebeat配置文件：
|
1 2 3 4 5 6 7 8 9 10 11 |
filebeat: prospectors: - paths: - "/var/log/*.log" fields: type: syslog output: elasticsearch: enabled: true hosts: ["http://localhost:5043"] |
命令行选项
大部分 logstash-forwarder命令行被删除并移到配置文件中，重命名的命令行选项列表如下：
| Command Line Option | Config File Option | Description |
|---|---|---|
-config |
-c |
The config options was split up in two part. The base and required config is linked with -c. Additional config files can be linked as part of the config file. Note: Additional config files must be in a different directory than the main config file. |
-config |
config_dir |
Path to directory with additional configuration files |
-idle-timeout |
idle_timeout |
idle_timeout was moved to the config file and removed as flag. |
-spool-size |
spool_size |
spool_size was moved to the config file and removed as flag. |
-harvester-buff-size |
harvester_buffer_size |
harvester_buffer_size was moved to the config file and removed as flag. It can now be configured specific for each harvester. |
-tail |
tail_files |
tail_files was moved to the config file and removed as flag. It can now be configured specific for each prospector. |
-cpuProfileFile |
cpuProfileFile option was removed. The profiling options of libbeat can be used instead. For more details on profiling see https://github.com/elastic/libbeat/issues/122 |
|
-quiet |
The quiet option was removed. Libbeat is used for logging and the libbeat configuration options have to be used. |
其它的一些改变
- 包
一个显着的变化是registry file名称取决于包封装类型：
.tar.gz 和 .tgz 名称为.filebeat
DEB 和 RPM 名称为/usr/lib/filebeat/registry
Windows zip包 名称为c:\ProgramData\filebeat\registry - TLS默认是关闭的
- 日志
Filebeat使用libbeat日志，也可以记录到轮滚的文件，而不是系统日志。
微信公众号
扫一扫关注运维生存时间公众号，获取最新技术文章~
07/09/2016 下午 6:36 沙发
:wink: :roll: :idea: :idea: :idea: 1121adasd啊飒飒的
07/09/2016 下午 6:37 1层
@匿名 俺的是多少 :arrow: 阿大多数 :cool: :mad: :roll: :wink: :wink: :wink: :idea: :idea: :neutral: :cry: :mrgreen: :mrgreen: :?: :?: :?: :?: :?: :razz: :razz: :sad:
24/02/2017 下午 4:01 2层
@匿名 @匿名 俺的是多少 :arrow: 阿大多数 :cool: :mad: :roll: :wink: :wink: :wink: :idea: :idea: :neutral: :cry: :mrgreen: :mrgreen: :?: :?: :?: :?: :?: :razz: :razz: :sad: