Logstash 日志管理工具

默北 日志系统1687,12737字数 4975阅读16分35秒阅读模式

Logstash是一个开源的日志管理工具。

ELK 部署,参见本博客内容《ELK部署》,以及elasticsearch相关文章。文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

ELK系列文章:https://www.ttlsa.com/elk/文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

项目地址:http://logstash.net/文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

Logstash安装使用以下组件:文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

服务端:文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

  • fqdn: dev.kanbier.lan (should be resolvable!)
  • ip: 10.37.129.8

安装所需的软件

作者更喜欢使用RPM包来安装软件,要注意版本号,不要去追求时髦用最新的最伟大的,Elasticsearch的版本应该匹配Logstash的版本。文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

$ vi /etc/yum.repos.d/logstash.repo
[logstash-1.4]
name=logstash repository for 1.4.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

$ vi /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-1.0]
name=Elasticsearch repository for 1.0.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.0/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

$ vi /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

$ rpm -Uvh http://mirror.1000mbps.com/fedora-epel/6/i386/epel-release-6-8.noarch.rpm

$ yum -y install elasticsearch redis nginx logstash

启用Kibana

$ wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0.tar.gz
$ tar -xvzf kibana-3.0.0.tar.gz
$ mv kibana-3.0.0 /usr/share/kibana3

我们需要告诉Kibana在哪里可以找到elasticsearch。打开配置文件并修改elasticsearch参数:文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

$ vi /usr/share/kibana3/config.js

搜索“elasticsearch”参数,并对其进行修改以适应您的环境:文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

elasticsearch: "http://dev.kanbier.lan:9200",

您还可以修改default_route参数,默认打开logstash仪表板而不是Kibana欢迎页面:文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

default_route     : '/dashboard/file/logstash.json',

通过web界面访问:文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

$ wget https://raw.github.com/elasticsearch/kibana/master/sample/nginx.conf
$ mv nginx.conf /etc/nginx/conf.d/
$ vi /etc/nginx/conf.d/nginx.conf
server_name           dev.kanbier.lan;

nginx配置如下:文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

#
# Nginx proxy for Elasticsearch + Kibana
#
# In this setup, we are password protecting the saving of dashboards. You may
# wish to extend the password protection to all paths.
#
# Even though these paths are being called as the result of an ajax request, the
# browser will prompt for a username/password on the first request
#
# If you use this, you'll want to point config.js at http://FQDN:80/ instead of
# http://FQDN:9200
#
server {
  listen                *:80 ;

  server_name           kibana.myhost.org;
  access_log            /var/log/nginx/kibana.myhost.org.access.log;

  location / {
    root  /usr/share/kibana3;
    index  index.html  index.htm;
  }

  location ~ ^/_aliases$ {
    proxy_pass http://127.0.0.1:9200;
    proxy_read_timeout 90;
  }
  location ~ ^/.*/_aliases$ {
    proxy_pass http://127.0.0.1:9200;
    proxy_read_timeout 90;
  }
  location ~ ^/_nodes$ {
    proxy_pass http://127.0.0.1:9200;
    proxy_read_timeout 90;
  }
  location ~ ^/.*/_search$ {
    proxy_pass http://127.0.0.1:9200;
    proxy_read_timeout 90;
  }
  location ~ ^/.*/_mapping {
    proxy_pass http://127.0.0.1:9200;
    proxy_read_timeout 90;
  }

  # Password protected end points
  location ~ ^/kibana-int/dashboard/.*$ {
    proxy_pass http://127.0.0.1:9200;
    proxy_read_timeout 90;
    limit_except GET {
      proxy_pass http://127.0.0.1:9200;
      auth_basic "Restricted";
      auth_basic_user_file /etc/nginx/conf.d/kibana.myhost.org.htpasswd;
    }
  }
  location ~ ^/kibana-int/temp.*$ {
    proxy_pass http://127.0.0.1:9200;
    proxy_read_timeout 90;
    limit_except GET {
      proxy_pass http://127.0.0.1:9200;
      auth_basic "Restricted";
      auth_basic_user_file /etc/nginx/conf.d/kibana.myhost.org.htpasswd;
    }
  }
}

配置redis

$ vi /etc/redis.conf
bind 10.37.129.8

配置Logstash 

可以使用Logstash文档上的logstash-complex.conf文件,并不是很负责,包含:文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

  • 从 /var/log目录读取文件
  • 打开5544端口以启用直接接收远程系统日志消息
  • 告诉logstash,利用本身的elasticsearch而不是嵌入的
$ vi /etc/logstash/conf.d/logstash-complex.conf
input {
  file {
    type => "syslog"
 
    # Wildcards work, here <img src="http://www.denniskanbier.nl/blog/wp-includes/images/smilies/icon_smile.gif" alt=":)" class="wp-smiley"> 
    path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]
    sincedb_path => "/opt/logstash/sincedb-access"
  }
  redis {
    host => "10.37.129.8"
    type => "redis-input"
    data_type => "list"
    key => "logstash"
  }
  syslog {
    type => "syslog"
    port => "5544"
  }
}
 
filter {
  grok {
    type => "syslog"
    match => [ "message", "%{SYSLOGBASE2}" ]
    add_tag => [ "syslog", "grokked" ]
  }
}
 
output {
 elasticsearch { host => "dev.kanbier.lan" }
}

启动服务

$ service redis start; chkconfig redis on
$ service elasticsearch start; chkconfig --add elasticsearch; chkconfig elasticsearch on
$ service logstash start; chkconfig logstash on
$ service nginx start; chkconfig nginx on

对于rsyslog现在你可以将这些行添加到/ etc/ rsyslog.conf文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
$WorkDirectory /var/lib/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* @@10.37.129.8:5544
# ### end of the forwarding rule ###

如果有防火墙需要放开这些端口:文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

  • port 80 (for the web interface)
  • port 5544 (to receive remote syslog messages)
  • port 6379 (for the redis broker)
  • port 9200 (so the web interface can access elasticsearch)

Elasticsearch文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

译自:http://www.denniskanbier.nl/blog/logging/installing-logstash-on-rhel-and-centos-6/文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/ 文章源自运维生存时间-https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/

weinxin
我的微信
微信公众号
扫一扫关注运维生存时间公众号,获取最新技术文章~
默北
  • 本文由 发表于 23/07/2014 01:00:46
  • 转载请务必保留本文链接:https://www.ttlsa.com/log-system/installing-logstash-on-rhel-and-centos/
  • Elasticsearch
  • Kibana
  • logstash
  • redis
  • 日志管理
评论  16  访客  16
    • 222
      222 9

      其实我没怎么太明白。这个nginx的配置文件。

      • carson
        carson 9

        之前的地址失效了,来这个地址下载nginx.conf
        https://gist.githubusercontent.com/thisismitch/2205786838a6a5d61f55/raw/f91e06198a7c455925f6e3099e3ea7c186d0b263/nginx.conf

        • wk
          wk 9

          https://raw.github.com/elasticsearch/kibana/master/sample/nginx.conf 404错误,求conf文件

          • LA
            LA 9

            这个是针对/var/log下的系统日志吗

              • 默北
                默北

                @ LA 随你指定

                  • tony
                    tony 9

                    @ 默北 请问是在哪个文件中指定呢,我实验了一下,查询不到任何日志,谢谢

                      • TTLSA
                        TTLSA 9

                        @ tony logstash配置文件里

                  • ivon_lee
                    ivon_lee 9

                    学习了

                  • 来自外部的引用

                  评论已关闭!