Fluentd

Fluentdのインストール

yumにtd-agentの設定を追加する

# vi /etc/yum.repo.d/td.repo

[treasuredata]
name=TreasureData
baseurl=http://packages.treasure-data.com/redhat/$basearch
gpgcheck=0

yumでtd-agentをインストールする

# yum install td-agent

Fluentd(td-agent)の起動・停止・再起動

# /etc/init.d/td-agent start
# /etc/init.d/td-agent stop
# /etc/init.d/td-agent restart

Shutting down td-agent:                                    [  OK  ]
Starting td-agent:                                         [  OK  ]

fluent-pluginをインストールする

fluent-plugin-file-alternative

/var/log/httpd/access_logと/var/log/httpd/error_logを別ファイルに出力する

# /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-file-alternative
# vi /etc/td-agent/td-agent.conf

<source>
  type forward
</source>
<match apache.access.**>
  type file_alternative
  path /var/log/td-agent/httpd/access.*.log
  time_slice_format %Y%m%d
  output_include_time false
  output_include_tag false
  output_data_type attr:message
  add_newline true
</match>
<match apache.error.**>
  type file_alternative
  path /var/log/td-agent/httpd/error.*.log
  time_slice_format %Y%m%d
  output_include_time false
  output_include_tag false
  output_data_type attr:message
  add_newline true
</match>

fluent-plugin-parser

/var/log/httpd/access_logと/var/log/httpd/error_logを別ファイルに出力する
ログのパースを行う

# /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-parser
# vi /etc/td-agent/td-agent.conf

<source>
  type forward
</source>

<match apache.access.**>
  type copy
  <store>
    type file_alternative
    path /var/log/td-agent/httpd/access.*.log
    time_slice_format %Y%m%d
    output_include_time false
    output_include_tag false
    output_data_type attr:message
    add_newline true
  </store>
  <store>
    type parser
    add_prefix parsed
    format /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<response>[^ ]*))?$/
    time_format %d/%b/%Y:%H:%M:%S %z
    key_name message
  </store>
</match>
<match parsed.apache.access.**>
  type file
  path /var/log/td-agent/httpd/parsed.apache.access
</match>

<match apache.error.**>
  type file_alternative
  path /var/log/td-agent/httpd/error.*.log
  time_slice_format %Y%m%d
  output_include_time false
  output_include_tag false
  output_data_type attr:message
  add_newline true
</match>