logstash分析日志

2017-01-13 15:05:11来源:csdn作者:wangyangzhizhou人点击

第七城市

待处理日志格式如下:

[totalCount: 298006556, count: 287347623, queryCount: 259027994, exeCount: 28319629, totalQPS: 10000.0, QPS: 1.0, queryQPS: 1.0, exeQPS: 0.0]
[totalCount: 298006556, count: 287347623, queryCount: 259027994, exeCount: 28319629, totalQPS: 20000.0, QPS: 1.0, queryQPS: 1.0, exeQPS: 0.0]
[totalCount: 298006556, count: 287347623, queryCount: 259027994, exeCount: 28319629, totalQPS: 40000.0, QPS: 1.0, queryQPS: 1.0, exeQPS: 0.0]
[totalCount: 298006556, count: 287347623, queryCount: 259027994, exeCount: 28319629, totalQPS: 100000.0, QPS: 1.0, queryQPS: 1.0, exeQPS: 0.0]

目标是得到每条记录的总QPS,即totalQPS

1、安装logstash。2、编写logstash处理配置文件,创建一个test.conf文件,内容如下:input {
file {
path => "/home/vagrant/logstash/logstash-2.2.2/dbpool-logs/dev/test.log"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "/[%{USERNAME}/: %{INT:totalCount}/, %{USERNAME}/: %{INT:count}/, %{USERNAME}/: %{INT:queryCount}/, %{USERNAME}/: %{INT:exeCount}/, %{USERNAME}/: %{BASE10NUM:totalQPS}/, %{USERNAME}/: %{BASE10NUM:QPS}/, %{USERNAME}/: %{BASE10NUM:queryQPS}/, %{USERNAME}/: %{BASE10NUM:exeQPS}/]" }
}
}
output {
if "_grokparsefailure" in [tags] {
}else{
file {
path => "/home/vagrant/logstash/logstash-2.2.2/dbpool-logs/dev/test-%{+yyyy-MM-dd}.log"
message_format => "总QPS为%{totalQPS}"
}
}
}

指定输入日志为/home/vagrant/logstash/logstash-2.2.2/dbpool-logs/dev/test.log。


filter为按照日志格式的匹配过滤规则。


最后以一定的格式输出到/home/vagrant/logstash/logstash-2.2.2/dbpool-logs/dev/test-%{+yyyy-MM-dd}.log。


3、启动logstash开始处理,./bin/logstash -f conf/test.conf4、处理后生成文件test-xxx.log,内容为:总QPS为10000.0
总QPS为20000.0
总QPS为40000.0
总QPS为100000.05、后面如果有日志继续写入test.log则会将解析结果追加到test-xxxx.log文件中。
第七城市

最新文章

123

最新摄影

微信扫一扫

第七城市微信公众平台