logstash输出到elasticsearch多索引

2017-01-13 15:05:27来源:csdn作者:wangyangzhizhou人点击

目标:将json格式的两类日志输出到elasticsearch两类索引

1. 安装logstash。

2. 编写logstash处理配置文件,创建一个test.conf文件,内容如下:

input {
file {
path => "/home/vagrant/logstash/logstash-2.2.2/dbpool-logs/dev/common-sql-*.log"
start_position => "beginning"
type => "sql"
codec => json {
charset => "UTF-8"
}
}
file {
path => "/home/vagrant/logstash/logstash-2.2.2/dbpool-logs/dev/common-transaction-*.log"
start_position => "beginning"
type => "transaction"
codec => json {
charset => "UTF-8"
}
}
}
output {
if "_grokparsefailure" in [tags] {
}else{
if [type] == "sql"{
elasticsearch {
hosts => ["http://192.168.33.10:9200"]
index => "common-sql-%{+YYYY.MM.dd}"
}
}
if [type] == "transaction"{
elasticsearch {
hosts => ["http://192.168.33.10:9200"]
index => "common-transaction-%{+YYYY.MM.dd}"
}
}
}
}

或者

input {
file {
path => "/home/vagrant/logstash/logstash-2.2.2/dbpool-logs/dev/common-sql-*.log"
start_position => "beginning"
type => "sql"
codec => json {
charset => "UTF-8"
}
}
file {
path => "/home/vagrant/logstash/logstash-2.2.2/dbpool-logs/dev/common-transaction-*.log"
start_position => "beginning"
type => "transaction"
codec => json {
charset => "UTF-8"
}
}
}
output {
if "_grokparsefailure" in [tags] {
}else{
if [type] == "sql"{
elasticsearch {
hosts => ["http://192.168.33.10:9200"]
index => "common-%{type}-%{+YYYY.MM.dd}"
}
}
}
}

指定输入日志的路径,按通配符匹配。分为两类:sql和transaction。


根据type分别输出到elasticsearch不同的索引。


3. 安装elasticsearch。

4. 启动elasticsearch,./bin/elasticsearch -d ,默认端口为9200。

5. 启动logstash开始处理,./bin/logstash -f conf/test.conf。

6. 完成。


最新文章

123

最新摄影

微信扫一扫

第七城市微信公众平台