之前做过elastic stack方案做过nginx日志实时采集系统,对elastic家的产品情有独钟。配置快捷,开发少,简单高效。于是决定继续采用elastic的部分产品作为工具。 一、解决方案:filebeat logstash mysql 脚本语言 二、准备工作(具体过程省略):
三、详细过程: #配置filebeate的输入 # ============================== Filebeat inputs =============================== #这里开启Filebeat的输入配置 filebeat.inputs: #这里指定输入类型 - type: log enabled: true paths: # 这里指定日志文件的路径 - /Users/zhang****/Downloads/test_log/logs/* #配置Filebeat的输出,这里输出到Logstash。(注意:输出到其他的地方要先注释关闭) # ------------------------------ Logstash Output ------------------------------- output.logstash: # The Logstash hosts hosts: ["localhost:5044"] 2.安装logstash-output-jdbc插件,下载mysql-connector-java #安装logstash-output-jdbc ./bin/logstash-plugin install logstash-output-jdbc #下载mysql-connector-java 可以在这里选择适合的版本:http:///artifact/mysql/mysql-connector-java 4.配置logstash的输入、过滤、输出 #这里配置输入源为beats(Filebeate属于Elastic的Beats系统产品的一种) input { beats { port => 5044 #客户端响应超时时间 client_inactivity_timeout => 60000 } } #配置日志过滤,这里采用grok组件进行日志格式匹配 #grok规则见https://github.com/logstash-plugins/logstash-patterns-core filter { grok { match => { "message" => "\[%{HTTPDATE:visited_time}\] %{IP:visited_ip} .* \"%{GREEDYDATA:referer_url}\" \"GET %{URI:target_url}\" .* \"%{GREEDYDATA:user_agent}\" \"%{GREEDYDATA :content-type}\"" } } #剔除掉Content-Type为icon的日志 if([content-type] == 'image/x-icon') { drop{} } } #配置结果的输出 output { #elasticsearch { # hosts => ["http://localhost:9200"] # index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{ YYYY.MM.dd}" # #user => "elastic" # #password => "changeme" #} #stdout{ # codec=>rubydebug{} #} #这里需要安装Logstash的logstash-jdbc-output插件,下载好mysql-connector-java jdbc { connection_string => "jdbc:mysql://localhost:3306/land_page?serverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=utf8&useSSL=false&allowMultiQueries=true" username => "root" password => "***" driver_jar_path => "/Users/zhang***/Downloads/logstash-7.11.2/mysql-connector-java/mysql-connector-java-8.0.16.jar" driver_class => "com.mysql.cj.jdbc.Driver" statement => [ "INSERT INTO land_page_log (visited_ip,visited_time,referer_url,target_url,content_type,user_agent) VALUES(?,?,?,?,?,?)", "[visited_ip]", "[visited_time]"," [referer_url]","[target_url]","[content-type]","[user_agent]" ] } 四、开启服务 #开启Logstash服务 cd logstash安装目录 ./bin/logstash -c ./config/logstash.config #开启Filebeat服务 cd filebeat安装目录 ./filebeat -e -c ./filebeat.yml 五、日志分析 |
|