mac下nginx+lua+kafka实现日志统一收集汇总

参考:

一:场景描述

对于线上大流量服务或者需要上报日志的nginx服务，每天会产生大量的日志，这些日志非常有价值。可用于计数上报、用户行为分析、接口质量、性能监控等需求。但传统nginx记录日志的方式数据会散落在各自nginx上，而且大流量日志本身对磁盘也是一种冲击。 
我们需要把这部分nginx日志统一收集汇总起来,收集过程和结果需要满足如下需求: 
支持不同业务获取数据,如监控业务，数据分析统计业务，推荐业务等。 
数据实时性 
高性能保证

二:技术方案

得益于openresty和kafka的高性能，我们可以非常轻量高效的实现当前需求，架构如下: 
 
方案描述: 
1:线上请求打向nginx后，使用lua完成日志整理:如统一日志格式，过滤无效请求，分组等。 
2:根据不同业务的nginx日志,划分不同的topic。 
3:lua实现producter异步发送到kafka集群。 
4:对不同日志感兴趣的业务组实时消费获取日志数据。

三:相关技术 
openresty: http:///cn/ (版本:1.13.6.2)

kafka: http://kafka. (版本:2.0.0)

lua-resty-kafka: https://github.com/doujiang24/lua-resty-kafka

四:安装配置 
为了简单直接，我们采用单机形式配置部署，集群情况类似。 
1)openresty

brew install openresty

参考:http:///cn/installation.html

2)安装lua-resty-kafka

wget https://github.com/doujiang24/lua-resty-kafka/archive/master.zip 
unzip lua-resty-kafka-master.zip -d /opt/nginx/

#拷贝lua-resty-kafka到openresty 
mkdir /usr/local/Cellar/openresty/1.13.6.2/lualib/kafka/

cp -rf /opt/nginx/lua-resty-kafka-master/lib/resty /usr/local/Cellar/openresty/1.13.6.2/lualib/kafka/

4):安装单机kafka 

brew install kafka
按照提示装java环境,然后自动安装zookeeper依赖

# 启动kafka服务 

zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties

# 查看所有topic

/usr/local/Cellar/kafka/2.0.0/bin/kafka-topics --zookeeper localhost:2181 --list

# 创建测试topic 

/usr/local/Cellar/kafka/2.0.0/bin/kafka-topics --zookeeper localhost:2181 --create --topic test --replication-factor 1 --partitions 1

# 产生数据到topic

/usr/local/Cellar/kafka/2.0.0/bin/kafka-console-producer --broker-list localhost:9092 --topic test

命令行输入测试数据

>aaa

>bbbb

# 查看测试topic数据

/usr/local/Cellar/kafka/2.0.0/bin/kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning

编辑/usr/local/etc/openresty/nginx.conf 实现kafka记录nginx日志功能，源码如下: 
http{

#其他不变,添加如下代码

lua_package_path "/usr/local/Cellar/openresty/1.13.6.2/lualib/kafka/?.lua;;"; #引入lua-resty-kafka

server {

        listen 8080;

        root /data1/XXXX;

        index  index.html;

        server_name test.com;

        access_log /data0/www/logs/test.com-access_log;

        error_log /data0/www/logs/test.com-error_log;

        lua_code_cache off; #关闭lua代码缓存,可以实时调试代码

        location / {

                default_type text/html;

                content_by_lua 'ngx.say(package.path)';

        }

        location ^~ /test/{

                default_type text/html;

                content_by_lua_file $document_root/test/index.lua;

        }

}

}

2 index.lua

--ngx.say([[{"status":{"code":100000,"msg":"SUCCESS"},"result":{"version":"1.0.01"}}]])

--ngx.say(package.path)

--引入 kafka 生产者 类库

local producer = require("resty.kafka.producer")

--引入json 解析类库

local cjson = require("cjson")

--构造kafka 集群节点 broker

local broker_list = {

    { host = "127.0.0.1", port = 9092},

--  { host = "192.168.0.17", port = 9092},

--  { host = "192.168.0.18", port = 9092}

}

--定义上报对象

local log_obj = {}

--自定义模块名称

log_obj["request_module"] = "product_detail_info"

--获取请求头信息

log_obj["headers"] = ngx.req.get_headers()

--获取请求uri 参数

log_obj["uri_args"] = ngx.req.get_uri_args()

--获取请求body

log_obj["body"] = ngx.req.read_body()

--获取请求的http协议版本

log_obj["http_version"] = ngx.req.http_version()

--获取请求方法

log_obj["method"] = ngx.req.get_method()

--获取未解析的请求头字符串

log_obj["raw_reader"] = ngx.req.raw_header()

--获取解析的请求body体内容字符串

log_obj["body_data"] = ngx.req.get_body_data()

--上报对象json 字符串编码

local message = cjson.encode(log_obj)

local uri_args = ngx.req.get_uri_args()

local product_id = uri_args["productId"]

--创建kafka producer 连接对象，producer_type = "async" 异步

local async_producer = producer:new(broker_list, {producer_type = "async"})

--请求上报kafka，kafka 生产者发送数据，async_prodecer:send（a，b，c），a : 主题名称，b：分区（保证相同id，全部到相同的kafka node 去，并且顺序一致），c：消息（上报数据）

local ok, err = async_producer:send("test", product_id, message)

--上报异常处理

if not ok then

   ngx.log(ngx.ERR, "kafka send err:", err)

   return

end

六:检测&运行

检测配置,只检测nginx配置是否正确，lua错误日志在nginx的error.log文件中 
./nginx -t /opt/openresty/nginx/conf/nginx.conf 
# 启动 
./nginx -c /opt/openresty/nginx/conf/nginx.conf 
# 重启 
./nginx -s reload

七:测试

1:使用任意http请求发送给当前nginx，如: 
引用

http://10.10.78.52/m/personal/AC8E3BC7-6130-447B-A9D6-DF11CB74C3EF/rc/v1?passport=[e-mail]&page=2&size=10

2:查看upstream代理是否工作正常 
3:查看kafka 日志对应的topic是否产生消息日志，如下: 

效果监测: 

 
4:ab压力测试 
引用

#单nginx+upstream测试: 
ab -n 10000 -c 100 -k http://10.10.34.15/m/personal/AC8E3BC7-6130-447B-A9D6-DF11CB74C3EF/rc/v1?passport=[e-mail]&page=2&size=10

#结果 
Server Software: nginx 
Server Hostname: 10.10.34.15 
Server Port: 80 
Document Path: /m/personal/AC8E3BC7-6130-447B-A9D6-DF11CB74C3EF/rc/v1?passport=[e-mail] 
Document Length: 13810 bytes 
Concurrency Level: 100 
Time taken for tests: 2.148996 seconds 
Complete requests: 10000 
Failed requests: 9982 
(Connect: 0, Length: 9982, Exceptions: 0) 
Write errors: 0 
Keep-Alive requests: 0 
Total transferred: 227090611 bytes 
HTML transferred: 225500642 bytes 
Requests per second: 4653.34 [#/sec] (mean) 
Time per request: 21.490 [ms] (mean) 
Time per request: 0.215 [ms] (mean, across all concurrent requests) 
Transfer rate: 103196.10 [Kbytes/sec] received 
Connection Times (ms) 
min mean[+/-sd] median max 
Connect: 0 0 0.1 0 2 
Processing: 5 20 23.6 16 701 
Waiting: 4 17 20.8 13 686 
Total: 5 20 23.6 16 701 
Percentage of the requests served within a certain time (ms) 
50% 16 
66% 20 
75% 22 
80% 25 
90% 33 
95% 41 
98% 48 
99% 69 
100% 701 (longest request)

引用

#单nginx+upstream+log_lua_kafka接入测试: 
ab -n 10000 -c 100 -k http://10.10.78.52/m/personal/AC8E3BC7-6130-447B-A9D6-DF11CB74C3EF/rc/v1?passport=[e-mail]&page=2&size=10

#结果 
Server Software: openresty/1.9.7.4 
Server Hostname: 10.10.78.52 
Server Port: 80 
Document Path: /m/personal/AC8E3BC7-6130-447B-A9D6-DF11CB74C3EF/rc/v1?passport=[e-mail] 
Document Length: 34396 bytes 
Concurrency Level: 100 
Time taken for tests: 2.234785 seconds 
Complete requests: 10000 
Failed requests: 9981 
(Connect: 0, Length: 9981, Exceptions: 0) 
Write errors: 0 
Keep-Alive requests: 0 
Total transferred: 229781343 bytes 
HTML transferred: 228071374 bytes 
Requests per second: 4474.70 [#/sec] (mean) 
Time per request: 22.348 [ms] (mean) 
Time per request: 0.223 [ms] (mean, across all concurrent requests) 
Transfer rate: 100410.10 [Kbytes/sec] received 
Connection Times (ms) 
min mean[+/-sd] median max 
Connect: 0 0 0.2 0 3 
Processing: 6 20 27.6 17 1504 
Waiting: 5 15 12.0 14 237 
Total: 6 20 27.6 17 1504 
Percentage of the requests served within a certain time (ms) 
50% 17 
66% 19 
75% 21 
80% 23 
90% 28 
95% 34 
98% 46 
99% 67 
100% 1004 (longest request)