大数据应用日志采集之Scribe 安装配置指南 1.概述Scribe是Facebook开源的日志收集系统,在Facebook内部已经得到大量的应用。它能从各种日志源收集日志,存储到一个中央存储系统上,便于进行集中统计分析处理。它为日志的”分布式收集,统一处理”提供了一个可扩展的,高容错的方案。scribe代码很简单,但是安装配置却很复杂,本文记录了作者实际的一次安装的过程,感觉真是不一般的琐碎,另外Scribe开源社区的版本已经是几年前的版本了,最新的维护信息一致没有看到,HDFS和Thrift的版本使用的都是比较旧的版本,考虑另开一个分支,升级一下Scribe,看到时候有没有时间了。 2.Scribe 安装Scribe 依赖于多个环境,有pthyon,boost,thirft,libevent,openssl等。在编译的时候还有一些编译错误需要进行修正,安装的和准备如下。 2.1 环境准备1.安装编译环境 #yum install gcc-c++ libevent libevent-devel automake autoconf m4 bison zlib zlib-devel bzip2 bzip2-devel flex pkgconfig python python-devel ruby ruby-devel mono-devel libxml2 libxml2-devel ant openssl-devel 2.安装boost环境:最低版本1.36 #wget http://nchc.dl./project/boost/boost/1.45.0/boost_1_45_0.tar.gz #tar zxvf boost_1_45_0.tar.gz #cd boost_1_45_0 #./bootstrap.sh #./bjam #./bjam --s HAVE_ICU=1 --prefix=/usr/local/boost --includedir=/usr/local/boost/include --libdir=/usr/local/boost/lib #./bjam install –prefix=/usr/local/boost #echo "/usr/local/boost/lib" >> /etc/ld.so.conf #echo /usr/local/boost/include >> /etc/ld.so.conf #ldconfig 3. 安装thrift环境 #wget https://dist./repos/dist/release/thrift/0.9.0/thrift-0.9.0.tar.gz #tar xzvf thrift-0.9.0.tar.gz #./configure --prefix=/usr/local/thrift --with-boost=/usr/local/boost --with-java=no --with-erlang=no --with-perl=no --with-php=no --with-ruby=no 备注:如果提示找不到libcrpty增加openssl路径 ./configure --prefix=/usr/local/thrift --with-boost=/usr/local/boost --with-java=no --with-erlang=no --with-perl=no --with-php=no --with- ruby=no CPPFLAGS="-I/usr/local/openssl/include" LDFLAGS="-ldl -L/usr/local/openssl/lib" 修改文件: Thrift/lib/ src/thrift/Thrift.cpp 增加头文件:#include<stdlib.h> 同时修改config.h文件,注释掉: /* Define to rpl_malloc if the replacement function should be used. */ //modby zhangzl++: //#define malloc rpl_malloc //-- … /* Define to rpl_realloc if the replacement function should be used. */ //modby zhangzl++: //#define realloc rpl_realloc //-- #make #make install # echo "/usr/local/thrift/lib" >> /etc/ld.so.conf #ldconfig #cd contrib. 4.安装fb303 #cd fb303 #./bootstrap.sh #./configure --prefix=/usr/local/thrift/fb303 --with-boost=/usr/local/boost/ --with-thriftpath=/usr/local/thrift #make //注意修改观察生成的Makefile关于头文件和库文件路径的信息是否正确,如果正确则可以直接编译。 #make install 4.安装scribe 在https://github.com/facebook/scribe下载zip包。 #unzip scribe-master.zip #cd scribe-master # export BOOST_ROOT=/usr/local/boost #export LD_LIBRARY_PATH=/usr/local/thrift/lib:/usr/lib:/usr/local/lib:/usr/local/boost/lib:/usr/local/thrift/fb303/lib #./bootstrap.sh --with-boost=/usr/local/boost --with-boost-filesystem=boost_filesystem 遇到问题: checking whether the Boost::System library is available… yes checking whether the Boost::Filesystem library is available… yes configure: error: Could not link against ! 解决办法,在configure 后加一个参数 --with-boost-filesystem=boost_filesystem
#./configure --prefix=/usr/local/scribe --with-boost=/usr/local/boost --with-thriftpath=/usr/local/thrift --with-fb303path=/usr/local/thrift/fb303 --with-boost-filesystem=boost_filesystem #make 遇到问题:
#tar zxvf libevent-2.0.18-stable.tar.gz 3.安装libevent-devel yum install libevent-devel error: there are no arguments to 'htons' that depend on a template parameter, so a declaration of 'htons' must be available 需要自己修改一下文件: vim /usr/local/thrift/include/thrift/protocol/TBinaryProtocol.tcc 在首行添加头文件引用: #include <arpa/inet.h> #echo /usr/local/libevent/include >> /etc/ld.so.conf #ldconfig #make install 2.2 测试#cp example/example1.conf /tmp #/usr/local/scribe/bin/scribed /tmp/example1.conf 进行测试。 如果失败,根据提示信息,进行相应修改。 我启动时出现下面的错误: ./scribed: error while loading shared libraries: libevent-2.0.so.5: cannot open shared object file: No such file or directory 解决办法是find这个库文件libevent-2.0.so.5,拷贝到/usr/lib64目录下。 启动成功后,出现下面的界面:
3.Scribe配置Scribe 的配置解析见http://my.oschina.net/guol/blog/110258 测试通过的server端和client端的配置文件如下:
port=1463 max_msg_per_second=100000000 max_queue_size=10000000 check_interval=1 new_thread_per_category=true <store> category=squid type=file file_path=/tmp/test/logcollect/ base_filename=21vianet__squid_access_all_ rotate_period=daily rotate_hour=0 rotate_minute=0 max_size=1610612736 add_newlines=0 </store>
port=1464 max_msg_per_second=2000000 max_queue_size=1000000 check_interval=1 new_thread_per_category=true
<store> category=default type=buffer target_write_size=20480 max_write_interval=1 buffer_send_rate=2 retry_interval=30 retry_interval_range=10
<primary> type=network #remote_host=218.93.205.106 remote_host=192.168.11.95 remote_port=1463 </primary>
<secondary> type=file fs_type=std write_meta=no file_path=/tmp/test/scribed base_filename=accesslog max_size=100000000 add_newlines=0 </secondary> </store> 注意: 在server端接收的日志,将以文件的方式来记录log,这个文件不能删除,它是个软连接。需要先copy过去,再调用touch命令将文件清空。 |
|