Scribe分布式日志系统

Scribe分布式日志系统

category=default //用来决定哪些信息由这个store来处理type=buffer //类型 Store::createStoretarget_write_size=20480 //在处理消息前允许多大的消息队列增长 单位是bytemax_write_interval=1 //在处理消息前多久允许消息队列增长间隔buffer_send_rate=2 //从secondary store多少次读取一个组消息发送到primary storeretry_interval=30 //写入到primary store失败后，多久时间再次发送到primary storeretry_interval_range=10 //重试间隔fs_type=std //目前只支持stdfile_path=/tmp/scribetest //存储目录base_filename=thisisoverwritten //文件名称，默认是category名称max_size=1000000 //文件轮转大小add_newlines=1 //每条日志是否一行rotate_period=daily //轮转周期，hourly，daily，never默认是never 创建新文件的频率rotate_hour=0 //0-23，默认是1 如果rotate_period=daily，几点轮转rotate_minute=10 //0-59，默认是15 如果rotate_period=daily/hourly，几点几分轮转remote_host //将日志发送到远程scribe服务器上remote_porttimeout //socket超时时间，单位msuse_conn_pool //是否使用连接池代替开放的多连接，默认是false

# vi scribe.confport=1463max_msg_per_second=2000000check_interval=3

category=defaulttype=buffertarget_write_size=20480max_write_interval=1buffer_send_rate=2retry_interval=30retry_interval_range=10

type=filefs_type=stdfile_path=/tmp/scribetestbase_filename=xuhhmax_size=1000000add_newlines=1# ../bin/scribed ./scribe.conf & //开启scribed服务# echo "test scribe" | ../bin/scribe_cat scribe_test //测试# cat /tmp/scribetest/scribe_test/scribe_test_current //查看日志信息# ../bin/scribe_ctrl counters //查看统计信息scribe_test:received good: 2

category=ignore*type=null

category=bucket_metype=buffertarget_write_size=20480max_write_interval=1buffer_send_rate=2retry_interval=30retry_interval_range=10

type=bucketnum_buckets=5bucket_subdir=bucketbucket_type=key_hashdelimiter=58

type=filefs_type=stdfile_path=/tmp/scribetestbase_filename=bucket_memax_size=10000

category=defaulttype=buffertarget_write_size=20480max_write_interval=1buffer_send_rate=2retry_interval=30retry_interval_range=10 type=filefs_type=stdfile_path=/tmp/scribetestbase_filename=thisisoverwrittenmax_size=1000000

Client端配置：# vi scribe-client.confport=1464max_msg_per_second=2000000check_interval=3

category=defaulttype=buffertarget_write_size=20480max_write_interval=1buffer_send_rate=1retry_interval=30retry_interval_range=10

type=networkremote_host=192.168.1.173remote_port=1463

[client]# echo "from scribe_client to scribe_central" | ../bin/scribe_cat -h localhost:1464 scribe_central[Thu Dec 1 14:30:37 2011] "Successfully sent messages to remote scribe server "[client]# echo "this message will be ignored" | ../bin/scribe_cat -h localhost:1464 ignore_me //将忽略掉[client]# echo "this message will be bucketed" | ../bin/scribe_cat -h localhost:1464 bucket_me[Thu Dec 1 15:04:21 2011] "[bucket_me] Creating new category store from model default"[Thu Dec 1 15:04:21 2011] "Opened connection to remote scribe server "[Thu Dec 1 15:04:21 2011] "[bucket_me] Opened file for writing"[Thu Dec 1 15:04:21 2011] "[bucket_me] Changing state from to "[Thu Dec 1 15:04:24 2011] "[bucket_me] read entries of bytes from file "[Thu Dec 1 15:04:28 2011] "Successfully sent messages to remote scribe server "[Thu Dec 1 15:04:28 2011] "[bucket_me] No more buffer files to send, switching to streaming mode"[Thu Dec 1 15:04:28 2011] "[bucket_me] Changing state from to "[central]# cat /tmp/scribetest/scribe_central/scribe_central_currentfrom scribe_client to scribe_central[central]# cat bucket000/bucket_me_currentthis message will be bucketed[central]# /home/xuhh/scribe-2.2/bin/scribe_ctrl countersbucket_me:received good: 1scribe_overall:received good: 5scribe_central:received good: 3scribe_overall:ignored: 1ignore_me:ignored: 1ignore_me:received good: 1[client]# /usr/local/scribe/bin/scribe_ctrl counters 1464bucket_me:received good: 1scribe_overall:received good: 4scribe_central:received good: 2ignore_me:received good: 1scribe_overall:sent: 4

配置PHP接口：生成PHP库：# cd /usr/local/scribe/# /usr/local/thrift-0.7.0/bin/thrift -o . -I /usr/local/thrift-0.7.0/fb303/share --gen php /usr/local/thrift-0.7.0/fb303/share/fb303/if/fb303.thrift# /usr/local/thrift-0.7.0/bin/thrift -o . -I /usr/local/thrift-0.7.0/fb303/share/ --gen php /data/src/software/pcting-scribe-2ee14d3/if/scribe.thrift# cp -r /data/src/software/thrift-0.7.0/lib/php/src includes# mkdir -p includes/packages/fb303# mkdir -p includes/packages/scribe# mv gen-php/fb303/FacebookService.php ./includes/packages/fb303/# mv gen-php/fb303/fb303_types.php ./includes/packages/fb303/# mv gen-php/scribe/scribe_types.php ./includes/packages/scribe/# mv gen-php/scribe/scribe.php ./includes/