preload preload


Scribe Utilities

Lately, in my company I have been started working on the large scale distributed computing and storage technologies involving Hadoop, MapReduce, HBase, Scribe, Flume etc. Our work is like building the analytics platform utilizing and improving multiple open source technologies as mentioned earlier. A major component of this platform is Scribe.

For those who are not familiar with the Scribe, it is a log aggregation server handling a massive streams of data in realtime. It is developed and used in Facebook. You can find the it @ https://github.com/facebook/scribe/wiki.

Technically, Scribe accepts the plain text messages wrapped up in Thrift RPC format and forward it to multiple end points like disk, HDFS etc. So it is essential that the logs messages should be converted into Thrift format before you can be able to send to Scribe server. There are some already existing Scribe utilities which do the above like Log4j Scribe appender, Python Scribe client etc.

The limitation of above are that log4j ScribeAppender are limited to Java based application where Python clients are not ideal for very large dataset. And since we had our customers who were not on Java, we decided to write  Scribe utilities which can handle these large dataset realtime.

The result of that is the Scribe utilities collection written in C++ which we decided to open source it so that others can use it as well.The log message processing and throughput of these utilities are very high around 13K msg/sec.

At present, the collection includes:
- scribe_stdin: The scribe_stdin client listens to standard input for the messages which are to be sent to scribe server.
- scribe_tail: The scribe_tail client is a file tailer which listens to a log file and send the messages to the scribe server.
- libscribewrapper: The libscribewrapper is the wrapper which converts the plain text messages to the Scribe server understandable format (i.e. Thrift RPC calls). It is available as a shared library which can be used for creating user derived clients.

Feel free to download it or fork it from https://github.com/deepeshmalviya/scribe-utils. All feedbacks and improvements are welcome.

Liked the content? Then why not share with your pals

  • 2 responses to "Scribe Utilities"

  • Fernando
    20:01 on June 7th, 2012

    Hy, I need help for installing the scribe-utilities. When I download your package, i run “sudo make install” but I get the next message: cant do ’stat’ on /home/fernando/deepeshmalviya-scribe-utils-c19484b/build/scribe_stdin»: doesnt exist the file or directory

    Thanks

  • Jason
    5:52 on February 23rd, 2013

    Hi,

    Do these tools require a specific version of thrift? I am trying to build w/thrift 0.9 c++ libraries and getting errors.

    Thanks,
    Jason

  • Leave a Reply

    * Required
    ** Your Email is never shared