in , , ,

PlatformLab / NanoLog, Hacker News

PlatformLab / NanoLog, Hacker News


                    

        

Nanolog is an extremely performant nanosecond scale logging system for C that exposes a simple printf-like API and achieves over 80millionlogs / second at a median latency of just over7 nanoseconds.

How it achieves this insane performance is by extracting static log information at compile-time, only logging the dynamic components in runtime hotpath, and deferring formatting to an offline process. This basically shifts work out of the runtime and into the compilation and post-execution phases.

More information about the techniques used in this logging system can be found in theNanoLog Paper published in the 2018 USENIX Annual Technical Conference.

spdlog v1.1.0,Log4j2 v2.8,Boost 1. 55,glog v0.3.5, and Windows Event Tracing with Windows Software Trace Preprocessor on Windows 10(WPP).

Throughput

Maximum throughput measured with 1 million messages logged back to back with no delay and 1 – (logging threads) NanoLog logged 100 million messages to generate a log file of comparable size). ETW is “Event Tracing for Windows.” The log messages used can be found in theLog Message Map below.N|Solid

Runtime Latency

Measured in nanoseconds and each cell represents the 50 TH / 99 tail latencies. The log messages used can be found in theLog Message Map below

Message NanoLog spdlog Log4j2 glog Boost ETW
staticString 7 / 37 214 / 2546 174 / 3364 1198 / 5968 1764 / 3772 161 / 2967
stringConcat 7 / 36 279 / 905 256 / 25087 1212 / 5881 1829 / 5548 191 / 3365
singleInteger 7 / 32 268 / 855 180 / 9305 1242 / 5482 1914 / 5759 167 / 3007
twoIntegers 8 / 62 437 / 1416 183 / 10896 1399 / 6100 2333 / 7235 177 / 3183
singleDouble 8 / 43 585 / 1562 175 / 4351 1983 / 6957 2610 / 7079 165 / 3182
complexFormat 8 / 40 1776 / 5267 202 / 18207 2569 / 8877 3334 / 11038 218 / 3426

Log Messages Map

Log messages used in the benchmarks above.Italicsindicate dynamic log arguments.

Message ID Log Message Used
staticString Starting backup replica garbage collector thread
singleInteger Backup storage speeds (min):MB / s read
twoIntegers buffer has consumed1032024bytes of extra storage, current allocation :1016544bytes
singleDouble Using tombstone ratio balancer with ratio=(0.4)
complexFormat Initialized InfUdDriver buffers:50000receive buffers (97MB),50transmit buffers (0MB), took26 .2ms
stringConcat Opened session with coordinator atbasic udp: host=192. 168 .1. 140, port=12246

Prerequisites

NanoLog depends on the following:

    ********************************************** (C ) Compiler: (GNU GCC 6.4.0) or greater

  • (GNU Make 4.0) or greater
  • Python 2.7.9or greater
  • POSIX AIO and Threads (usually installed with Linux)

NanoLog Pipeline

The NanoLog system enables low latency logging by deduplicating static log metadata and outputting the dynamic log data in a binary format. This means that log files produced by NanoLog are in binary and must be passed through a separate decompression program to produce the full, human readable ASCII log.

Compiling NanoLog

There are two versions of NanoLog (Preprocessor version and C 17 version) and you must chose (one) to use with your application as they’re not interoperable. The biggest difference between the two is that the Preprocessor version requires one to integrate a Python script in their build chain while the C 17 version is closer to a regular library (simply build and link against it). The benefit of using the Preprocessor version is that it performs more work at compile-time, resulting in a slightly more optimized runtime.

If you don’t know which one to use, go with C 17 NanoLog as it’s easier to use.

C (NanoLog)

The C 17 version of NanoLog works like a traditional library; just# include "NanoLogCpp 17 .H "and link against the NanoLog library. A sample application can be found in thesample directory.

To build the C 17 NanoLog Runtime library, go in theruntime directoryand invokemake. This will produce./ libNanoLog.ato against link your application and a. / decompressorapplication that can be used to re-inflate the binary logs.

When you compile your application, be sure to include the NanoLog header directory (- I ./ Runtime), and link against NanoLog, pthreads, and POSIX AIO (- L ./runtime/ -lNanoLog -lpthreads -lrt). Sample g invocations can be found in thesample GNUmakefile.

After you compile and run the application, the log file generated can then be passed to the./Decompressorapplication to generate the full human-readable log file (instructions below).

Preprocessor NanoLog

The Preprocessor version of NanoLog requires a tighter integration with the user build chain and is only for advanced / extreme users.

Itrequiresthe user’s GNUmakefile to include theNanoLogMakeFrag, declare USR_SRCS and USR_OBJS variables to list all app’s source and object files respectively, and use the pre-defined (run-cxx) macro to compileALLthe user .cc files into .o files instead of (g ​​ ) . See thepreprocessor sample GNUmakefilefor more details.

Internally, therun-cxxinvocation will run a Python script over the source files and generate library code that isspecificto each compilation of the user application. In other words, the compilation builds a version of the NanoLog library that isnon-portable, even between compilations of the same applicationand eachmakeinvocation rebuilds this library.

Additionally, the compilation should also generate a./Decompressorexecutable in the app directory and this can be used to reconstitute the full human-readable log file (instructions below).

NanoLog API

To use the NanoLog system in the code, one just has to include the NanoLog header (eitherNanoLogCpp 17. hfor C 17 NanoLog orNanoLog.hfor Preprocessor NanoLog) and invoke theNANO_LOG ()function in a similar fashion to printf, with the exception of a log level before it. Example below:

#  (include)""NanoLogCpp 17 .H"`usingnamespaceNanoLog  :: LogLevels;int(main)  () {   NANO_LOG(NOTICE,"Hello World! This is an integer% d and a double% lf r  n",  (1) ,  (2.0) );   return(0) ; }

Valid log levels are DEBUG, NOTICE, WARNING, and ERROR and the logging level can be set viaNanoLog :: setLogLevel (...)

The rest of the NanoLog API is documented in theNanoLog.hheader file.

Post-Execution Log Decompressor

The execution of the user application should generate a compressed, binary log file (default locations: ./compressedLog or / tmp / logFile). To make the log file human-readable, simply invoke thedecompressorapplication with the log file.

./ decompressor decompress ./compressedLog

After building the NanoLog library, the decompressor executable can be found in either the./ runtime directory(for C (NanoLog (or the user app directory) for Preprocessor NanoLog).

  

Brave Browser

Payeer

Read More

What do you think?

Leave a Reply

Your email address will not be published.

GIPHY App Key not set. Please check settings

Max Keiser Sets $ 28,000 Bitcoin Price Target as Market Falters – CCN Markets, Crypto Coins News

Max Keiser Sets $ 28,000 Bitcoin Price Target as Market Falters – CCN Markets, Crypto Coins News

Show HN: Square Game, Hacker News