PlatformLab / NanoLog, Hacker News

Nanolog is an extremely performant nanosecond scale logging system for C that exposes a simple printf-like API and achieves over 80millionlogs / second at a median latency of just over7 nanoseconds.

How it achieves this insane performance is by extracting static log information at compile-time, only logging the dynamic components in runtime hotpath, and deferring formatting to an offline process. This basically shifts work out of the runtime and into the compilation and post-execution phases.

More information about the techniques used in this logging system can be found in theNanoLog Paper published in the 2018 USENIX Annual Technical Conference.

(Performance)

This section shows the performance of NanoLog with existing logging systems such asspdlog v1.1.0,Log4j2 v2.8,Boost 1. 55,glog v0.3.5, and Windows Event Tracing with Windows Software Trace Preprocessor on Windows 10(WPP).

Throughput

Maximum throughput measured with 1 million messages logged back to back with no delay and 1 – (logging threads) NanoLog logged 100 million messages to generate a log file of comparable size). ETW is “Event Tracing for Windows.” The log messages used can be found in theLog Message Map below.

Runtime Latency

Measured in nanoseconds and each cell represents the 50 TH / 99 tail latencies. The log messages used can be found in theLog Message Map below

Log Messages Map

Log messages used in the benchmarks above.Italicsindicate dynamic log arguments.

Prerequisites

NanoLog depends on the following:

********************************************** (C ) Compiler: (GNU GCC 6.4.0) or greater
• (GNU Make 4.0) or greater
• Python 2.7.9or greater
• POSIX AIO and Threads (usually installed with Linux)

NanoLog Pipeline

The NanoLog system enables low latency logging by deduplicating static log metadata and outputting the dynamic log data in a binary format. This means that log files produced by NanoLog are in binary and must be passed through a separate decompression program to produce the full, human readable ASCII log.

Compiling NanoLog

There are two versions of NanoLog (Preprocessor version and C 17 version) and you must chose (one) to use with your application as they’re not interoperable. The biggest difference between the two is that the Preprocessor version requires one to integrate a Python script in their build chain while the C 17 version is closer to a regular library (simply build and link against it). The benefit of using the Preprocessor version is that it performs more work at compile-time, resulting in a slightly more optimized runtime.

If you don’t know which one to use, go with C 17 NanoLog as it’s easier to use.

C (NanoLog)

The C 17 version of NanoLog works like a traditional library; just# include "NanoLogCpp 17 .H "and link against the NanoLog library. A sample application can be found in thesample directory.

To build the C 17 NanoLog Runtime library, go in theruntime directoryand invokemake. This will produce./ libNanoLog.ato against link your application and a. / decompressorapplication that can be used to re-inflate the binary logs.

When you compile your application, be sure to include the NanoLog header directory (- I ./ Runtime), and link against NanoLog, pthreads, and POSIX AIO (- L ./runtime/ -lNanoLog -lpthreads -lrt). Sample g invocations can be found in thesample GNUmakefile.

After you compile and run the application, the log file generated can then be passed to the./Decompressorapplication to generate the full human-readable log file (instructions below).

Preprocessor NanoLog

The Preprocessor version of NanoLog requires a tighter integration with the user build chain and is only for advanced / extreme users.

Itrequiresthe user’s GNUmakefile to include theNanoLogMakeFrag, declare USR_SRCS and USR_OBJS variables to list all app’s source and object files respectively, and use the pre-defined (run-cxx) macro to compileALLthe user .cc files into .o files instead of (g ​​ ) . See thepreprocessor sample GNUmakefilefor more details.

Internally, therun-cxxinvocation will run a Python script over the source files and generate library code that isspecificto each compilation of the user application. In other words, the compilation builds a version of the NanoLog library that isnon-portable, even between compilations of the same applicationand eachmakeinvocation rebuilds this library.

Additionally, the compilation should also generate a./Decompressorexecutable in the app directory and this can be used to reconstitute the full human-readable log file (instructions below).

NanoLog API

To use the NanoLog system in the code, one just has to include the NanoLog header (eitherNanoLogCpp 17. hfor C 17 NanoLog orNanoLog.hfor Preprocessor NanoLog) and invoke theNANO_LOG ()function in a similar fashion to printf, with the exception of a log level before it. Example below:

#  (include)""NanoLogCpp 17 .H"`usingnamespaceNanoLog  :: LogLevels;int(main)  () {   NANO_LOG(NOTICE,"Hello World! This is an integer% d and a double% lf r  n",  (1) ,  (2.0) );   return(0) ; }

Valid log levels are DEBUG, NOTICE, WARNING, and ERROR and the logging level can be set viaNanoLog :: setLogLevel (...)

The rest of the NanoLog API is documented in theNanoLog.hheader file.

Post-Execution Log Decompressor

The execution of the user application should generate a compressed, binary log file (default locations: ./compressedLog or / tmp / logFile). To make the log file human-readable, simply invoke thedecompressorapplication with the log file.

./ decompressor decompress ./compressedLog

After building the NanoLog library, the decompressor executable can be found in either the./ runtime directory(for C (NanoLog (or the user app directory) for Preprocessor NanoLog).