Shahzad Bhatti Welcome to my ramblings and rants!

Log locally and query globally I have found the rule “log locally and query globally” quite handy in large distributed applications. However, I still find many projects that try to create a centralized logging server for receiving all log messages. In many cases this adds too much overhead for logging. Though, asynchronous logging can reduce direct overhead, but it is still problematic. This also results in loss of log messages or confusion when the log server crashes. A single logging server often becomes bottleneck for resources such as network, disk and CPU. I admit I have designed some systems where I setup centralized logging service. In some instances I used wrapper on top of UNIX’s syslog server. One of the drawback of syslog was that it drops messages under heavy load. Though, syslog service provides useful logging for system level logs where you don’t want to log system related messages locally especially security violations because if the local machine is compromised you will loose all logging information. However, I have found for application logging, local logging works much better. On the other hand this requires that a centralized service exist that can search logs from all machines. One of
the difficulty with local logging is creating a complete picture of logs. For example a user request goes throught various services running on different servers and for debugging it is important to trace complete business transaction information. In such cases, I have found a transaction correlation id quite useful. For example, each business transaction id can be stored as part of the log messages or user’s session information can be used, which is later
used to create complete picture. Also, it is important that all servers use NTP service and have their clocks synchronized so that the log messages are in right time order. For query, you can create a local searching agent on each machine which is contacted by centralized query system. This way you can query multiple log files simultaneously and return results much more
efficiently.