I am always hesitant to incorporate too many debug logging statements in my code as it tend to clutter the code, sometimes get left turned on in production where it can dramatically slow down the system, expose confidential information and also consume all free storage space quickly. Of course, if done properly this is an incredibly powerful aspect of system design.

I was reminded today how important it is to have this looking glass built into systems, especially background processes. I had created a system that integrates many disparate systems, and communicates almost exclusively using REST API calls. However, one feature was not yet turned on, so the code followed a logic path that I did not anticipate at the time. In the end, the system ended up making hundreds of API calls per second to a specific service endpoint for days on end, without me knowing.

I started picking up that something was amiss as every day I would get one or two notifications that the API call failed - due to timeout, internal error or other reasons. However, that method was being invoked by any of up to 30 different callers - so tracking it was very hard.

I decided to inject some low level HTTP request logging in my common HTTP interface class - this conveniently caught all REST API calls made by my system in one place. Once deployed, I discovered the runaway queries as the log file grew. With the help of some simple stack trace logic I managed to pinpoint the culprit, fix it and now the logs are silent, and my service endpoint calm.