Clojure vs Java: Troubleshooting an application in production

I have written a new controller that modifies slightly the incoming requests and sends it further to another service, returning its result. It worked just fine locally but failed with 400 Bad Request in our test environment in the cloud. Thus I needed to get more insight into what was happening in the application to find out what the crucial difference between local and remote was.

The ultimate troubleshooting tool in Java is the debugger. Remote debugging requires that you can access the debuggers port directly or via SSH and that the application is actually running in debug mode.

The Java HotSpot VM uses “full-speed debugging”. But it isn’t really full speed as some optimizations are reportedly impossible in debug mode. One source claims that the overhead of running in debug mode about 5% while an old AMD report found performance between 17% and 100% (2010, JDK6).

For our purposes running in the debug mode (or making it possible to switch over to it) would likely be just fine, especially given this is a test environment, not production. In prod we might be more hesitant to incur any overhead. And if you do then you run under slightly different conditions in test and prod, opening a window for weird and likely rare issues to slip through undetected. (I guess it is mainly a cost-benefit-security consideration.)

A debugger has one big limitation compared to Clojure REPL - its ability to change the code is fairly limited.

Truth be told, I have not used the debugger because our application and infrastructure is not configured for it. Running in Docker on Kubernetes, it is not so easy (for me) to change its startup arguments to enable debug mode.

So I did the next best thing - added some println to log the incoming and outgoing headers and bodies and prayed that it would provide enough information to understand the cause of the problem. Luckily, it did - the Authorization: header we were adding contained an error HTML instead of a valid token, indicating that there were both problems reaching the authorization service and our handing of its non-OK responses.

So I changed the code, committed (I did not actually want this on master but anything else was too complicated), waited 20 min (I know, that is extreme) for our build to finish and publish a new Docker image that I could deploy. That is a terribly slow feedback loop.

The primary tool in Clojure is the REPL. It is the coding terminal connected to your codebase and your running application, making it possible to explore the application state, run code in its context, and evolve the application (while it is running and without losing any state). This is what you use both to develop your application and to troubleshoot it.

To be able to connect the REPL to your application, you need either be able to reach a port as with the debugger or use REPL-over-HTTP.

To “debug” your application, you can use your brainpower and binary search, instrument the code, or, in rare cases, use an actual Clojure-aware debugger. Puppet developers advise:

So how would I proceeded if this was a Clojure application?

I would create an atom (a “global variable”) and modify the request handler function to store the incoming and outgoing requests there. Then I would trigger the failing request from the browser and go back to the REPL to look at the requests. I would fix the code - both locally and in the running app - and, directly from the REPL, run the request through the function(s) again. Rinse and repeat until fixed. (Remembering to commit and deploy your changes at the end :-)).

As you can see, the feedback loop here is about instantaneous.

For me, effective development is all about quick feedback loops. With Clojure REPL I can both troubleshoot a problem and explore how to fix it at the same time, with instantaneous feedback - whether the application is local or remote. Java, even with a debugger, requires long cycles resulting in frustration and inefficiency.

Disclaimer: I do not intend to participate in a flame war or bash Java. My intention is to demonstrate, on concrete examples, the strengths of Clojure I value. Your values and needs might differ and thus Java might be the perfect solution for you. I don’t mind that :-).