Fixing The Tool

Most of Infobip’s messaging platform and web services are written in Java. While this widespread technology provides stability and ease of development, with many support libraries and good debugging and profiling tools, sometimes you find such an irritating defect in it that it makes you wonder if it is real. And it’s real: on Linux, you cannot see the Java thread name while inspecting native processes. Because Java threads map one-to-one to native system threads, it would come in real handy if you could just hit ‘top’ and take a look which Java thread is using most of the CPU at the moment, without having to run a debugging tool. What’s more irritating is that this used to work on Oracle JRockit 6 and somehow got lost in transition to OpenJDK 7 and 8. And to make it even more agonizing for a proud Linux user – it works on Windows!

So what is your average Java coder to do? You can feel discouraged when confronted with big boys like Oracle and their software, give up easily and live with it. Or, you can challenge the status quo, push your skills to the limits and persist on solving it. We at Infobip are this kind of people.

A little investigation shows there is a GNU extension to POSIX threading API for setting Linux thread name called pthread_setname_np. The documentation reveals that Linux keeps process names (descriptions, command names, call it whatever you like) in /proc/[pid]/comm limited to 15 characters. Unix tools like ps and top can make use of it. Let’s see how this works:

In top you can hit the ‘c’ key to switch between cmd line and command name display.

Now let’s just make Java use this API to set native thread name. Easy to say, eh? OpenJDK is open source, so let’s download the sources first:

Let’s try to dig where java.lang.Thread.setName(String arg0) ends. You’ll need all your long forgotten C++ reading skills for this, but it’s fairly easy to find this piece of code:

By the way, you can also see here that setting CPU affinity definitely won’t work either, but we won’t go there as there’s enough material for another blog post.

If you feel uncomfortable about modifying JVM code, you’re not alone – someone has already done it before. There is a ticket for this enhancement together with a working solution opened in 2011 and followed by a long and rather academic discussion, resulting in a few other open tickets for glibc requesting to release the 15 characters restriction. It’s not going to happen because the restriction is hard-coded deep in the Linux kernel. Looks like the enhancement finally made it into Java 9, but for those of us who are impatient, here is the patch.

Building OpenJDK is surprisingly easy. Java 9 switched to cmake but with Java 8 you'll have to cope with autotools:

The build produced a file build/linux-x86_64-normal-server-release/hotspot/dist/jre/lib/amd64/server/libjvm.so which you can use as a drop-in replacement for the same file in standard JVM distribution. Now try the magic:

(By Milan Mimica, Software Engineer / Team Leader)