Fixing The Tool

WHEN YOUR JAVA IS BROKEN

Most of Infobip’s messaging platform and web services are written in Java. While this widespread technology provides stability and ease of development, with many support libraries and good debugging and profiling tools, sometimes you find such an irritating defect in it that it makes you wonder if it is real. And it’s real: on Linux, you cannot see the Java thread name while inspecting native processes. Because Java threads map one-to-one to native system threads, it would come in real handy if you could just hit ‘top’ and take a look which Java thread is using most of the CPU at the moment, without having to run a debugging tool. What’s more irritating is that this used to work on Oracle JRockit 6 and somehow got lost in transition to OpenJDK 7 and 8. And to make it even more agonizing for a proud Linux user – it works on Windows!

So what is your average Java coder to do? You can feel discouraged when confronted with big boys like Oracle and their software, give up easily and live with it. Or, you can challenge the status quo, push your skills to the limits and persist on solving it. We at Infobip are this kind of people.

MAKE YOUR OWN JAVA

A little investigation shows there is a GNU extension to POSIX threading API for setting Linux thread name called pthread_setname_np. The documentation reveals that Linux keeps process names (descriptions, command names, call it whatever you like) in /proc/[pid]/comm limited to 15 characters. Unix tools like ps and top can make use of it. Let’s see how this works:

		$ ps | grep $$ # $$ is current PID
7880 pts/3 00:00:00 bash
$ echo -n “Hi mum, look at me!” > /proc/$$/comm
$ ps | grep $$
7880 pts/3 00:00:00 Hi mum, look at me!
							
											1
											2
											3
											4
											5

In top you can hit the ‘c’ key to switch between cmd line and command name display.

Now let’s just make Java use this API to set native thread name. Easy to say, eh? OpenJDK is open source, so let’s download the sources first:

		$ hg clone http://hg.openjdk.java.net/jdk8u/jdk8u60
$ cd jdk8u60
$ sh get_source.sh
							
											1
											2
											3

Let’s try to dig where java.lang.Thread.setName(String arg0) ends. You’ll need all your long forgotten C++ reading skills for this, but it’s fairly easy to find this piece of code:

HOTSPOT/SRC/OS/LINUX/VM/OS_LINUX.CPP

		void os::set_native_thread_name(const char *name) {
// Not yet implemented.
return;
}
 
bool os::distribute_processes(uint length, uint* distribution) {
// Not yet implemented.
return false;
}
 
bool os::bind_to_processor(uint processor_id) {
// Not yet implemented.
return false;
}
							
											1
											2
											3
											4
											5
											6
											7
											8
											9
											10
											11
											12
											13
											14

By the way, you can also see here that setting CPU affinity definitely won’t work either, but we won’t go there as there’s enough material for another blog post.

If you feel uncomfortable about modifying JVM code, you’re not alone – someone has already done it before. There is a ticket for this enhancement together with a working solution opened in 2011 and followed by a long and rather academic discussion, resulting in a few other open tickets for glibc requesting to release the 15 characters restriction. It’s not going to happen because the restriction is hard-coded deep in the Linux kernel. Looks like the enhancement finally made it into Java 9, but for those of us who are impatient, here is the patch.

APPLYING THE PATCH

		jdku60$ patch -d hotspot -p1 <../patches/thread_name.cpp.patch
							
											1

Building OpenJDK is surprisingly easy. Java 9 switched to cmake but with Java 8 you’ll have to cope with autotools:

		$ sh conigure.sh 
# <do whatever the script tells you to do>
$ make
							
											1
											2
											3

The build produced a file build/linux-x86_64-normal-server-release/hotspot/dist/jre/lib/amd64/server/libjvm.so which you can use as a drop-in replacement for the same file in standard JVM distribution. Now try the magic:

THREADNAMETEST.JAVA

		public class ThreadNameTest {
    public static void main(String[] args) throws Exception {
        Thread.currentThread().setName(“Hi mum, look at me!”);
        Thread.sleep(100_000);
    }
}
							
											1
											2
											3
											4
											5
											6

JAVA WITH THREAD NAME SUPPORT

		$ javac ThreadNameTest.java
$ java ThreadNameTest &
[1] 19391
$ for PID in `ls /proc/$!/task/`; do echo $PID – `cat /proc/$!/task/$PID/comm`; done
19563 – java
19564 – Hi mum, look at
19565 – java
19566 – java
19567 – java
19568 – java
19569 – java
19570 – java
19571 – java
19572 – java
19573 – java
19574 – Reference Handl
19575 – Finalizer
19576 – Signal Dispatch
19577 – C2 CompilerThre
19578 – C2 CompilerThre
19579 – C2 CompilerThre
19580 – C1 CompilerThre
19581 – Service Thread
19582 – java
$ kill -s 15 $!
[1]+  Exit 143                java ThreadNameTest
							
											1
											2
											3
											4
											5
											6
											7
											8
											9
											10
											11
											12
											13
											14
											15
											16
											17
											18
											19
											20
											21
											22
											23
											24
											25
											26

VANILLA JAVA

		$ javac ThreadNameTest.java
$ java ThreadNameTest &
[1] 19391
$ for PID in `ls /proc/$!/task/`; do echo $PID – `cat /proc/$!/task/$PID/comm`; done
19563 – java
19564 – java
19565 – java
19566 – java
19567 – java
19568 – java
19569 – java
19570 – java
19571 – java
19572 – java
19573 – java
19574 – java
19575 – java
19576 – java
19577 – java
19578 – java
19579 – java
19580 – java
19581 – java
19582 – java
$ kill -s 15 $!
[1]+  Exit 143                java ThreadNameTest
							
											1
											2
											3
											4
											5
											6
											7
											8
											9
											10
											11
											12
											13
											14
											15
											16
											17
											18
											19
											20
											21
											22
											23
											24
											25
											26

(By Milan Mimica, Software Engineer / Team Leader)

Dec 21st, 2015

3 min read