Heap snapshot analysis and objects with different class versions

I’ve been working lately on the detection of a particular problem, called stale references, that may happen on OSGi based applications. It is a consequence of bad OSGi programming practices that may lead to memory retention and the utilization of inconsistent objects.

As a part of the diagnosis process I need to analyze heap snapshots to find the referrers of services that have been unregistered. I’ve found some interesting stuff (at least for me) concerning memory inspection and classloading. Each module (bundle) in OSGi is provided with its own class loader instance. If you replace (update) a module during runtime, it will basically stop, refresh and restart; and get a new class loader. Objects and classes from the previous class loader must not be referenced anymore, and that previous CL is supposed to be “discarded” and GC’d.

In our custom tool I was seeing that objects from “discarded” class loaders were still being referenced by other modules. However, when I used JHat (either embedded or standalone) to inspect the heap snapshots by performing queries like “select x from com.foo.TheClassOfTheReferencedObject” the result set listed only one object instance (from the running module) when I was supposed to get two instances (one loaded by the old class loader and other by the new one).

I thought my tracking code was falsely accusing the service object from the old bundle version. After patiently and manually (maybe “stupidly”) verifying each class loader instance, I found the two different class loaders that referred to the same bundle ID, and could also found the “lost” object.

After a few weeks I’ve tried to track the same problem with the heap inspection provided in the VisualVM. It worked like a charm! I could see all object instances of the same class name, no matter what class loader provided the object class. It does not have advanced queries like JHat, but in this case I only needed a filter to find the class instances that interest me.

It’s a pity. JHat has powerful queries to analyze heap snapshots, but it can’t deal clearly with the same class names loaded by different class loaders (i.e. different versions of the same class). VisualVM has much more limited power that JHat, but allows to see attribute values, referrers and referring trees just like JHat. In addition, VisualVM deals with no problem with classes carrying the same name but loaded by different class loaders, and that what’s more interesting for me, at least for now.

"Hacking" JHat API to integrate it to my application

Those that already used jhat may say: "JHat API? I thought it was just a java tool that comes with JDK 6."
I'll explain.

JHat (Java Heap Analysis Tool) is an experimental tool that comes with JDK 6. It can parse a heap dump file (HPROF format) and provide a simple http server (port 7000 as default) where we can execute queries in a simple SQL like language that has some predefined functions.

No big deal so far. All that is documented. A while ago I needed to integrate a memory inspection tool in a Java application that I was developing. I know there are few tools to do that externally but I wanted do memory inspection from a menu option in my application.

Although jhat is available as a command line tool, actually its classes are in the tools.jar file. So, I thought I could try do integrate that with my app. I tried to google some javadocs on jhat. Couldn’t find it. The tool is experimental, I forgot that.

Thanks to decompilers I could see how to start up the query engine. I extracted the code and could adapt it to my needs in my application.

I’ve tried to write the examples in a more didactic method in order to be more understandable. I hope I’ve put it in an easy way.

In order to run the example you would need an HPROF file. There are some ways to obtain a heap dump. The easiest (I think) is to run jmap and get the file dump.

Memory inspection done this way is not real time, since we are analyzing a snapshot of memory. But that helped me a lot. Remember that this is a heavy process that may consume lots of memory.

1. Run jps to find out the PID of the Java application to take the memory snapshot
2. Now run: jmap –dump:fomat=b,file=myheapdump.bin
3. Now: java JHatExample myheapdump.bin

In case of out of memory errors when running the example use the –Xmx option:

java –Xmx512m JHatQueryExample myheapdump.bin

The example consists in three classes:

• JHatExample: The GUI code
• JHatWrapper: A class that wraps the access to the jhat API
• MyQueryVisitor: Implementation of the jhat’s ObjectVisitor interface that provides a call back method on the results of the query.

Running the code will show the following GUI:

With the same query from the picture, if you click execute it will show all referrers to all instances of java.io.File.
Actually the way I print the result of the query is quite confusing...
If anybody ever happen to try to run the code, there is a good link on constructing queries in Sundarajan's blog with nice examples. The code I provided makes some calls to the javascript library that is provided with JHat.
If you uncompress the tools.jar file you may find that library and the help page in:
tools.jar\com\sun\tools\hat\resources\

The purpose of this code is to show the possibility to integrate jhat usage into your application. Normally it is used as a stand alone tool. The stand alone tool is ready to use, but is a side tool that you have to open. What I did in my app (not the one shown here) was to automate the memory dump via a Runtime.exec call to jmap, and then the resulting file was used as a parameter to instatiate the jhat engine.

Since jhat is an experimental tool, it may completely change or become unavailable on Java 7. Use it like I did at your own risk.

Have fun playing with the code!

The GUI Code:

import java.awt.*;
import java.io.File;
import javax.swing.*;

public class JHatExample extends JFrame {
private JHatWrapper jhat;

public JHatExample(JHatWrapper jhat) {
 super("JHat wrapper");
 this.jhat = jhat;
 initComponents();
}


private void initComponents() {
 final JTextArea queryText = new JTextArea();
 JPanel buttonPanel = new JPanel();
 JButton executeBtn = new JButton("Execute");
 buttonPanel.add(executeBtn);
 this.add(new JScrollPane(queryText),BorderLayout.CENTER);
 this.add(buttonPanel,BorderLayout.SOUTH);
 executeBtn.addActionListener(new ActionListener() {
  public void actionPerformed(ActionEvent e) {
   try {
    jhat.query(queryText.getText());
   } catch (Exception e1) {
    JOptionPane.showMessageDialog(JHatExample.this, "Error executing query");
   }
  }
 });
}

public static void main(String[] args) throws Exception {
 if (args.length == 0 || ! new File(args[0]).exists()) {
  System.out.println("The program must take a valid file path as parameter");
  return;
 }
 JHatWrapper jhat = new JHatWrapper(args[0]);
 JHatExample frame = new JHatExample(jhat);
 frame.setSize(400,200);
 frame.setVisible(true);
 frame.setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE);
}
}

Now the wrapper class:

import java.io.*;
import com.sun.tools.hat.internal.model.Snapshot;
import com.sun.tools.hat.internal.oql.OQLEngine;
import com.sun.tools.hat.internal.parser.HprofReader;
import com.sun.tools.hat.internal.parser.PositionDataInputStream;

public class JHatWrapper {
private OQLEngine engine;

public JHatWrapper(String s) throws Exception {
 PositionDataInputStream positiondatainputstream =
 new PositionDataInputStream(new BufferedInputStream(new FileInputStream(s)));
  int l;
  Snapshot snapshot;
  l = positiondatainputstream.readInt();
   //I "inferred" the meaning of the parameters below by comparing the decompiled code
//and jhat's command line parameters
  int debuglevel = 0;
  //Don't know what this is...
  int j =1;
  HprofReader hprofreader = new HprofReader(s, positiondatainputstream,
j, true, debuglevel);
  snapshot = hprofreader.read();
  positiondatainputstream.close();
  //resolve references
  snapshot.resolve(true);
  engine = new OQLEngine(snapshot);
}

public void query(String query) throws Exception {
  engine.executeQuery(query, new MyQueryVisitor(engine) );
}
}

And the Visitor code:

import java.util.Enumeration;

import com.sun.tools.hat.internal.model.JavaHeapObject;
import com.sun.tools.hat.internal.oql.OQLEngine;
import com.sun.tools.hat.internal.oql.ObjectVisitor;

public class MyQueryVisitor implements ObjectVisitor {
private OQLEngine engine;

public MyQueryVisitor(OQLEngine engine) {
 this.engine = engine;
}

public boolean visit(Object obj) {
 try {
  Object iterator = engine.call("wrapIterator",new Object[]{obj});
  //Handles result items that are instance of enumeration. Ex: select referrers(x) from java.io.File x
  if (iterator instanceof Enumeration) {
   System.out.println("enum");
   handleEnumeration((Enumeration)iterator);
  } else {
   //handles single result items. Ex: select x from java.io.File x
   JavaHeapObject javaObj = (JavaHeapObject)engine.call("unwrapJavaObject",new Object[]{obj});
   handleEnumeration(javaObj.getReferers());
  }
 } catch (Exception e) {
  e.printStackTrace();
 }
 return false;
}
void handleEnumeration(Enumeration en) {
 while (en.hasMoreElements()) {
  try {
   Object obj = en.nextElement();
   JavaHeapObject unwrapped = (JavaHeapObject)engine.call("unwrapJavaObject",new Object[]{obj});
   System.out.println(unwrapped);
  } catch (Exception e) {
   e.printStackTrace();
  }
 }
}
}