VisualVM OSGi Plugin available at the plugins center

The VisualVM OSGi plugin has become part of the OW2 Chameleon project, which tries to deliver a set of pluggable components for OSGi frameworks, allowing to create sorts of lightweight OSGi-based application servers with just the needed functionality. The VisualVM OSGi plugin was the initial step for a set of VisualVM plugins that will be provided in the Chameleon project.

Recently Jiri Sedlacek and Tomas Hurka have proposed to add a link to the OSGi plugin in the VisualVM 1.2 plugins center. Now you can install the plugin from the VisualVM 1.2 or above, in the menu Tools > Plugins. Don'f forget to download and install the appropriate OSGi bundle in your OSGi application (this is how the plugin will get the info about the OSGi application).
Detailed installation instructions can be found on the plugin page in Chameleon's wiki

Many thanks to my thesis advisor, Didier Donsez, for the idea of the plugin.
Thanks to Clément Escoffier, for inviting the VisualVM OSGi Plugin to the Chameleon project.
Thanks to Jiri and Tomas for all the tips, testing and for adding the plugin to the VisualVM plugins center, and thanks in advance for anyone who would eventually use the tool and hopefully provide any comments for improving it.

Withdrawn JSRs

I've checked the status of JSR-277, which was announced as being halted by Sun, but its status did not change to withdrawn. That makes sense, since it was announced as temporarily halted.
So I went to check the list of all withdrawn JSRs, just for curiosity, and the number surprised me: 52. A lot of those were concerning JAIN (which I haven't heard of until now). Some of the JSRs had comments like "withdrawn at the request of the submitter".

If you look at the number of all JSRs (355) we can do simple math and find that 15% of all JSRs have been withdrawn. It is true that they vary a lot in size, as there are umbrella JSRs that have a lot of stuff behind and we have some successful efforts that are heavyweight. Also, among that list there are "spinoffs", like the JSR-8 which originated OSGi. But, is 15% a high rate or considering the fact that standardization is a complex effort that involves people from different organizations which may have different interests, could we say that it is a fairly acceptable percentage ?

Can AOP be some sort of elegant workaround?

In the pursuit of problem solving (or at least trying to) in my research, I guess I am taking AOP as a strategy to give me what I don't have. Maybe I am becoming too addicted to pointcuts...

I have used Aspect Oriented Programming for intercepting method calls to the OSGi framework and for adding my "concerns". These interceptions allowed me to, mainly, add hooks to service registration and retrieval. I think the good thing is that the solution was elegant enough and enabled portability accross different OSGi implementations, since the aspects targeted the API. The bad thing is that it does not rely in a standard mechanism supported by the platform, and it needs an additional building step (i.e. weaving the framework).

After taking a look at the draft OSGi spec 4.2, I've checked the Service Registry Hooks with the hope that my weaved hooks could be replaced by a standard mechanism. They have exactly proposed the hooks I wanted but sadly that alternative solution had been rejected. The hooks they will provide still do not allow me to do what I need. I'll still be doing the AOP trick...

I just can think of more pointcuts to add to my future solutions/attempts in order to have other types of hooks. As I said previously, this approach is useful to avoid changing the source code of a chosen OSGi implementation. There are lots of people in the research area who usually customize an OSGi impl to their needs (I still see recent articles talking about customizations of OSCAR). One disadvantage is that if you touch the source code directly, you're stuck with a given version which makes it difficult to follow the evolution of the chosen impl. (either Equinox, KF or Felix). You would need constant effort for merging/synchronization your branch with the current version. I believe they could avoid that by doing the AOP trick as well. However, sometimes (like while writing this) I am not sure if this a true elegant solution or if I've just found a nice and powerful workaround for adding my hooks to the OSGi core.

JSR 296 in coma ?

I’ve been checking the upcoming features of Java 7 and found a cool detailed list here. Also, some Java One 2008 slides from Danny Coward’s presentation show a little on that too.

Among the JSRs mentioned in the first link, I already had checked some stuff from JSRs 277, 294, concerning Java Modules, JSR 284, concerning resource consumption and JSRs 295, 296 and 303 concerning Swing.

I’ve known these swing-related JSRs since, more or less, they became available as JSRs. I had special interest on them at the time because I was working in a project where we needed a product built on top of a Rich Client Platform, to enable the development of plugins from third parties. The idea was to have a simple and pluggable structure.

For us, Eclipse RCP was overkill so was the Netbeans platform. Spring RCP was in its initial stages. We’ve decided to go from scratch, using OSGi as the base for a pluggable architecture, and build our own straightforward RCP. The JSR 296 (Swing Application Framework) proposes most of what we needed (and developed) as a Swing foundation for our RCP:

• Resource management for i18n
• Task services/monitoring
• Storage for session state
• Events/actions framework
• Managed application exiting (exit listeners), startup, shutdown
  

I can't say that this is a PCP (Poor Client Platorm). Maybe an ARCP (Almost Rich Client Platform). But, as I said, it does most of what we needed at that time. And I believe there are hundreds or thousands of other applications that don't need all the heavy richness provided by the RCPs.

So I decided to take a look at JSR 296 (Swing Application Framework) to see what they have for us so far. I’ve downloaded it from the project site. (You should also need to download the SwingWorker since they do not use the SwingWorker delivered with Java 6)

This tutorial has good examples concerning the features of the JSR 296, like the magic stuff for saving session state (components dimensioning, positioning, etc):

//The non-qualified getters below are inherited from the SingleFrameApplication class
//
//Saving session State
getContext().getSessionStorage().save(getMainFrame(), sessionFile);
...
//Restoring session State
getContext().getSessionStorage().restore(getMainFrame(), sessionFile);

Lots of weeks of modelling and coding can be saved by using that framework.

There was a weird thing about the last available version: it was from November 2007. It seems that it is not evolving anymore. I’ve read a few months ago in blogs that I don’t recall, that some senior engineers from the Swing team were leaving Sun Microsystems. I could find this blog entry mentioning some of them.

The guys that took care of JSR 296 (Hans Muller) and 295 (Scott Violet) are gone. However, I think that there are no worries with JSR 295 and 303 which are just “standardizing” the concepts used in JGoodies by Karsten Lentz, who is a member of both expert groups. But the JSR 296 is apparently a home grown implementation from Sun. The guy that was handling that is gone, and there are no new versions for 8 months.

Weird… It appears that the project is in some sort of "coma".

Don’t know if it will make it until Java 7

Heap snapshot analysis and objects with different class versions

I’ve been working lately on the detection of a particular problem, called stale references, that may happen on OSGi based applications. It is a consequence of bad OSGi programming practices that may lead to memory retention and the utilization of inconsistent objects.

As a part of the diagnosis process I need to analyze heap snapshots to find the referrers of services that have been unregistered. I’ve found some interesting stuff (at least for me) concerning memory inspection and classloading. Each module (bundle) in OSGi is provided with its own class loader instance. If you replace (update) a module during runtime, it will basically stop, refresh and restart; and get a new class loader. Objects and classes from the previous class loader must not be referenced anymore, and that previous CL is supposed to be “discarded” and GC’d.

In our custom tool I was seeing that objects from “discarded” class loaders were still being referenced by other modules. However, when I used JHat (either embedded or standalone) to inspect the heap snapshots by performing queries like “select x from com.foo.TheClassOfTheReferencedObject” the result set listed only one object instance (from the running module) when I was supposed to get two instances (one loaded by the old class loader and other by the new one).

I thought my tracking code was falsely accusing the service object from the old bundle version. After patiently and manually (maybe “stupidly”) verifying each class loader instance, I found the two different class loaders that referred to the same bundle ID, and could also found the “lost” object.

After a few weeks I’ve tried to track the same problem with the heap inspection provided in the VisualVM. It worked like a charm! I could see all object instances of the same class name, no matter what class loader provided the object class. It does not have advanced queries like JHat, but in this case I only needed a filter to find the class instances that interest me.

It’s a pity. JHat has powerful queries to analyze heap snapshots, but it can’t deal clearly with the same class names loaded by different class loaders (i.e. different versions of the same class). VisualVM has much more limited power that JHat, but allows to see attribute values, referrers and referring trees just like JHat. In addition, VisualVM deals with no problem with classes carrying the same name but loaded by different class loaders, and that what’s more interesting for me, at least for now.

Do you know weak references? Have you ever used them?

I’ve been using a lot of weak references lately. A couple of years ago I’ve read a blog entry that said that a lot of experienced Java developers did not know what they were. I’m glad I already knew it at the time I’ve read that :)

I won’t go into details of weak references. I’ll just say that an object is not prevented to be garbage collected if it only has weak references pointing to it. You can get a nice overview here.

In my current project I’m tracking garbage collection of special types of objects by using weak references. Maybe in another post I’ll provide more info on that.

Throughout my career I’ve used that nice resource of Java a few times. The most important ones that I can remember are:

• Weak Listeners
• Memory Sensitive Caches

Weak Listeners

Memory leaks are a painful problem that sometimes we don’t have a clue how to find them. I’ve worked in a desktop project where we had lots of instantiations of GUI panels, models, etc. That implied in a bunch of model listeners, property listeners, and so forth. We had a major memory leak due to a simple problem in most of the panels: objects from each created panel were being added as a listener of the model, but after the panel was discarded they did not remove themselves from the listeners list of the model. So, they kept being reference by the listeners list on the model and could not be garbage collected.

A naïve example tries to illustrate that:

public void setActivePanel() {
MyPanel thePanel = new MyPanel();
ApplicationModel.getInstance().addXYZListener(thePanel);
this.activePanel = thePanel;
}

public void releaseActivePanel() {
this.activePanel = null;
this.clearSelection();
}

One solution was to provide a way to unsubscribe "thePanel" from the XYZListener list:

public void releaseActivePanel() {
ApplicationModel.getInstance().addXYZListener(this.activePanel);
this.activePanel = null;
this.clearSelection();
}

But.... you could also use weak references. They would avoid such problems. At the time I found good stuff that I’ve used as a reference.

The only issue that I see in his solution is that we may have sometimes a bunch of empty WeakPropertyChangeListeners (pointing to null) when the referred object is GC’d. He provides a lazy approach for clean up. However, if you use your own ReferenceQueue you can provide a thread that once in a while polls the queue and does the sweeping of listener lists. Or also you can use a WeakHashMap which does the dirty work of clean up.

Memory sensitive caches

There are several caching strategies (e.g. LRU, MRU) but this one is very simple. It caches everything, but when the system runs out of memory the cache is cleaned up.

Depending on the memory footprint of your system this strategy may be useless and your cache will not work really well.

This memory sensitive cache relies on SoftReferences. They are special types of WeakReference which hang around for a while.

The following class is not an optimal implementation of a cache, but it works…

public class ImageCache {
private HashMap<String, SoftReference<ImageIcon>> map = new HashMap<String, SoftReference<ImageIcon>>();
private static final String IMAGE_FOLDER = "/images/";

public ImageIcon getImage(String imageFile) {
ImageIcon icon = null;
if (map.containsKey(imageFile)) {
icon = map.get(imageFile).get();
if (icon == null) {
icon = cacheImage(imageFile);
}
} else {
icon = cacheImage(imageFile);
}
return icon;
}
private ImageIcon cacheImage(String imageFile) {
ImageIcon icon = new ImageIcon(this.getClass().getResource(
imageFile));
map.put(imageFile, new SoftReference<ImageIcon>(icon));
return icon;
}
}

Another cool stuff that can be done (although I've never done it in a "real life" project as I did with the other stuff) is to provide an alternative way to finalization using PhantomReferences. You can use them combined with a ReferenceQueue to provide your own finalization mechanism avoiding the error-prone finalize method. You could write your own finalizer thread :)

There is a good reference here.

I guess that it for the moment.

Scriptconsole4j: Embed a scripting console in your Java App.

Using scripting languages in Java became very easy. The JSR 223 standardizes the usage of scripting languages in the Java Platform; and Java 6 already ships with built-in support to JavaScript (Mozilla Rhino engine).

If you need to embed a multi-language enabled scripting “visual console” to your application you may try the scriptconsole4j.

I was developing a Java application where I needed to do some coding on the fly during runtime, change variable values, run snippets of code, etc. So, I’ve created a simple scripting console to use on that application. I don’t know if something like that already existed, but my code and its usage are really simple.

I’ve extracted the classes and made a few changes on it and created a separate reusable jar library (the scriptconsole4j). It is hosted at http://code.google.com/p/scriptconsole4j/

There are two ways to use it:

• A JPanel with scripting functionality that can be embedded in another application (e.g. you can add a scripting window, or a scripting console to your app and access some of your application variables through the scripting window)
• A standalone frame that embeds the above scripting console (e.g. you can practice the scripting language of your choice with it)

This scripting console provides a standard output variable, but allows variables from your own application to be available through the scripting console.
The combobox shows all available scripting engines on the running JVM. By default, Java 6 comes with JavaScript.

By doing the following as in the example, it is possible to add you custom(s) variable(s):

import scriptconsole4j.*;
...

JFrame frame = new JFrame("[Scriptconsole4j] Default Window");
//No textual description available. The console will just display the classname as info
ScriptContextVariable myObjectVar = new ScriptContextVariable(new MyObject(),"myobj","");
//The scripting panel can take several ScriptContextVariable as parameter
ScriptingPanel panel = new ScriptingPanel(myObjectVar);
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.setSize(400, 400);
frame.add(panel);
frame.setVisible(true);

You can add any variable that you want and use it from the scripting panel. In the above code we add an object of type MyObject that will be available in the console with the alias of "myobj". There is no big deal here, since the JSR 223 allows context objects. However, we wrap it and put some additional info so other can see what are the variables that we intentionally want to let available to the console, and hopefully some (optional) textual description of it.

Other JSR 223 compliant engines of existing scripting languages are available at http://scripting.java.net .The download on that site provides the JSR 223 engines but it is still necessary to make additional downloads to the scripting libraries themselves.

Several improvements still need to be done in the console (e.g. ability to load scripts, script embelisher, undo support), but this little panel brought a lot of flexibility for me during development time of my apps.

The first release can be downloaded here.

I hope you can give it a try to see if it helps you too.

"Hacking" JHat API to integrate it to my application

Those that already used jhat may say: "JHat API? I thought it was just a java tool that comes with JDK 6."
I'll explain.

JHat (Java Heap Analysis Tool) is an experimental tool that comes with JDK 6. It can parse a heap dump file (HPROF format) and provide a simple http server (port 7000 as default) where we can execute queries in a simple SQL like language that has some predefined functions.

No big deal so far. All that is documented. A while ago I needed to integrate a memory inspection tool in a Java application that I was developing. I know there are few tools to do that externally but I wanted do memory inspection from a menu option in my application.

Although jhat is available as a command line tool, actually its classes are in the tools.jar file. So, I thought I could try do integrate that with my app. I tried to google some javadocs on jhat. Couldn’t find it. The tool is experimental, I forgot that.

Thanks to decompilers I could see how to start up the query engine. I extracted the code and could adapt it to my needs in my application.

I’ve tried to write the examples in a more didactic method in order to be more understandable. I hope I’ve put it in an easy way.

In order to run the example you would need an HPROF file. There are some ways to obtain a heap dump. The easiest (I think) is to run jmap and get the file dump.

Memory inspection done this way is not real time, since we are analyzing a snapshot of memory. But that helped me a lot. Remember that this is a heavy process that may consume lots of memory.

1. Run jps to find out the PID of the Java application to take the memory snapshot
2. Now run: jmap –dump:fomat=b,file=myheapdump.bin
3. Now: java JHatExample myheapdump.bin

In case of out of memory errors when running the example use the –Xmx option:

java –Xmx512m JHatQueryExample myheapdump.bin

The example consists in three classes:

• JHatExample: The GUI code
• JHatWrapper: A class that wraps the access to the jhat API
• MyQueryVisitor: Implementation of the jhat’s ObjectVisitor interface that provides a call back method on the results of the query.

Running the code will show the following GUI:

With the same query from the picture, if you click execute it will show all referrers to all instances of java.io.File.
Actually the way I print the result of the query is quite confusing...
If anybody ever happen to try to run the code, there is a good link on constructing queries in Sundarajan's blog with nice examples. The code I provided makes some calls to the javascript library that is provided with JHat.
If you uncompress the tools.jar file you may find that library and the help page in:
tools.jar\com\sun\tools\hat\resources\

The purpose of this code is to show the possibility to integrate jhat usage into your application. Normally it is used as a stand alone tool. The stand alone tool is ready to use, but is a side tool that you have to open. What I did in my app (not the one shown here) was to automate the memory dump via a Runtime.exec call to jmap, and then the resulting file was used as a parameter to instatiate the jhat engine.

Since jhat is an experimental tool, it may completely change or become unavailable on Java 7. Use it like I did at your own risk.

Have fun playing with the code!

The GUI Code:

import java.awt.*;
import java.io.File;
import javax.swing.*;

public class JHatExample extends JFrame {
private JHatWrapper jhat;

public JHatExample(JHatWrapper jhat) {
 super("JHat wrapper");
 this.jhat = jhat;
 initComponents();
}


private void initComponents() {
 final JTextArea queryText = new JTextArea();
 JPanel buttonPanel = new JPanel();
 JButton executeBtn = new JButton("Execute");
 buttonPanel.add(executeBtn);
 this.add(new JScrollPane(queryText),BorderLayout.CENTER);
 this.add(buttonPanel,BorderLayout.SOUTH);
 executeBtn.addActionListener(new ActionListener() {
  public void actionPerformed(ActionEvent e) {
   try {
    jhat.query(queryText.getText());
   } catch (Exception e1) {
    JOptionPane.showMessageDialog(JHatExample.this, "Error executing query");
   }
  }
 });
}

public static void main(String[] args) throws Exception {
 if (args.length == 0 || ! new File(args[0]).exists()) {
  System.out.println("The program must take a valid file path as parameter");
  return;
 }
 JHatWrapper jhat = new JHatWrapper(args[0]);
 JHatExample frame = new JHatExample(jhat);
 frame.setSize(400,200);
 frame.setVisible(true);
 frame.setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE);
}
}

Now the wrapper class:

import java.io.*;
import com.sun.tools.hat.internal.model.Snapshot;
import com.sun.tools.hat.internal.oql.OQLEngine;
import com.sun.tools.hat.internal.parser.HprofReader;
import com.sun.tools.hat.internal.parser.PositionDataInputStream;

public class JHatWrapper {
private OQLEngine engine;

public JHatWrapper(String s) throws Exception {
 PositionDataInputStream positiondatainputstream =
 new PositionDataInputStream(new BufferedInputStream(new FileInputStream(s)));
  int l;
  Snapshot snapshot;
  l = positiondatainputstream.readInt();
   //I "inferred" the meaning of the parameters below by comparing the decompiled code
//and jhat's command line parameters
  int debuglevel = 0;
  //Don't know what this is...
  int j =1;
  HprofReader hprofreader = new HprofReader(s, positiondatainputstream,
j, true, debuglevel);
  snapshot = hprofreader.read();
  positiondatainputstream.close();
  //resolve references
  snapshot.resolve(true);
  engine = new OQLEngine(snapshot);
}

public void query(String query) throws Exception {
  engine.executeQuery(query, new MyQueryVisitor(engine) );
}
}

And the Visitor code:

import java.util.Enumeration;

import com.sun.tools.hat.internal.model.JavaHeapObject;
import com.sun.tools.hat.internal.oql.OQLEngine;
import com.sun.tools.hat.internal.oql.ObjectVisitor;

public class MyQueryVisitor implements ObjectVisitor {
private OQLEngine engine;

public MyQueryVisitor(OQLEngine engine) {
 this.engine = engine;
}

public boolean visit(Object obj) {
 try {
  Object iterator = engine.call("wrapIterator",new Object[]{obj});
  //Handles result items that are instance of enumeration. Ex: select referrers(x) from java.io.File x
  if (iterator instanceof Enumeration) {
   System.out.println("enum");
   handleEnumeration((Enumeration)iterator);
  } else {
   //handles single result items. Ex: select x from java.io.File x
   JavaHeapObject javaObj = (JavaHeapObject)engine.call("unwrapJavaObject",new Object[]{obj});
   handleEnumeration(javaObj.getReferers());
  }
 } catch (Exception e) {
  e.printStackTrace();
 }
 return false;
}
void handleEnumeration(Enumeration en) {
 while (en.hasMoreElements()) {
  try {
   Object obj = en.nextElement();
   JavaHeapObject unwrapped = (JavaHeapObject)engine.call("unwrapJavaObject",new Object[]{obj});
   System.out.println(unwrapped);
  } catch (Exception e) {
   e.printStackTrace();
  }
 }
}
}