Extreme Java When vanilla Java is not enough


The quest for the Java Portal – Profiling Liferay

It's time to run the Profiler. I start JBoss with "run-nbprofiler.bat" (created by NetBeans) and ask NB to attach profiler. I'm using "analyze performance", filtering Java core classes. JBoss runs very slow, but this is normal, since profiler is collecting execution of every method.

My first try showed that Lucene is called a lot of times, even when server is on "idle" state. Maybe it is a background job that should have a smaller frequency.

The second try filters out Lucene classes (using NB Profiler options). I found two bootlenecks: com.liferay.portal.lucene.IndexWriterFactory.decrement (takes 58% of startup time) and JBoss classloaders. And I found that Liferay auto deploy is running too often (~5 seconds of delay). Nice for develoment, awful for production use.

I had to remove JBoss classes from profiling, and I got amazed on things I discovered. I got a OutOfMemoryError (perm gen space) - solved by adding "-XX:MaxPermSize=256m" to run script - but I could find a lot of interesting things:

  1. getResourceAsStream was called 12068 times;
  2. Xerces' ChildNode was instantiated 853808 times (does anybody imagine why Java is "slow"?);
  3. Stacktrace got really deep. About 50 levels or more, not including JBoss, Lucene and JDK classes;
  4. A chain of 7 filters was called before hitting Liferay's MainServlet. If you consider a little forward made after MainServlet, we reach 14 filters before hitting a JSP file;
  5. JSP compilation took 19s on this environment - total execution was 180s;
  6. LR's VelocityTaglib has 8 iconXXX() methods that took 8 seconds each - detail: each one forwards another request;
  7. Everytime a "include" is made, the chain of 7 filter is called. And there's a LOT of includes.

Remember that this was only ONE hit and the request wasn't complete, because of the OOM error. After that, I'm going to have some fun in the "real world". I'll try to go deeper tomorrow. Maybe I can send a RFE to Liferay team after I organize the arguments.


The quest for the Java Portal – Running Liferay

Now, I have a profiled environment. But, when I tried to run JBoss (without profiling), I got an error in counter-ejb module. Its classpath is not correctand, so, I added this line to my build.[user|computer].properties:

classpath.manifest=[original line in build.properties]

I reported this bug on Liferay JIRA [#LEP-2406]. After another 16 minutes of compilation, I forgot to initialize the database. I had two options: use a diferent connection pool or initialize it before deploying Liferay. I prefer the second, so, the easiest way was to create a MBean that depends on LiferayPool. Hypersonic is smart enough to allow multiple SQL commands in one Statement. I will upload the code later, but it is a matter of create a MBean that reads the script provided by Liferay and runs it on the poll.

This leads to an strange error about Spring transactions with EJB, Hibernate, JBoss and all. I don't remember the details, but the solution is to change the debug attribute of CachedConnectionManager. This is something I can't do in my shared JBoss server, so, I used an dirty trick:

  1. Start JBoss with no Liferay, but with the original data source (liferay-db.xml configured);
  2. JBoss translates your -ds file into and -service file and logs it with debug level;
  3. Grab the translated file in log, create the -service, delete the -ds and indent the file - this will help you understand its structure;
  4. Now, the funny part: copy CacheConnectionManager definition from jboss-jca.xml, paste into liferay-service.xml (inside the "CachedConnectionManager" optional attribute), and rename the MBean to an unique name - I put a ",name=xxx" suffix.

This will create a custom CCM to Liferay, without violating the original instance. That's what I love in Java (specially in JBoss): you can create a Lego-like software that is just a matter of do the bindings.

Liferay runs fine, but I found two bugs: a ClassNotFoundException about ical4j, and the contents of Guest community are blank. The first, I solve by adding lib/ical4j.jar to manifest classpath (as above). The second I don't care, since I will clean everything when I deploy the real application.


The quest for the Java Portal – Compiling Liferay

I got some weird exceptions with JBoss Portal, so, I decided to play around with Liferay. I have it running on my shared JBoss, and I have some ideas to their structured articles feature, so I will try harder on Liferay.

I've decided to check the bootlenecks on Liferay, because it is too slow. I'll use NetBeans Profiler. Using NB's ability to create projects using existing Ant scripts, I've done some setup:

  1. Create an Java Application project using an already existing Ant script;
  2. Add all "src" folder to the source folders list - about 20 of them (I guess this isn't necessary unless we want to change anything);
  3. Adjust compilation build to the "start" target and run to "deploy" (Liferay does not set "start" as dependency to "deploy");
  4. Create a "build.[user|computer].properties" and "app.server.[user|computer].properties" to customize some build parameters. I dislike Jikes, so, I'm using "javac.compiler=modern". The rest of properties are straighforward to customize, but folders on app.server must be correctly configured - JBoss predefined values does not work on an out-of-box installation;
  5. After building a lot of modules (about 10 minutes on an almost empty Windows box powered by an Athlon XP 3200+), more than 2000 classes - yes, two thousands - are compiled without errors. Running the "deploy" target will install Liferay on JBoss. It installs some JARs on server's lib folder - I will change this later, before uploading to the real server.

Now, you can run JBoss. But first, some one-time configuration:

  1. Add a datasource. To use profiling, I created a memory-only HSQLDB:
    <?xml version="1.0" encoding="UTF-8"?>
  2. Use NetBeans Profiler (Profile | Attach Profiler) to prepare a special run script to active profiling on JBoss - this file will be called "run-nbprofiler".

Now, we have an working environment. Next step is test and profile.


The quest for the Java Portal – Testimonial

I was supposed to be an user of Java Portals, not a developer. Things like JBoss management portlet are annoying, but it is just a matter of user experience. At least JBoss Portal deploys with minimal changes. Actually, I only need to change deployment descriptors.

Portals like Gridsphere, Jetspeed and others are unusable to normal users, since they need to know Java very well. How can I replace Zope/Plone in my job if every environment admin will need to know how to use Maven/Ant/etc?

Is that hard to develop an JavaEE application that makes users' life easier?

If I need to do any developer task, I prefer to develop my own portal that conforms to JSR-168 without violating JavaEE specifications.

BTW, I'm talking as a Java Portal user (i.e. portal deployer/admin). I'm not talking as the user of the portal I will create with a Java Portal (i.e. my "costumers").

End note: I'll try JBoss Portal beta-1, since it was released this week, but I won't use it if I still needs that buggy management layout.

Tagged as: , No Comments

The quest for the Java Portal – Gridsphere Tests

GridSphere compiled in an zero-error-in-deploy WAR file. But, with my well-known luck and my incredible skill of catch the weirdest bugs on Earth, I got the classic "error-page-that-throws-error" bug. A page throwed an error. This triggers the error page. But, the same error was throwed. This triggers the error page. I love when this happens. My luck was the StackOverflowError. In some EPTTE bugs the error page leads to HTTP infinite redirection.

The origin of the error? This little piece of code:

public synchronized Object deepCopy(Object oldObj) throws Exception {
  ObjectOutputStream oos = null;
  ObjectInputStream ois = null;
  try {
    ByteArrayOutputStream bos = new ByteArrayOutputStream(); // A
    oos = new ObjectOutputStream(bos); // B
    // serialize and pass the object
    oos.writeObject(oldObj); // C
    oos.flush(); // D
    ByteArrayInputStream bin =
      new ByteArrayInputStream(bos.toByteArray()); // E
    ois = new ObjectInputStream(bin); // F
    // return the new object
    return ois.readObject(); // G
  } catch (Exception e) {
    throw e;
  } finally {
    if (oos != null) oos.close();
    if (ois != null) ois.close();

I will not comment the weird "catch e throw e" fragment, but this was the most expensive way to follow the Prototype pattern (deep copy variant) I ever seen. You don't need to be an extreme Java programmer to notice this code will fail on the first non-serializable class found by oos.writeObject. Now, guess what was the exception I got?

java.io.NotSerializableException: org.apache.log4j.Level

After digging into the code, I discovered that GS team knows the use of transient keyword, but didn't used it on BasePortletComponent.java, line 42:

protected Log log = LogFactory.getLog(PortletPageFactory.class);

How this works on Tomcat is a mistery to me. Changing it to a final static attribute and adding some dependencies of dependencies (Jackrabbit needs Lucene and yet another logging implementation - SLF4J), made it work, but not as expected.

Result: an installation page with some really ugly URL rewrite techniques.

<link ... href="/tmp54563GridSphere-exp.war/...css" ...>
<img src="/@GRIDSPHERE_DEPLOY@/...png" ...>

The second one is a side-effect of using my own build script. The first one (along with cool design patterns) gives GridSphere an one-way ticket to my attic.


The quest for the Java Portal – Gridsphere Compilation

Current version of Gridsphere, 3.0.3, supports Tomcat and Jetty only. Since I'm extreme (-lly nuthead), I've tryed to run it on JBoss. I haven't found any live demo on its site - too bad.

Like Jetspeed and eXo, there's no WAR/EAR/SAR/CAR/etcAR file to download - I must download the source and compile (there's a corrupted, hidden WAR file on server, if you edit by hand the URL of download). Good news is: it uses Ant (not Maven) - that means I does not need to dig into obscure and unstructured XML. IMHO, Maven is good for the main developer, but not for the bug hunters, like me.

As expected, it wasn't straightforward. The steps:

  1. Set CATALINA_HOME to JBoss' server folder - "$JBOSSHOME/server/$instance". Compilation script needs a "lib" folder with JavaEE API. Tomcat 5 stores on "$C_H/common/lib", Tomcat 6 stores on "$C_H/lib". Gridsphere "detects" tomcat version by verifying if "$C_H/common/lib" exists.
  2. This leads to a "$C_H/bin" not found. After some digging in build scripts (I love Ant), I found an strange task that adds Tomcat's bin dir to classpath:
    <fileset dir="${appserver.home}/bin/">
        <include name="*.jar">
  3. Removing it and running the undocumented "ant war" task, creates the web archive.
  4. The archive will not deploy: a ClassNotFoundException nested in a WSException. The class that was not found was the CaptchaServlet. The CNFE is strange, since the class is on "WEB-INF/lib" folder. Since I will not allow new users, I removed the reference from web.xml, but the same exception was thrown, with class GridSphereServlet.

At this point, I got tired of editing files by hand, and I created a NetBeans project to handle everything to me. This is the approach I'm doing with Liferay, and, likewise, I found a lot of dependencies, like Castor (yet another binding framework), JFreeChart, and others.

BTW, JFreeChart deserves some lines. First, I thought one of JFree JAR files were corrupt in GS distribution. I've decided to download it from JFree site and I can't see the documentation because IT IS NOT FREE. Yup. I can download the sources, but not the developer documentation.

And, like every giant project, I found useless things, like Perl5Util. I don't want to discuss if P5U is useful, but use it to replace regex like "\.xml$" with "" is a waste of resources, since String.replaceFirst can do the job.

Finally, I could compile with zero errors (and a lot of unchecked warnings). But it didn't run. I got an weird exception in Castor. After some reverse engineering, since there isn't architetural documentation, I found I need to add six XML files to their correct folders. In original GS structure, these XML lies on a "config" dir, but are copied to different folders by one of the 10 Ant scripts. Copying them made a deployable WAR file.

This is a thumbs up no NetBeans: users will always have an well organized project structure. There are no lost files on lost folders needing a lost task on a lost script to copy them to the correct place.

BTW, the problem was a static Class.getResource returning null. I had to change the code (adding some assertions) to figure it out.

Tip to Gridsphere team: use NetBeans to manage the project...


The quest for the Java Portal – inside Liferay, ep. 1

Liferay is running slowly on my JBoss server, crashes lots of times, but works. I have decided to play with its source code, maybe optimizing here and there. NetBeans Profiler is perfect for this job, but Liferay was written with Eclipse. NetBeans project importer worked, but it didn't help, because the Liferay source structure is monolithic. Since I had some free time this weekend, I'm manually converting the project. I converted about 15%, but look what I found so far:

  • Tab size = 8. This is a HUGE tab size. Usually Java projects use 4. But this is a matter of taste.
  • Actually, this is what I didn't found: documentation. No JavaDocs on classes I opened. I don't know how they can work on classes without any documentation. I always forget the use of any class I don't open often... :)
  • Lots of third part libraries. And I'm not talking about the classic ones (like commons-logging). I found things like EasyConf (looks like Java Preferences API), OSCache (object cache that works on cluster environments) and Trone (yet another Collections framework). This explains the need for a bigger PermSize.

Liferay is a great product, but it really needs a good cleanup. Some of their optimization tricks creates overheads, like using Colt (yet another Collections framework - yes, they have TWO different YACF). Colt have some "sync collections" that works using the same synchronization principle as Hashtable and Vector. The question is: why they leave synchronization to the collection? What about using the "synchronized(xxx)" structure?

I will follow GrOG's tip (in last blog's comments): report my discoveries to the community. I preferred to report only after deciding which Portal I will use, but the decision is taking me too deep on each implementation I'm testing.


The quest for the Java Portal – JBoss Portal 2.6 live

No changes in JBoss Portal 20070307 nightly build. I saw JBoss ajax4jsf and JBoss Rich Faces and could not imagine why these incredibly cool components aren't used by JBoss Portal.

Tagged as: , 2 Comments

The quest for the Java Portal – JBoss Portal 2.6

JBoss Portal version 2.6 is still alpha, but I tryed it (since it claims to have a better user experience). I've downloaded its binary version - a JBoss' SAR archive. Out-of-box installation was a matter of copy datasource definition and the SAR file to my deploy folder.

It has the same limitations as 2.4:

  • Delete the management portlet and your layout is locked;
  • You can delete ALL users and have a nice NPE;
  • The layout arrangement is not inline (needs the management portlet).

But, as I expected, it is pretty light. Unlink Liferay, the startup was really quick. If it was more AJAX-aware, JBoss Portal could be my portal of choice. Well, according to Grog, beta 1 will be released soon. I'm downloading a nightly build and I'll try it again.

Tagged as: , 1 Comment

The quest for the Java Portal – JBoss Portal

Today mission: JBoss Portal. My first impression: negative. I've tryed its live demo and, after two minutes, I render it useless. How? Follow the steps:

  1. Login as admin/admin
  2. Go to the admin page
  3. Remove the administration portlet

What you got? A portal with a layout that can't be changed... That's typical from JBoss community: they create a simple kernel and everything is extension. JBoss Application Server is a microkernel that manages JMX beans. Everything else is JMX. Likewise, JBoss Portal is just a portlet container. You need that ugly and unintuitive manager (as ugly and unintuitive as AS managers) to manage everything.

BTW, that management portlet does a lot of roundtrips to the server (yep, no AJAX). If you plan to use it on your server, create a page that contains only the manager portlet.

JBoss Portal 2.4 is off my list. I will try 2.6alpha2.

Tagged as: , 2 Comments