Extreme Java When vanilla Java is not enough


The quest for the Java Portal – Profiling Liferay

It's time to run the Profiler. I start JBoss with "run-nbprofiler.bat" (created by NetBeans) and ask NB to attach profiler. I'm using "analyze performance", filtering Java core classes. JBoss runs very slow, but this is normal, since profiler is collecting execution of every method.

My first try showed that Lucene is called a lot of times, even when server is on "idle" state. Maybe it is a background job that should have a smaller frequency.

The second try filters out Lucene classes (using NB Profiler options). I found two bootlenecks: com.liferay.portal.lucene.IndexWriterFactory.decrement (takes 58% of startup time) and JBoss classloaders. And I found that Liferay auto deploy is running too often (~5 seconds of delay). Nice for develoment, awful for production use.

I had to remove JBoss classes from profiling, and I got amazed on things I discovered. I got a OutOfMemoryError (perm gen space) - solved by adding "-XX:MaxPermSize=256m" to run script - but I could find a lot of interesting things:

  1. getResourceAsStream was called 12068 times;
  2. Xerces' ChildNode was instantiated 853808 times (does anybody imagine why Java is "slow"?);
  3. Stacktrace got really deep. About 50 levels or more, not including JBoss, Lucene and JDK classes;
  4. A chain of 7 filters was called before hitting Liferay's MainServlet. If you consider a little forward made after MainServlet, we reach 14 filters before hitting a JSP file;
  5. JSP compilation took 19s on this environment - total execution was 180s;
  6. LR's VelocityTaglib has 8 iconXXX() methods that took 8 seconds each - detail: each one forwards another request;
  7. Everytime a "include" is made, the chain of 7 filter is called. And there's a LOT of includes.

Remember that this was only ONE hit and the request wasn't complete, because of the OOM error. After that, I'm going to have some fun in the "real world". I'll try to go deeper tomorrow. Maybe I can send a RFE to Liferay team after I organize the arguments.


The quest for the Java Portal – Running Liferay

Now, I have a profiled environment. But, when I tried to run JBoss (without profiling), I got an error in counter-ejb module. Its classpath is not correctand, so, I added this line to my build.[user|computer].properties:

classpath.manifest=[original line in build.properties]

I reported this bug on Liferay JIRA [#LEP-2406]. After another 16 minutes of compilation, I forgot to initialize the database. I had two options: use a diferent connection pool or initialize it before deploying Liferay. I prefer the second, so, the easiest way was to create a MBean that depends on LiferayPool. Hypersonic is smart enough to allow multiple SQL commands in one Statement. I will upload the code later, but it is a matter of create a MBean that reads the script provided by Liferay and runs it on the poll.

This leads to an strange error about Spring transactions with EJB, Hibernate, JBoss and all. I don't remember the details, but the solution is to change the debug attribute of CachedConnectionManager. This is something I can't do in my shared JBoss server, so, I used an dirty trick:

  1. Start JBoss with no Liferay, but with the original data source (liferay-db.xml configured);
  2. JBoss translates your -ds file into and -service file and logs it with debug level;
  3. Grab the translated file in log, create the -service, delete the -ds and indent the file - this will help you understand its structure;
  4. Now, the funny part: copy CacheConnectionManager definition from jboss-jca.xml, paste into liferay-service.xml (inside the "CachedConnectionManager" optional attribute), and rename the MBean to an unique name - I put a ",name=xxx" suffix.

This will create a custom CCM to Liferay, without violating the original instance. That's what I love in Java (specially in JBoss): you can create a Lego-like software that is just a matter of do the bindings.

Liferay runs fine, but I found two bugs: a ClassNotFoundException about ical4j, and the contents of Guest community are blank. The first, I solve by adding lib/ical4j.jar to manifest classpath (as above). The second I don't care, since I will clean everything when I deploy the real application.


The quest for the Java Portal – Compiling Liferay

I got some weird exceptions with JBoss Portal, so, I decided to play around with Liferay. I have it running on my shared JBoss, and I have some ideas to their structured articles feature, so I will try harder on Liferay.

I've decided to check the bootlenecks on Liferay, because it is too slow. I'll use NetBeans Profiler. Using NB's ability to create projects using existing Ant scripts, I've done some setup:

  1. Create an Java Application project using an already existing Ant script;
  2. Add all "src" folder to the source folders list - about 20 of them (I guess this isn't necessary unless we want to change anything);
  3. Adjust compilation build to the "start" target and run to "deploy" (Liferay does not set "start" as dependency to "deploy");
  4. Create a "build.[user|computer].properties" and "app.server.[user|computer].properties" to customize some build parameters. I dislike Jikes, so, I'm using "javac.compiler=modern". The rest of properties are straighforward to customize, but folders on app.server must be correctly configured - JBoss predefined values does not work on an out-of-box installation;
  5. After building a lot of modules (about 10 minutes on an almost empty Windows box powered by an Athlon XP 3200+), more than 2000 classes - yes, two thousands - are compiled without errors. Running the "deploy" target will install Liferay on JBoss. It installs some JARs on server's lib folder - I will change this later, before uploading to the real server.

Now, you can run JBoss. But first, some one-time configuration:

  1. Add a datasource. To use profiling, I created a memory-only HSQLDB:
    <?xml version="1.0" encoding="UTF-8"?>
  2. Use NetBeans Profiler (Profile | Attach Profiler) to prepare a special run script to active profiling on JBoss - this file will be called "run-nbprofiler".

Now, we have an working environment. Next step is test and profile.


The quest for the Java Portal – inside Liferay, ep. 1

Liferay is running slowly on my JBoss server, crashes lots of times, but works. I have decided to play with its source code, maybe optimizing here and there. NetBeans Profiler is perfect for this job, but Liferay was written with Eclipse. NetBeans project importer worked, but it didn't help, because the Liferay source structure is monolithic. Since I had some free time this weekend, I'm manually converting the project. I converted about 15%, but look what I found so far:

  • Tab size = 8. This is a HUGE tab size. Usually Java projects use 4. But this is a matter of taste.
  • Actually, this is what I didn't found: documentation. No JavaDocs on classes I opened. I don't know how they can work on classes without any documentation. I always forget the use of any class I don't open often... :)
  • Lots of third part libraries. And I'm not talking about the classic ones (like commons-logging). I found things like EasyConf (looks like Java Preferences API), OSCache (object cache that works on cluster environments) and Trone (yet another Collections framework). This explains the need for a bigger PermSize.

Liferay is a great product, but it really needs a good cleanup. Some of their optimization tricks creates overheads, like using Colt (yet another Collections framework - yes, they have TWO different YACF). Colt have some "sync collections" that works using the same synchronization principle as Hashtable and Vector. The question is: why they leave synchronization to the collection? What about using the "synchronized(xxx)" structure?

I will follow GrOG's tip (in last blog's comments): report my discoveries to the community. I preferred to report only after deciding which Portal I will use, but the decision is taking me too deep on each implementation I'm testing.


Thumbs down to Java Portals

One of the most frustrating things I have tryed to do was to use a Java Portal. I haven't found a JSR-168 compliant portal that could work at all. I have simple needs:

  1. The portal must run on JBoss: I'm renting a shared JBoss server;
  2. I need to put a "virtual-host" tag in every jboss-web.xml: this is how JBoss will map my domain to my applications;
  3. I must have total control over JNDI/JMX naming conventions: "default" names are unacceptable, since other "clients" may want to use the same implementation;
  4. The portal must NOT create any files on the filesystem: JBoss runs with an specific user that does not have write access to my private folders; and
  5. Memory and CPU consumption must be light: obviously.

What I found:

  • JBoss Portal:
    • PrĂ³s: Runs on JBoss (of course) and seems to be light
    • Cons: UGLY. Portal layout is defined with some weird arrows on an weird admin page
  • Liferay
    • Pros: beautiful, lots of AJAX, lots of out-of-box portlets
    • Cons: invasive (needs to create a lot of files), eats CPU and memory like I eat chocolate and creates JMX in runtime (but does not release them on undeploy). "Professional" version (WAR) takes 30 minutes to deploy, "Enterprise" version (EAR) crashed the server (yikes) and requires a lot of customization to run on a shared server
  • eXo
    • Pros: like Liferay, it's beautiful, lots of AJAX, lots of portlets, and have an awesome "desktop" version
    • Cons: I have to compile EVERYTHING and do a lot of digging on its source to customize it to my needs. I simply can't understand its compilation steps. Poor documentation. I could only test tomcat version, since I could not compile an working version...
  • Gridsphere
    • Pros: dunno
    • Cons: runs only on Tomcat

This is amazing... With JavaEE version 5, I thought I could find a portal that could be deployed as easy as 1-2-3:

  1. Download a WAR/EAR file;
  2. Add some jboss-web.xml (to map JNDI and virtual host) - this is my "deployer" role on an JavaEE application lifecycle (as defined by JavaEE specs);
  3. Deploy the file on server.

But step 1 is non-existent on portals like GridSphere. Step 2 becames steps 2.a to 2.zzzz on eXo and Liferay... Every portal wants the user to download a package bundled with tomcat/jboss or to follow this recipe:

  1. Download source (WORA, anyone?)
  2. Compile using Maven (yet-another-build-automation-tool)
  3. Package using Maven (its "custom" tasks does not allow me to add a jboss-web.xml, since even web.xml is generated by it)
  4. Deploy on tomcat (JBoss sometimes)

Why?? Is that hard to create a REAL JavaEE-compliant application that implements JSR-168??? That's a thumb down to Java Portals (and JSR-168)...