Extreme Java When vanilla Java is not enough


The quest for the Java Portal – Profiling Liferay

It's time to run the Profiler. I start JBoss with "run-nbprofiler.bat" (created by NetBeans) and ask NB to attach profiler. I'm using "analyze performance", filtering Java core classes. JBoss runs very slow, but this is normal, since profiler is collecting execution of every method.

My first try showed that Lucene is called a lot of times, even when server is on "idle" state. Maybe it is a background job that should have a smaller frequency.

The second try filters out Lucene classes (using NB Profiler options). I found two bootlenecks: com.liferay.portal.lucene.IndexWriterFactory.decrement (takes 58% of startup time) and JBoss classloaders. And I found that Liferay auto deploy is running too often (~5 seconds of delay). Nice for develoment, awful for production use.

I had to remove JBoss classes from profiling, and I got amazed on things I discovered. I got a OutOfMemoryError (perm gen space) - solved by adding "-XX:MaxPermSize=256m" to run script - but I could find a lot of interesting things:

  1. getResourceAsStream was called 12068 times;
  2. Xerces' ChildNode was instantiated 853808 times (does anybody imagine why Java is "slow"?);
  3. Stacktrace got really deep. About 50 levels or more, not including JBoss, Lucene and JDK classes;
  4. A chain of 7 filters was called before hitting Liferay's MainServlet. If you consider a little forward made after MainServlet, we reach 14 filters before hitting a JSP file;
  5. JSP compilation took 19s on this environment - total execution was 180s;
  6. LR's VelocityTaglib has 8 iconXXX() methods that took 8 seconds each - detail: each one forwards another request;
  7. Everytime a "include" is made, the chain of 7 filter is called. And there's a LOT of includes.

Remember that this was only ONE hit and the request wasn't complete, because of the OOM error. After that, I'm going to have some fun in the "real world". I'll try to go deeper tomorrow. Maybe I can send a RFE to Liferay team after I organize the arguments.


The quest for the Java Portal – Running Liferay

Now, I have a profiled environment. But, when I tried to run JBoss (without profiling), I got an error in counter-ejb module. Its classpath is not correctand, so, I added this line to my build.[user|computer].properties:

classpath.manifest=[original line in build.properties]

I reported this bug on Liferay JIRA [#LEP-2406]. After another 16 minutes of compilation, I forgot to initialize the database. I had two options: use a diferent connection pool or initialize it before deploying Liferay. I prefer the second, so, the easiest way was to create a MBean that depends on LiferayPool. Hypersonic is smart enough to allow multiple SQL commands in one Statement. I will upload the code later, but it is a matter of create a MBean that reads the script provided by Liferay and runs it on the poll.

This leads to an strange error about Spring transactions with EJB, Hibernate, JBoss and all. I don't remember the details, but the solution is to change the debug attribute of CachedConnectionManager. This is something I can't do in my shared JBoss server, so, I used an dirty trick:

  1. Start JBoss with no Liferay, but with the original data source (liferay-db.xml configured);
  2. JBoss translates your -ds file into and -service file and logs it with debug level;
  3. Grab the translated file in log, create the -service, delete the -ds and indent the file - this will help you understand its structure;
  4. Now, the funny part: copy CacheConnectionManager definition from jboss-jca.xml, paste into liferay-service.xml (inside the "CachedConnectionManager" optional attribute), and rename the MBean to an unique name - I put a ",name=xxx" suffix.

This will create a custom CCM to Liferay, without violating the original instance. That's what I love in Java (specially in JBoss): you can create a Lego-like software that is just a matter of do the bindings.

Liferay runs fine, but I found two bugs: a ClassNotFoundException about ical4j, and the contents of Guest community are blank. The first, I solve by adding lib/ical4j.jar to manifest classpath (as above). The second I don't care, since I will clean everything when I deploy the real application.


The quest for the Java Portal – Compiling Liferay

I got some weird exceptions with JBoss Portal, so, I decided to play around with Liferay. I have it running on my shared JBoss, and I have some ideas to their structured articles feature, so I will try harder on Liferay.

I've decided to check the bootlenecks on Liferay, because it is too slow. I'll use NetBeans Profiler. Using NB's ability to create projects using existing Ant scripts, I've done some setup:

  1. Create an Java Application project using an already existing Ant script;
  2. Add all "src" folder to the source folders list - about 20 of them (I guess this isn't necessary unless we want to change anything);
  3. Adjust compilation build to the "start" target and run to "deploy" (Liferay does not set "start" as dependency to "deploy");
  4. Create a "build.[user|computer].properties" and "app.server.[user|computer].properties" to customize some build parameters. I dislike Jikes, so, I'm using "javac.compiler=modern". The rest of properties are straighforward to customize, but folders on app.server must be correctly configured - JBoss predefined values does not work on an out-of-box installation;
  5. After building a lot of modules (about 10 minutes on an almost empty Windows box powered by an Athlon XP 3200+), more than 2000 classes - yes, two thousands - are compiled without errors. Running the "deploy" target will install Liferay on JBoss. It installs some JARs on server's lib folder - I will change this later, before uploading to the real server.

Now, you can run JBoss. But first, some one-time configuration:

  1. Add a datasource. To use profiling, I created a memory-only HSQLDB:
    <?xml version="1.0" encoding="UTF-8"?>
  2. Use NetBeans Profiler (Profile | Attach Profiler) to prepare a special run script to active profiling on JBoss - this file will be called "run-nbprofiler".

Now, we have an working environment. Next step is test and profile.


The quest for the Java Portal – inside Liferay, ep. 1

Liferay is running slowly on my JBoss server, crashes lots of times, but works. I have decided to play with its source code, maybe optimizing here and there. NetBeans Profiler is perfect for this job, but Liferay was written with Eclipse. NetBeans project importer worked, but it didn't help, because the Liferay source structure is monolithic. Since I had some free time this weekend, I'm manually converting the project. I converted about 15%, but look what I found so far:

  • Tab size = 8. This is a HUGE tab size. Usually Java projects use 4. But this is a matter of taste.
  • Actually, this is what I didn't found: documentation. No JavaDocs on classes I opened. I don't know how they can work on classes without any documentation. I always forget the use of any class I don't open often... :)
  • Lots of third part libraries. And I'm not talking about the classic ones (like commons-logging). I found things like EasyConf (looks like Java Preferences API), OSCache (object cache that works on cluster environments) and Trone (yet another Collections framework). This explains the need for a bigger PermSize.

Liferay is a great product, but it really needs a good cleanup. Some of their optimization tricks creates overheads, like using Colt (yet another Collections framework - yes, they have TWO different YACF). Colt have some "sync collections" that works using the same synchronization principle as Hashtable and Vector. The question is: why they leave synchronization to the collection? What about using the "synchronized(xxx)" structure?

I will follow GrOG's tip (in last blog's comments): report my discoveries to the community. I preferred to report only after deciding which Portal I will use, but the decision is taking me too deep on each implementation I'm testing.