Shahzad Bhatti Welcome to my ramblings and rants!

I am about to release another J2EE web application and I have been profiling and load testing the application before the deployment. In my last blog I showed how I am collecting the profiling data. I found that after running load testing with increasingly number of users, the system becomes slower and eventually crashes. In past, I have used a throttling module in Apache to stop accepting new connections, but it didn’t take into memory/cpu into account. I am going to add a throttling to the Java server (Tomcat) so that the system will reject any new requests when the memory becomes too low or system load becomes very high. Luckily, we are using Java 6 which has added a number of nice JMX APIs for collecting system information, for example I have added following

import java.lang.management.ManagementFactory;
...
   long freeMemory = ManagementFactory.getMemoryMXBean().getHeapMemoryUsage().getMax() -
                ManagementFactory.getMemoryMXBean().getHeapMemoryUsage().getUsed();

   double loadAverage = ManagementFactory.getOperatingSystemMXBean().getSystemLoadAverage();

These parameters were enough for my work, though I also tracked other information such as CPU time by thread:

        int threads = ManagementFactory.getThreadMXBean().getThreadCount();
        long[] threadIds = ManagementFactory.getThreadMXBean().getAllThreadIds();
        StringBuilder threadInfo = new StringBuilder();
        for (long id : threadIds) {
            threadInfo.append("thread " + id + " cputime " +  ManagementFactory.getThreadMXBean().getThreadCpuTime(id) +
                ", usertime " + ManagementFactory.getThreadMXBean().getThreadUserTime(id));
        }

Also system start time:

            Date started = new Date(ManagementFactory.getRuntimeMXBean().getStartTime());
            long uptime ManagementFactory.getRuntimeMXBean().getUptime();

and other system information:

            List jvmArgs = ManagementFactory.getRuntimeMXBean().getInputArguments();
            int cpus = ManagementFactory.getOperatingSystemMXBean().getAvailableProcessors();

Java 6 also comes with plenty of command line tools to monitor server such as

Memory Usage

The jstat command can be used to monitor the memory usage and garbage collection statistics as follows:

jstat -gcutil
  

In order to get heap histogram use jmap as follows:

jmap -histo:live pid

Creating memory dump automatically when running out of memory with following command option:

java -XX:+HeapDumpOnOutOfMemoryError

You can use jhat to analyze heap as follows:

jhat heap.dump.out

Monitoring threads

Use jstack to dump stack traces of all threads as follows:

jstack pid

Use JTop and JConsole to monitor the application, e.g.

java -jar /demo/management/JTop/JTop.jar