Shahzad Bhatti

February 15, 2008

Throttling Server in Java

Filed under: Computing,Java — admin @ 4:11 pm

I am about to release another J2EE web application and I have been profiling and load testing the application before the deployment. In my last blog I showed how I am collecting the profiling data. I found that after running load testing with increasingly number of users, the system becomes slower and eventually crashes. In past, I have used a throttling module in Apache to stop accepting new connections, but it didn’t take into memory/cpu into account. I am going to add a throttling to the Java server (Tomcat) so that the system will reject any new requests when the memory becomes too low or system load becomes very high. Luckily, we are using Java 6 which has added a number of nice JMX APIs for collecting system information, for example I have added following

import java.lang.management.ManagementFactory;
...
   long freeMemory = ManagementFactory.getMemoryMXBean().getHeapMemoryUsage().getMax() -
                ManagementFactory.getMemoryMXBean().getHeapMemoryUsage().getUsed();

   double loadAverage = ManagementFactory.getOperatingSystemMXBean().getSystemLoadAverage();

These parameters were enough for my work, though I also tracked other information such as CPU time by thread:

        int threads = ManagementFactory.getThreadMXBean().getThreadCount();
        long[] threadIds = ManagementFactory.getThreadMXBean().getAllThreadIds();
        StringBuilder threadInfo = new StringBuilder();
        for (long id : threadIds) {
            threadInfo.append("thread " + id + " cputime " +  ManagementFactory.getThreadMXBean().getThreadCpuTime(id) +
                ", usertime " + ManagementFactory.getThreadMXBean().getThreadUserTime(id));
        }

Also system start time:


            Date started = new Date(ManagementFactory.getRuntimeMXBean().getStartTime());
            long uptime ManagementFactory.getRuntimeMXBean().getUptime();

and other system information:

            List jvmArgs = ManagementFactory.getRuntimeMXBean().getInputArguments();
            int cpus = ManagementFactory.getOperatingSystemMXBean().getAvailableProcessors();

Java 6 also comes with plenty of command line tools to monitor server such as

Memory Usage

The jstat command can be used to monitor the memory usage and garbage collection statistics as follows:

jstat -gcutil
  

In order to get heap histogram use jmap as follows:

jmap -histo:live pid

Creating memory dump automatically when running out of memory with following command option:

java -XX:+HeapDumpOnOutOfMemoryError

You can use jhat to analyze heap as follows:

jhat heap.dump.out

Monitoring threads

Use jstack to dump stack traces of all threads as follows:

jstack pid

Use JTop and JConsole to monitor the application, e.g.

java -jar /demo/management/JTop/JTop.jar

February 14, 2008

Israel ethnic cleansing Palestine village to make way for Jewish settlement

Filed under: Palestine,Politics — admin @ 6:42 pm

Tracking server health in Java

Filed under: Computing,Java — admin @ 5:27 pm

In order to deploy a scalable application, it is necessary to know how the application behaves with varying number of users. I am about to release another web application at work so I have been profiling and load testing it. I wrote a blog entry a year ago on Load and Functional Testing with Selenium and Grinder. Today, I am showing somewhat simpler things that can be done on J2EE application to keep track of its health.

Data Collector

Though, when working for big companies, often we used commercial software such as complex event processor or messaging middleware to keep track of all kind of alerts and events. But here I am showing a simple class that uses LRU based map to keep track of profile data:

  1 import java.lang.management.ManagementFactory;

  2 import java.util.ArrayList;
  3 import java.util.Collections;
  4 import java.util.Date;

  5 import java.util.List;
  6
  7 import com.amazon.otb.cache.CacheMap;
  8 

  9 /**
 10  * Profile DataCollector used to store profiling data.

 11  */
 12 public class ProfileDataCollector {
 13     private static ProfileDataCollector instance = new ProfileDataCollector();

 14     private CacheMap<String, Object> map;
 15     private ProfileDataCollector() {
 16         map = new CacheMap<String, Object>(1000, 300, null);

 17         map.put("ProfileStarted", new Date());
 18     }
 19
 20     public static ProfileDataCollector getInstance() {

 21         return instance;
 22     }
 23
 24     public void add(String name, Object value) {

 25         map.put(name, value);
 26     }
 27     public long increment(String name) {
 28         Long number = (Long) map.get(name);

 29         if (number == null) {
 30             number = new Long(1);

 31         } else {
 32             number = new Long(number.longValue()+1);
 33         }

 34         add(name, number);
 35         return number.longValue();
 36     }
 37 

 38     public long decrement(String name) {
 39         Long number = (Long) map.get(name);
 40         if (number == null) {

 41             number = new Long(0);
 42         } else {
 43             number = new Long(number.longValue()-1);

 44         }
 45         add(name, number);
 46         return number.longValue();
 47     }

 48
 49     public long sum(String name, long total) {
 50         Long number = (Long) map.get(name);

 51         if (number == null) {
 52             number = new Long(total);
 53         } else {

 54             number = new Long(number.longValue()+total);
 55         }
 56         add(name, number);
 57         return number.longValue();

 58     }
 59
 60
 61     public long average(String name, long elapsed) {

 62         long number = increment(name + "_times");
 63         long sum = sum(name + "_total", elapsed);
 64         long average = sum / number;

 65         add(name + "_average", average);
 66         return average;
 67     }
 68 

 69     public void lapse(String name) {
 70         add(name, new Long(System.currentTimeMillis()));
 71     }

 72
 73     public long elapsed(String name) {
 74         Long started = (Long) map.get(name);
 75         if (started != null) {

 76             long time = System.currentTimeMillis() - started.longValue();
 77             return average(name, time);
 78         } else {

 79             return -1;
 80         }
 81     }
 82 

 83     public String[][] getProfileData(String keyPrefix) {
 84         List<String> keys = new ArrayList<String>(map.keySet()); 

 85         Collections.sort(keys);
 86         String[][] data = new String[keys.size()+5][];
 87         Runtime runtime = Runtime.getRuntime();

 88         Date started = (Date) map.get("ProfileStarted");
 89         long elapsed = (System.currentTimeMillis() - started.getTime()) / 1000;
 90         int n = 0;

 91         double systemLoad = ManagementFactory.getOperatingSystemMXBean().getSystemLoadAverage();
 92         data[n++] = new String[] {keyPrefix + "TotalMemoryInMegs", String.valueOf(runtime.totalMemory()/1024/1024)};

 93         data[n++] = new String[] {keyPrefix + "FreeMemoryInMegs", String.valueOf(runtime.freeMemory()/1024/1024)};
 94         data[n++] = new String[] {keyPrefix + "ActiveThreads", String.valueOf(Thread.activeCount())};

 95         data[n++] = new String[] {keyPrefix + "ServerRunningInSecs", String.valueOf(elapsed)};
 96         data[n++] = new String[] {keyPrefix + "SystemLoadAverage", String.valueOf(systemLoad)};

 97
 98         for (String key : keys) {
 99             CacheMap.TimedValue tv = map.getTimedValue(key);
100             data[n++] = new String[] {keyPrefix + key, tv.value + " @" + new Date(tv.time)};

101         }
102         return data;
103     }
104 }
105
106

Where CacheMap is a simple LRU based map, i.e.,

  1 import java.util.ArrayList;

  2 import java.util.Collection;
  3 import java.util.Collections;
  4 import java.util.concurrent.Callable;

  5 import java.util.concurrent.Executors;
  6 import java.util.concurrent.ExecutorService;
  7 import java.util.concurrent.locks.ReentrantLock;

  8 import java.util.HashMap;
  9 import java.util.HashSet;
 10 import java.util.Iterator;

 11 import java.util.LinkedHashMap;
 12 import java.util.List;
 13 import java.util.Map;

 14 import java.util.Set;
 15 import org.apache.log4j.Logger;
 16
 17 

 18
 19 /**
 20  * CacheMap - provides lightweight caching based on LRU size and timeout

 21  * and asynchronous reloads.
 22  *

 23  */
 24 public class CacheMap<K, V> implements Map<K, V> {

 25     private final static Logger log = Logger.getLogger(CacheMap.class);
 26     private final static int MAX_THREADS = 10; // for all cache items across VM

 27 private final static int MAX_ITEMS = 1000; // for all cache items across VM
 28     private final static ExecutorService executorService = Executors.newFixedThreadPool(MAX_THREADS);

 29     private final static boolean lockSync = false;
 30 

 31     public class TimedValue<V> {
 32         public final long time;

 33         public V value;
 34         private boolean updating = false;
 35         TimedValue(V value) {

 36             this.value = value;
 37             this.time = System.currentTimeMillis();
 38         }
 39         public boolean isExpired(long timeoutInSecs) {

 40             long timeDiff = System.currentTimeMillis() - time;
 41             return timeDiff > timeoutInSecs * 1000;
 42         }

 43         public synchronized boolean markUpdating() {
 44             if (!this.updating) {

 45                 this.updating = true;
 46                 return true;
 47             }

 48             return false;
 49         }
 50
 51         @Override

 52         public String toString() {
 53             long timeDiff = System.currentTimeMillis() - time;
 54             return "TimedValue(" + value + ") expiring in " + timeDiff;

 55         }
 56     }
 57
 58
 59     class FixedSizeLruLinkedHashMap<K, V> extends LinkedHashMap<K, V> {

 60         private final int maxSize;
 61
 62         public FixedSizeLruLinkedHashMap(int initialCapacity, float loadFactor, int maxSize) {

 63             super(initialCapacity, loadFactor, true);
 64             this.maxSize = maxSize;
 65         }
 66 

 67         @Override
 68         protected boolean removeEldestEntry(Map.Entry<K, V> eldest) {
 69             return size() > maxSize;

 70         }
 71     }
 72
 73     private final Cacheable classCacheable;

 74     private final Map<K, TimedValue<V>> map;
 75     private final Map<Object, ReentrantLock> locks;

 76     private CacheLoader<K, V> defaultCacheLoader;
 77     private final long expireEntriesPeriod;

 78     private long lastexpireEntriesTime;
 79
 80
 81     public CacheMap(Cacheable cacheable) {

 82         this(cacheable, null, 5);
 83     }
 84
 85     public CacheMap(int timeoutInSecs, CacheLoader<K,V> defaultCacheLoader) {

 86         this(0, timeoutInSecs, defaultCacheLoader);
 87     }
 88
 89     public CacheMap(int maxCapacity, int timeoutInSecs, CacheLoader<K,V> defaultCacheLoader) {

 90         this(new CacheableImpl(maxCapacity, timeoutInSecs, true, false), defaultCacheLoader, 5);
 91     }

 92
 93     public CacheMap(Cacheable cacheable, CacheLoader<K,V> defaultCacheLoader, long expireEntriesPeriodInSecs) {
 94         this.classCacheable = cacheable;

 95         this.defaultCacheLoader = defaultCacheLoader;
 96         this.expireEntriesPeriod = expireEntriesPeriodInSecs * 1000;
 97         int maxCapacity = cacheable != null && cacheable.maxCapacity() > 0 && cacheable.maxCapacity() < MAX_ITEMS ? cacheable.maxCapacity() : MAX_ITEMS;

 98         this.map = Collections.synchronizedMap(new FixedSizeLruLinkedHashMap<K, TimedValue<V>>(maxCapacity/ 10, 0.75f, maxCapacity));
 99         this.locks = new HashMap<Object, ReentrantLock>();

100     }
101
102     public void setDefaultCacheLoader(CacheLoader<K,V> defaultCacheLoader) {
103         this.defaultCacheLoader = defaultCacheLoader;

104     }
105
106     public int size() {
107         return map.size();

108     }
109
110     public boolean isEmpty() {
111         return map.isEmpty();

112     }
113
114     public boolean containsKey(Object key) {
115         return map.containsKey(key);

116     }
117
118     public boolean containsValue(Object value) {
119         return map.containsValue(value);

120     }
121
122     public V put(K key, V value) {
123         TimedValue<V> old = map.put(key, new TimedValue<V>(value));

124         if (old != null) {
125             return old.value;
126         } else {

127             return null;
128         }
129     }
130
131     public V remove(Object key) {

132         TimedValue<V> old = map.remove(key);
133         if (old != null) {
134             return old.value;

135         } else {
136             return null;
137         }
138     }

139
140
141     public void putAll(Map<? extends K, ? extends V> m) {

142         for (Entry<? extends K, ? extends V> e : m.entrySet()) {
143             put(e.getKey(), e.getValue());

144         }
145     }
146
147
148     public void clear() {

149         map.clear();
150     }
151
152
153     public Set<K> keySet() {

154         return map.keySet();
155     }
156
157
158     public Collection<V> values() {

159         List<V> list = new ArrayList<V>(map.size());
160         for (TimedValue<V> e : map.values()) {

161             list.add(e.value);
162         }
163         return list;
164     }
165
166     public Set<Map.Entry<K,V>> entrySet() {

167         Set<Map.Entry<K,V>> set = new HashSet<Entry<K,V>>();
168         for (final Map.Entry<K, TimedValue<V>> e : map.entrySet()) {

169             set.add(new Map.Entry<K,V>() {
170                 public K getKey() {
171                     return e.getKey();

172                 }
173                 public V getValue() {
174                     return e.getValue().value;
175                 }
176                 public V setValue(V value) {

177                     TimedValue<V> old = e.getValue();
178                     e.getValue().value = value;
179                     if (old != null) {
180                         return old.value;

181                     } else {
182                         return null;
183                     }
184                 }

185             });
186         }
187         return set;
188     }
189
190     public TimedValue<V> getTimedValue(Object key) {

191         return this.map.get(key);
192     }
193
194     public V get(Object key) {
195         V value = null;

196         if (classCacheable != null && defaultCacheLoader != null) {
197             value = get((K)key, classCacheable, defaultCacheLoader, null);
198         } else {

199             TimedValue<V> item = this.map.get(key);
200             if (item == null) {
201                 value = null;

202             } else {
203                 value = item.value;
204             }
205         }
206         return value;

207     }
208
209     public V get(K key, Cacheable cacheable, CacheLoader<K, V> loader, boolean[] wasCached) {
210         TimedValue<V> item = this.map.get(key);

211         V value = null;
212         ReentrantLock lock = null;
213
214         //expire old entries
215         if (System.currentTimeMillis() - this.lastexpireEntriesTime > this.expireEntriesPeriod) {

216             expireEntries(cacheable);
217         }
218
219
220         try {
221             synchronized(this) {

222                 if (lockSync && cacheable.synchronizeAccess()) {
223                     lock = lock(key);
224                 }
225             }
226             //

227             if (item == null) {
228                 // load initial value
229                 value = reloadSynchronously(key, loader);
230
231             } else {

232
233                 value = item.value;
234
235                 boolean cached = true;
236                 if (cacheable.timeoutInSecs() > 0 && item.isExpired(cacheable.timeoutInSecs())) {

237
238                     if (! cacheable.canReloadAsynchronously()) {
239                         // ---> reload it now - don't used cached value
240                         cached = false;

241                         log.debug("Reloading expired entry synchronously " + key);
242                         value = reloadSynchronously(key, loader);
243                     } else if (item.markUpdating()) {

244                         log.debug("Reloading expired entry asynchronously " + key);
245                         reloadAsynchronously(key, loader);
246                     }
247                 }
248                 if (wasCached != null) {

249                     wasCached[0] = cached;
250                 }
251             }
252         } finally {

253             if (lock != null) {
254                 lock.unlock();
255                 locks.remove(key);
256                 //log.debug("Unlocking " + key);

257             }
258         }
259         return value;
260     }
261
262 

263     @Override
264     public String toString() {
265         return super.toString() + "--" + map;

266     }
267
268
269     private ReentrantLock lock(Object key) {
270         ReentrantLock lock = null;

271         synchronized (locks) {
272             lock = locks.get(key);
273             if (lock == null) {
274                 lock = new ReentrantLock();

275                 locks.put(key, lock);
276             }
277         }
278         //log.debug("Locking " + key);
279         lock.lock();

280         return lock;
281     }
282
283     private V reloadSynchronously(final K key, final CacheLoader<K, V> loader) {

284         try {
285             V value = loader.loadCache(key);
286             put(key, value);
287             //log.info("------reloadSynchronously loaded key " + key + ", cache size: " + this.size() + " -- " + System.identityHashCode(map));

288             return value;
289         } catch (Exception e) {
290             log.error("Failed to load " + key, e);

291             throw new RuntimeException("Failed to load " + key + " for " + classCacheable, e);
292         }

293     }
294
295
296     private void reloadAsynchronously(final K key, final CacheLoader<K,V> loader) {

297         if (log.isDebugEnabled()) {
298             log.debug("requesting reload for "+key.toString()+": "+Thread.currentThread().getName());
299         }
300         executorService.submit(new Callable<V>() {

301             public V call() throws Exception {
302                 if (log.isDebugEnabled()) {
303                     log.debug("reloading for "+key.toString()+": "+Thread.currentThread().getName());

304                 }
305                 return reloadSynchronously(key, loader);
306             }
307         });
308     }

309
310     private void expireEntries(final Cacheable cacheable) {
311         Iterator<TimedValue<V>> it = new ArrayList<TimedValue<V>>(this.map.values()).iterator();

312         while (it.hasNext()) {
313             TimedValue<V> value = it.next();
314             if (cacheable.timeoutInSecs() > 0 && value.isExpired(cacheable.timeoutInSecs())) {

315                 it.remove();
316             }
317         }
318         this.lastexpireEntriesTime = System.currentTimeMillis();
319     }
320 }

321
322

Session Tracking

I added a listener to keep track of active users/sessions (who accessed the system in last 5 mintutes) as opposed to all sessions which may take 30 minutes or more to expire.

 1 public class SessionLogger implements HttpSessionListener, HttpSessionBindingListener {

 2     private transient static Log log = LogFactory.getLog(SessionLogger.class);
 3     private static int sessionCount;

 4     private static final String OBJECTS_IN_SESSION_COUNT = "Objects in Session";
 5     private static final long ACTIVE_THRESHOLD = 5 * 60 * 1000; // 5 minutes

 6     private static Map<HttpSession, HttpSession> activeSessions = Collections.synchronizedMap(new HashMap<HttpSession, HttpSession>());
 7     private static Map<HttpSession, String> allUsers = Collections.synchronizedMap(new HashMap<HttpSession, String>());

 8     private static Map<HttpSession, String> activeUsers = Collections.synchronizedMap(new HashMap<HttpSession, String>());
 9 

10     private static void checkActiveSessions() {
11         long now = System.currentTimeMillis();
12         Iterator<HttpSession> it = activeSessions.keySet().iterator();

13         activeUsers.clear();
14         while (it.hasNext()) {
15             HttpSession session = it.next();
16             if (now-session.getLastAccessedTime() > ACTIVE_THRESHOLD) {

17                 it.remove();
18             } else {
19                 activeUsers.put(session,(String) session.getAttribute(UserUtil.OTB_REAL_USER));
20             }
21         }

22         ProfileDataCollector.getInstance().add("TotalSessions", String.valueOf(sessionCount));
23         ProfileDataCollector.getInstance().add("ActiveSessions", String.valueOf(activeSessions.size()));
24         ProfileDataCollector.getInstance().add("ActiveUsers", activeUsers.values().toString());
25         ProfileDataCollector.getInstance().add("AllUsers", allUsers.values().toString());

26     }
27
28     public void sessionCreated(HttpSessionEvent se) {
29         HttpSession session = se.getSession();
30         synchronized (this) {

31             sessionCount++;
32             activeSessions.put(session, session);
33             activeSessions.put(session, session);
34             checkActiveSessions();
35         }
36         allUsers.put(session, (String) session.getAttribute(UserUtil.OTB_REAL_USER));

37     }
38
39     public void sessionDestroyed(HttpSessionEvent se) {
40         HttpSession session = se.getSession();
41         allUsers.remove(session);

42         synchronized (this) {
43             sessionCount--;
44             activeSessions.remove(session);
45             checkActiveSessions();
46         }

47     }
48
49     public void valueBound(HttpSessionBindingEvent event) {
50         String username = (String) event.getSession().getAttribute(UserUtil.OTB_REAL_USER);
51         Integer old = (Integer) event.getSession().getAttribute(OBJECTS_IN_SESSION_COUNT);

52         if (old == null) {
53             old = new Integer(0);
54         }

55         Integer count = new Integer(old.intValue()+1);
56         event.getSession().setAttribute(OBJECTS_IN_SESSION_COUNT, count);
57         ProfileDataCollector.getInstance().add("TotalSessionValues", String.valueOf(count));
58     }

59
60     public void valueUnbound(HttpSessionBindingEvent event) {
61         String username = (String) event.getSession().getAttribute(UserUtil.OTB_REAL_USER);
62         Integer old = (Integer) event.getSession().getAttribute(OBJECTS_IN_SESSION_COUNT);
63         if (old == null) {

64             old = new Integer(0);
65         }
66         Integer count = new Integer(old.intValue()-1);

67         event.getSession().setAttribute(OBJECTS_IN_SESSION_COUNT, count);
68         ProfileDataCollector.getInstance().add("TotalSessionValues", String.valueOf(count));
69     }
70 }
71

Publishing profiling data

I then added code that needs to be profiled. Though, I have used AspectJ in past for this kind of work, but for now I am just adding this code where needed, e.g.

 1 try {

 2     ProfileDataCollector.getInstance().increment("ActiveSearches");
 3     ProfileDataCollector.getInstance().lapse("SearchLapse");
 4 ...

 5
 6 } finally {
 7      ProfileDataCollector.getInstance()..increment("TotalSearches");
 8      ProfileDataCollector.getInstance()..decrement("ActiveSearches");

 9      ProfileDataCollector.getInstance()..elapsed("SearchLapse");
10 }

Viewing Profile Data

Finally, I added a servlet to return profiling data with AJAX call, e.g.

  1 import java.io.IOException;

  2
  3 import javax.servlet.ServletException;
  4 import javax.servlet.http.HttpServlet;
  5 import javax.servlet.http.HttpServletRequest;

  6 import javax.servlet.http.HttpServletResponse;
  7 public class ProfileServlet extends HttpServlet {

  8     protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
  9         response.setContentType("text/plain");

 10         String format = request.getParameter("format");
 11         if ("line".equals(format)) {
 12             response.getWriter().println(getProfileLine());

 13         } else if ("json".equals(format)) {
 14             response.getWriter().println(getProfileJson());
 15         } else {

 16             response.getWriter().println(getProfileTable());
 17         }
 18     }
 19
 20     protected void doGet(HttpServletRequest req, HttpServletResponse rsp) throws ServletException, IOException {

 21         doPost(req, rsp);
 22     }
 23
 24
 25     public static String[][] getProfileData() {

 26         return ProfileDataCollector.getInstance().getProfileData("");
 27     }
 28
 29     public static String getProfileLine() {

 30         StringBuilder sb = new StringBuilder();
 31         String[][] data = getProfileData();
 32         for (String[] param: data) {

 33             sb.append(param[0] + "=" + param[1]);
 34         }
 35         return sb.toString();

 36     }
 37
 38     public static String getProfileTable() {
 39         StringBuilder sb = new StringBuilder("<table width='100%' border='2'>");

 40         String[][] data = getProfileData();
 41         sb.append("<tr>");
 42         for (String[] param: data) {
 43             sb.append("<th>" + param[0] + "</th>");

 44         }
 45         sb.append("</tr>");
 46         sb.append("<tr>");
 47         for (String[] param: data) {

 48             sb.append("<td>" + param[1] + "</td>");
 49         }
 50         sb.append("</tr>");

 51         sb.append("</table>");
 52         return sb.toString();
 53     }
 54 

 55     public static String getProfileJson() {
 56         StringBuilder sb = newJsonString();
 57         String[][] data = getProfileData();
 58         for (String[] param: data) {

 59             appendJsonString(sb, param[0], param[1]);
 60         }
 61         return endJsonString(sb);

 62     }
 63
 64
 65     protected static StringBuilder newJsonString() {

 66         return new StringBuilder('[');
 67     }
 68 

 69     protected static void appendJsonString(StringBuilder sb, String name, String id) {
 70         if (sb.length() > 1) {

 71             sb.append(',');
 72         }
 73         sb.append("{ name:'").append(name.replace("'", "\'")).append("', id:'").append(id.replace("'", "\'")).append(

 74         "' }");
 75     }
 76
 77     protected static String endJsonString(StringBuilder sb) {

 78         sb.append(']');
 79         return sb.toString();
 80     }
 81 

 82     protected static void addJsonProperty(StringBuilder sb, String name, String value) {
 83         sb.append(""" + name + """);

 84         sb.append(":");
 85         sb.append(""" + value + """);
 86     }

 87
 88     protected static void startJsonObj(StringBuilder sb) {
 89         sb.append("{");

 90
 91     }
 92
 93     protected static void addNewJsonProperty(StringBuilder sb) {

 94         sb.append(",");
 95
 96     }
 97
 98     protected static void endJsonObj(StringBuilder sb) {

 99         sb.append("}");
100
101     }
102 }
103

And then a JSP to show the results using AJAX calls with Prototype library:

       <script src="/otb-static/scriptaculous/prototype.js" type="text/javascript"></script>

        <script>
        var updater = null;
        function initAjax() {
            updater = new Ajax.PeriodicalUpdater('profile_div', '/profile?format=table', {
                method: 'get',
                insertion: Insertion.Top,
                frequency: 15,
                decay: 2
                });
        }
        function startRequest() {
            updater.start();
        }
        function stopRequest() {
            updater.stop();
        }
        </script>
   </head>
    <body onLoad="initAjax(), startRequest()">
    <ul id="profile_div">
    </ul>

Testing

I then wrote a unit test to call various actions of the web application, I am going to use my old pal Grinder to do some real load testing and monitor the health of the server. I am not showing the test here, because it’s very application specific.

February 9, 2008

I am a muslim

Filed under: Politics — admin @ 9:54 am

February 7, 2008

Iraqi woman and her son

Filed under: iraq,Politics — admin @ 11:16 am


February 5, 2008

Does experience matter?

Filed under: Computing — admin @ 1:27 pm

I came across a blog entry of DHH, Years of irrelevance
and and an old blog post of Jeff Atwood on Skill Disparities in Programming. Both of them cover somewhat similar topic, i.e. are programmers with more experience better than programmers with less experience. For example, DHH asserts that after six month of experience with a technology you can be pretty experienced. Similarly, Jeff shows various research findings where person with just two year experience is just as good as person with seven or more years of experience. This topic often comes up with job postings when companies require applicant to have certain number of years of experience. Martin Fowler also spoke recently in his blog entry PreferDesignSkills, where he prefers person with broad experience in design and programming than a specialized person. I have seen my share of discrimination in job market when recruiter is looking for X years of experience or is only looking for person with Websphere and would not consider Weblogic experience. It is even worse when new technology is involved and I faced similar discrimination when transitioning from J2EE to Rails. I totally agree that requiring experience with some technology does not matter because a smart person can easily learn it. This also has been discussed many times by Scott Ambler in Generalists vs Specialists, which I have quoted in my website for many years. Since, there is a considerable amount of difference between productivity and quality between programmers, this is an important question. Based on my twenty years of programming experience with sixteen years doing professionally I agree with these findings. I have seen people with twenty years of experience doing same job in big companies with no desire to learn anything else. I have learned it’s incredibly important to diversify your skills and be a generalist than specialist. It is the only way a programmer with more years of experience can be more valuable than programmer with fewer years of experience. When hiring I would look for a person with broad skills who has track record of getting things done and has passion to learn new things. As Joel Spolsky often says when hiring look for smart people who get the things done. There are always new things coming and a good programmer will find a way to learn that in short amount of time as DHH mentioned. I have worked as software developer and systems engineer/administrator and that helped my understanding of overall systems. I have used various languages over the years including Basic, FORTRAN, COBOL, C, C++, Java, PERL, Python, Ruby, Erlang, Haskell. As Dave Thomas talks about learning new language to think differently, and it can show you finding new solutions.

Besides being generalist, another way experience matters is with design. Though with small system, productivity and quality of experienced and junior programmer may seem similar, but I have found with larger systems quality does show up. (Note, I am only considering smart programmers with varying number of years here and not considering really bad programmers.) I have observed that design of experienced programmer (more than five years of experience) will be more flexible because he/she would have likely worked on similar problem before (which may also give productivity advantage). I have also found junior programmers struggle with roles and responsibilities (Rebecca J Wirfs) and good object oriented design (or domain driven design). I have observed that senior programmers have better understanding of things like separation of concerns, modularization, encapsulation, loose coupling and scalability.

In nutshell, any good programmer can learn a new technology very quickly and can solve any problem, but I believe experienced programmer with more general experience will be more valuable than junior programmer and quality of design of a senior programmer with broad experience will be much better in terms of good design principles and -ilities. I think, design is something you learn over the years because for each design decision you may be using thousands of small lessons you have learned over the years. As far as development teams are concerned, I like to have one really experienced programmer and others with junior to mid level experience. This gives a good apprenticeship environment for junior people to learn as software development is still an art. Finally, as everything in software, there aren’t hard rules and everything depends on the environment and people.

Powered by WordPress