Shahzad Bhatti Welcome to my ramblings and rants!

July 22, 2009

Day 1 at #oscon 2009

Filed under: Computing — admin @ 5:25 pm

The first day of OSCon 2009 covered a number of tutorials and I decided to attend Google App Engine tutorial for the first half of the day. Google App Engine API follows CGI model of web development, i.e., it uses stdin and stdout files and assumes stateless applications. There is limit of 10MB response size and 30 requests per second, but it does not allow streaming. The tutorial started pretty slow and we spent first hour just installing the SDK and tutorial. The Google App Engine SDK is available from http://code.google.com/appengine/downloads.html. I downloaded Mac image and then dragged the image to my hard drive. I then double clicked the app icon for Google Appe Engine SDK, which installed the SDK under /usr/local/google_appengine. Once the SDK is installed, you have to install Google App Engine tutorials from http://code.google.com/p/app-engine-tutorial/.

After installing SDK and tutorial, I copied all files named tutorial? under the SDK. The rest of session covered those tutorials one by one, though we ran out of time in the end and completed only upto tutorial7. In order to install first tutorial, I went into tutorial1 directory, e.g.

 cd /usr/local/google_appengine/tutorial1
 

Then started local app server as follows:

 python ../dev_appserver.py .
 

When I pointed my browser to the http://localhost:8080, I was able to see “Hello World!”.

Next, I registered myself to http://appspot.com. After registering, I received an SMS message for confirmation and was able to fully register after entering the confirmation number. Next, I created an application-id on Google App Engine. You can only create 10 app-ids and you cannot delete app-ids, so be careful with ids. Also, you can also use your own domain instead appspot.com. For my testing purpose, I chose the id “shahbhat”.

Next, I changed app.yaml inside my local tutorial1 directory that describes how your application is configured. You may also notice index.yaml, which describes list of indices in the database, though Google App Engine can figure out what queries are being used and creates indices automatically. I changed application name in app.yaml to “shahbhat”, e.g.

 application: shahbhat
 

I then pushed my application to the Google App Engine by typing

 python ../appcfg.py update .
 

I was then able to go to http://shahbhat.appspot.com/ and see my application, Voila. You can also see your application usage from http://appengine.google.com/dashboard?app_id=shahbhat (you will have to change app_id parameter in your application).

Unfortunately, a lot of people had problems getting to that state so we wasted another half hour in break where other folks sort out configuration and deployment issues.
Next, I went through another tutorial to turn on authentication by setting:

 login: required
 

in app.yaml file. Next I added caching by adding expires options in the app.yaml. I was also able to use curl to test my applications and see headers to verify caching, e.g.

 curl --include http://localhost:8080
 

Which showed following when caching was not configured:

 Cache-Control: no-cache
 

When I configured the caching to 2d, I was able to see:

 Cache-Control: public, max-age=172800
 

The Google App Engine SDK also includes development that you can view by going to:

 http://localhost:8080/_ah/admin
 

The Google App Engine supports Django based templates, e.g.

 #!/usr/bin/env python
   
   import os
   from google.appengine.ext.webapp import template
   
   def main():
       template_values = {"foo" : [1,2,3]}
       template_file = os.path.join(
                    os.path.dirname(__file__), "index.html")
       body = template.render(
         template_file, template_values)
       print "Status: 200 OK"
       print "Content-type: text/html"
       print
       print body
   
   if __name__ == '__main__':
     main()
 

In addition, Google App Engine supports WSGI standard (PEP 333), e.g.

   import os
   import wsgiref.handlers
   
   from google.appengine.ext import webapp
   from google.appengine.ext.webapp import template
   
   class IndexHandler(webapp.RequestHandler):
   
     def get(self):
       template_values = {"foo" : 1}
   
       template_file = os.path.join(os.path.dirname(__file__), "index.html")
       self.response.out.write(template.render(template_file, template_values))
   
   
   def main():
     application = webapp.WSGIApplication([('/', IndexHandler)], debug=True)
     wsgiref.handlers.CGIHandler().run(application)
   
   
   if __name__ == '__main__':
     main()
 

Other tutorials included authentication APIs such as

 create_login_url(dest_url)
 create_logout_url(dest_url)
 get_current_user()
 is_current_user_admin()
 

The SDK also includes decorator to add authentication automitcally using

 from gogole.appengine.ext.webapp.util import login_required
 ...
     @login_required
     def get(self):
 

Finally, we went over datastore APIs for persistence support, e.g.

   import os
   import wsgiref.handlers
   
   from google.appengine.ext import webapp
   from google.appengine.ext.webapp import template
   from google.appengine.ext import db
   
   class ToDoModel(db.Model):
     description = db.StringProperty()
     created = db.DateTimeProperty(auto_now_add=True)
     foo = db.FloatProperty(default=3.14)
     bar = db.IntegerProperty()
     baz = db.BooleanProperty(default=False)
     N = db.IntegerProperty()
     l = db.ListProperty(str, default=["foo", "bar"])
   
   
   class IndexHandler(webapp.RequestHandler):
   
     def get(self):
       todo = ToDoModel(description = "Hello World", bar=1, baz=True)
       todo.put()
   
   def main():
     application = webapp.WSGIApplication([('/', IndexHandler)], debug=True)
     wsgiref.handlers.CGIHandler().run(application)
   
   if __name__ == '__main__':
     main()
 

You can view the model by going to http://localhost:8080/_ah/admin/datastore. The data store supports a number of types such as string, boolean, blob, list, time, text. However, there are some limitations,
e.g. TextProperty can only store upto 500 bytes and Google App Engine will create index if needed, however it won’t create index on TextProperty. For each row, the datastore assigns a numeric id and UUID based key,
though you can provide your own key. Also, a row cannot exceed 1MB.
Unfortunately, we ran out of time At this time, so I had to go to http://code.googlecom/appengne/docs for further documentation. Overall, I thought it was good introduction to Google App Engine, but I was disappointed that instructor wasted a lot of time with setup that could have been used to cover rest of the tutorials.

For the second half of the day, I attended session on “Building applications with XMPP”. This was interesting session that showed usage of XMPP for a number of usecases such as IM, gaming, real-time social networking, , monitoring, etc. The session started with history of XMPP (Extensible Messaging and Presnce Protocol), its Jabber roots and its use of streaming XML. A number of factors contributed to the popularity of XMPP such as open source, XML, federeated network, low latency, etc. The XMPP is also very extensible and supports audio/video via jingle, geo location, TLS, SASL, etc. XMPP architecture is based on client server, where servers are decentralized and federated. XMPP identifies a user with jabber id that looks like email address and consists of local part, domain and resource, e.g. alise@wonderland.lit/TeaParty, where domain is mandatory, but local-part and resource are optional. There are three types of XMPP message stanzas, i.e., presence, IQ, message. The presence-stanza is asynchronous, but IQ stanza requires response. As opposed to web architecture that is based on short lived connections and stateless architecture, XMPP uses one long lived session and events are received asynchronously.

Next, the tutorial showed how to build a javascript client for jabber using sleekxmpp and Strophe library (alternatively you can use twistedword). The example used Bosh protocol to wrap XMPP protocol with HTTP protocol. Unfortunately, there was a lot of fast typing and I could not follow all that. I am waiting for the presenters to post the slides online so that I can use those examples in my own applications.

June 15, 2009

My plan B for being a programmer

Filed under: Computing — admin @ 11:29 am

I came across an interesting blog on Programmers: Before you turn 40, get a plan B, where the author sites age discrimination in programming career. The author shows how in IT industry, technology shift every 10 years levels the plainfield so there is a little difference between junior and senior programmer. The author also asserts that more experienced programmer will have a lot of irrelevent experience that employer may not be willing to consider when hiring or may not pay higher salary for those skills. I have been programming for over twenty two years and working professionally for over seventeen years and I certainly have seen effects of technology shift. In early 90s, I worked for a physics lab that had plenty of VAX computers using FORTRAN, and they were migrating most of the system to UNIX so people with VMS skills had to adopt to UNIX. I then had to work on mainframe system for another company and used COBOL and CICS. I saw plenty of people who worked on mainframe system for over 10-15 years and knew a little outside mainframe technology. When that company also migrated to NT and UNIX platform, it was quite stressful to many people. Though, there are still companies who are willing to pay top dollars for mainframe experience but opportunities are very few. I myself had to adopt and learn C++ and object-oriented design and found other opportunties that used those skills. I was young and unmarried so switching to another technology was not hard. In mid 90s I switched to Java, CORBA and Web development and there was some learning and adopting involved. In late 90s and early 2000s, I started using Java Enterprise Beans (EJBs) and SOAP based Web services. From 2003-onward, I started using more open source technologies like Spring, Hibernate, Tomcat, REST, etc and I had to abandoned my past CORBA or EJB expertise. This was followed by rise of dynamic languages and I started using Ruby on Rails and built a couple of sites using it. After that I became interested functional programming such as Erlang. These days, I have been spending my spare time in iPhone development.

I completely agree that in programming career, the past experience becomes irrelevant and most companies won’t pay me for the irrelevant experience such as VMS, mainframe, CORBA or EJBs. Also, as I became older and started family, switching jobs just for technology became a lot harder. I believe experienced programmers who have worked with varied technologies bring useful perspective and a bit of pessimism that is often needed. I also believe most experienced programmers have better design skills and pay more attention to writing maintainable or scalable code, but it probably doesn’t matter in may places that just want to get shit done.

Another aspect that differentiates programming career from other professions is lack of predictability. I have found that in our industry, project scope, requirements and estimates are poorly defined and most companies expect programmers to be heros or martyrs who will abandoned their life and dedicate completely to the work. IT people also have to pay for the global culture of 24×7 service and have to be oncall when things break. This becomes a lot harder with the family so many companies prefer young and unmarried folks who are more willing to be heros and those young programmers like the extra attention.

So what is my plan B? It is to continually learn new skills and be a little better than the competition. Many of my past colleagues have moved to the dark site and now are directors or senior managers at other companies. And truth is that in most companies managers carry more respect and authority so no wonder most programmers aspire to become one. I on the other hand, just can’t give up programming and usually spend my spare time on a number of small projects. So I will continue to be a lowly code monkey for the rest of my life.

June 13, 2009

ActiveObject based O/R Mapping library for iPhone development

Filed under: iPhone development — admin @ 6:38 pm

I have been learning iPhone development lately and needed to add some persistence capability to my application. There are varied options available for persistence such as using “User Defaults” for small user specific settings, serialization similar to serialization or pickle features of other languages and builtin support of embeded Sqlite. I found Sqlite more performant, memory efficient and flexible than other options so I chose it. Using Sqlite with iPhone is fairly straight forward and there are tons of examples such Creating Todo list using Sqlite. However, when I looked for some O/R mapping framework for iPhone, I found that iPhone SDK unlike Mac development didn’t have any support and neither I could find any other solution elsewhere. So, I started writing a simple O/R mapping library based on Active Object pattern. This is similar to how Rails and Django implement O/R mapping. Based on convention over configuration, it simply maps object properties to the database table fields. At this time, this is very basic O/R mapping library and does not support relations, validation, database integrity support, etc. Nevertheless, it met my simple needs and I have released it as open source project under OCActiveObjects on GitHub.

The OCActiveObjects library is fairly small and consists of following classes:

ActiveObject

This is the base class that you extend in order to add automatic O/R support. You have to override following Class methods to specify name of the database and table:

     + (NSString *) getTableName;
   

Above method defines name of table where instances of the object will be stored.

     + (NSString *) getDatabaseName;
   

Above method defines name of the database to be used. You will then be able to call following methods to interact with the Sqlite database:

   + (void) openDatabase;
   

Above method must be called once before any other methods, usually at the start of your application.

   + (void) closeDatabase;
   

Above method must be called once before you shutodnw your application.

  
   - (void) save;
 

Above method saves a new object or updates an existing object. Each subclass of ActiveObject is automatically assigned a unique database id with a property
named “objectId”. This is another example of convention where all tables will use a numeric surrogate key to identify each row.
If that property is nil then it assumes this is new object and inserts a new row in the database, otherwise it updates an existing row in the
database. It assumes that name of database fields are same as property names, though you can override that behavior by overriding _getPropertyNamesAndTypes Class method.

   + (ActiveObject *) findByPrimaryKey:(NSNumber *)objId;
 

Above method queries an object in the database matching objectId property, which identifies each object in the database.

   + (NSArray *) findWithCriteria:(NSDictionary *)criteria;
 

Above method returns an array of objects that match criteria. The criteria at this time is simple dictionary, i.e., pair of name and values that are joined by
“AND” clause. There is a immediate need to extend this to support more flexible queries.

   + (NSArray *) findAll;
 

Above method returns all objects, which may not be good for iPhone application due to limited amount of memory. This is another area that needs immediate attention.

   + (int) removeAll;
 

Above method removes all rows in the table so be careful with this.

   + (int) removeWithCriteria:(NSDictionary *)criteria;
 

Above method removes only methods matching criteria. Again criteria consists of name/value pairs.

   + (int) countWithCriteria:(NSDictionary *)criteria;
 

Above method counts the number of rows in the database matching criteria.

   + (int) countAll;
 

Above method returns count of all rows in the table.

Exension Methods in ActiveObject

There are number of extension methods to customize SQLs or behavior of the object such as

 - (void) _insert;
 

Above method inserts an object into the database.

 - (void) _update;
 

Above method updates an existing object into the database.

 + (NSDictionary *) _getPropertyNamesAndTypes;
 

You can override above method to change properties that needs to be persisted.

 + (NSString *) _getCreateSQL;
 

Above method generates an SQL for creating table.

 + (NSMutableString *) _getInsertSQL;
 

Above method generates an SQL for inserting a row in the table.

 + (NSMutableString *) _getUpdateSQL;
 

Above method generates an SQL for updating a row in the table.

 + (NSMutableString *) _getSelectSQL;
 

Above method generates an SQL for selecting fields from the database.

 + (void) _createTable;
 

Above method creates database table.

IntrospectHelper

The OCActiveObjects library uses some Objective C magical runtime support to query for properties and this class encapsulates those methods.

SqliteHelper

This class some helper methods for Sqlite3.

How to use

In order to test it, let’s define a simple Person class that extends ActiveObject, e.g.

 #import <Foundation/Foundation.h>
 #import "ActiveObject.h"
 
 @interface Person : ActiveObject {
         NSString *name;
         short age;
         int rank;
         long votes;
         char sex;
         double income;
         BOOL active;
         NSInteger flags;
         NSNumber *rating;
         NSDate *birthdate;      
 }
 
 
 @property (nonatomic, retain) NSString *name;
 @property (nonatomic, assign) short age;
 @property (nonatomic, assign) int rank;
 @property (nonatomic, assign) long votes;
 @property (nonatomic, assign) char sex;
 @property (nonatomic, assign) double income;
 @property (nonatomic, assign) BOOL active;
 @property (nonatomic, assign) NSInteger flags;
 @property (nonatomic, retain) NSNumber *rating;
 @property (nonatomic, retain) NSDate *birthdate;
 
 - (BOOL)isEqualToPerson:(Person *)aPerson;
 
 @end
 

Implemention of Person.m looks like:

 #import "Person.h"
 
 @implementation Person
 @synthesize name;
 @synthesize age;
 @synthesize rank;
 @synthesize votes;
 @synthesize sex;
 @synthesize income;
 @synthesize active;
 @synthesize flags;
 @synthesize rating;
 @synthesize birthdate;
 
 - (BOOL)isEqual:(id)other {
     if (other == self)
         return YES;
     if (!other || ![other isKindOfClass:[self class]])
         return NO;
     return [self isEqualToPerson:other];
 }
 
 - (BOOL)isEqualToPerson:(Person *)aPerson {
         if (self == aPerson)
         return YES;
     if (![(id)[self name] isEqual:[aPerson name]])
         return NO;
     return YES;
 }
 
 - (NSUInteger)hash {
         NSUInteger hash = 0;
         hash += [[self name] hash];
         return hash;
 }
 
 
 
 - (NSString *)description {
         return [NSString stringWithFormat:@"id %@, name %@", self.objectId, self.name);
 }
 
 
 - (void) dealloc {
         [name release];
         [birthdate release];
         [super dealloc];
 }
 
 + (NSString *) getTableName {
         return @"persons";
 }
 
 + (NSString *) getDatabaseName {
         return @"personsdb";
 }
 
 @end
 

Then you can first open the database, e.g.

 
         [Person openDatabase];
 

Then create a new person object, e.g.

         Person *person = [[[Person alloc] init] autorelease];
         person.birthdate = [[NSDate alloc]init];
         int random = [person.birthdate timeIntervalSince1970];
         person.age =  random % 30;
         person.rank = random % 20;
         person.votes = random % 10;
         person.sex = 'M';
         person.name = [NSString stringWithFormat:@"Joe #%d", random % 1000];
         person.income = random % 3000;
         person.active = YES;
         person.flags = random % 30 + 0.5;
         person.rating = [NSNumber numberWithInt:20.5];  
         return person;
 

You will then be able to save the person object as

         [person save];
 

You can see how many rows are in the database by

         int count = [Person countAll];
 

And then retrieve the object that we saved as

         Person *person2 = (Person *) [Person findByPrimaryKey:person.objectId];
 

When you are done, you can then close the database:

         [Person closeDatabase];
 

One of the frustrating aspect of iPhone development has been lack of good unit testing support. Though, XCode comes with OCUnit, but it is hard to install and use with iPhone. I kept getting weird errors like:

 exited abnormally with code 139
 
 failed tests for architecture 'i386'
 

Though, there are some basic tutorials like Test Driving Your Code with OCUnit or OCUnit: Integrated Unit Testing In Xcode, but they didn’t help. I also tried adding google-toolbox-for-mac but macro errors are extremely frustrating. Besides better testing, OCActiveObjects needs a lot of help to add better support of criteria, paging, relational mapping and validation. I also had hard time figuring out how to create a static library until I found Building static libraries with the iPhone SDK though I still need help in adding framework level support. Hopefuly, other people can contribute to the open source project. You can send me your suggestions and comments as well via email “bhatti AT plexobject DOT com” or tweet me at bhatti_shahzad.

May 26, 2009

Tracing memory leaks with command line Java tools

Filed under: Java — admin @ 3:42 pm

I was recently tracking memory leak in one of our applicatios, however the problem was only occuring in our production environment and the network latency between production subnet and my local subnet was too high to run profiler (I tried). So, I had to use some other tools to look at the snapshot of memory that was in use. I found jmap and jhat that are shipped with JDK (1.6 in our case) pretty handy. When memory usage was high, I dumped the memory using jmap, e.g.

 jmap -F -dump:file=/tmp/dump.out 
 

I then copied the file to my local machine and ran jhat using

 jhat dump.out
 

Though, jhat is fairly slow, but it finally starts the web server on port 7000 and you can then point your browser to

 http://mymachine:7000
 

The jhat can also read data from hprof that you can dump by adding following vm option:

 -agentlib:hprof=heap=sites
 

and then read the file using jhat. In addition to above tool, you can run garbage collection in verbose mode using following vm option:

 -verbose:gc -verbose:class -Xloggc:gc.out -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
 

Finally, you can use jconsole to look at memory graph by adding support of jmx to your application server, .e.g.

 -Dcom.sun.management.jmxremote.port=9003 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote
 

Despite all these options, I find profiler is much better for finding memory leaks, but these tools helped a bit.

May 6, 2009

Integrating Springs event notification with JMS

Filed under: Java — admin @ 3:56 pm
I recently had to deploy a service that needed deployment in cluster fashion, however I wanted to synchronize some cache between the instances of servers. Since that service was making some use of Spring, I decided to leverage Spring’s builtin event notification with some glue to convert those Spring events into JMS based messaging. First, I defined an event that I can use in the application. As Map of Strings to Strings seemed simple, I used it to store message properties, e.g.

  1 import org.springframework.context.ApplicationEvent;
 
  2 import org.apache.commons.lang.builder.ToStringBuilder;
  3 import org.apache.commons.lang.builder.ToStringStyle;
  4 import java.util.Map;
 
  5 import java.util.HashMap;
  6 import java.util.Collection;  
  7                                                                   
  8 public class MessageEvent extends ApplicationEvent {       
 
  9     private String msg;
 10     private Map<String, String> map;
 11     public MessageEvent(final Object source, final String ... properties) {
 
 12         super(source);                                                            
 13         map = new HashMap<String, String>();
 14         for (int i=0; i<properties.length-1; i+=2) { 
 
 15             String name = properties[i];
 16             String value = properties[i+1];
 17             map.put(name, value);
 18         }
 19     }   
 20     public MessageEvent(final Object source, final Map<String, String> map) {
 
 21         super(source);
 22         this.map = map; 
 23     }       
 24     public String getProperty(final String key) {
 
 25         return map.get(key);
 26     }   
 27     public boolean isProperty(final String key) {
 
 28         String value = map.get(key);
 29         if (value == null) {
 30             return false;
 31         }
 
 32         return new Boolean(value).booleanValue();
 33     }   
 34     public Map<String, String> getProperties() {
 
 35         return map;
 36     }   
 37     @Override
 38     public String toString() {
 39         return new ToStringBuilder(this, ToStringStyle.MULTI_LINE_STYLE)
 
 40                 .append("source", getSource())
 41                 .append("map", this.map)
 42                 .toString();
 43     }
 
 44 }
 45 
 46 
 

I then created a class to convert above message into JMS message, e.g.

  1 import javax.jms.Session;
 
  2 import javax.jms.JMSException;
  3 import javax.jms.Message;
  4 import javax.jms.MapMessage;                                      
 
  5                                                            
  6 import org.springframework.jms.support.converter.MessageConversionException;
  7 import org.springframework.jms.support.converter.MessageConverter;
  8                                                                            
 
  9 import java.util.Map;
 10 import java.util.HashMap;
 11 import java.util.Enumeration;
 12                                         
 
 13 public class MapMessageConverter implements MessageConverter {
 14     public Object fromMessage(final Message message) throws JMSException, MessageConversionException {
 
 15         if (!(message instanceof MapMessage)) {
 16             throw new MessageConversionException("Message isn't a MapMessage");
 
 17         }
 18         MapMessage mapMessage = (MapMessage) message;
 19         Map<String, String> map = new HashMap<String, String>();
 
 20         Enumeration<String> en = mapMessage.getMapNames();
 21         while (en.hasMoreElements()) {
 22             String name = en.nextElement();
 23             String value = mapMessage.getString(name);
 
 24             map.put(name, value);
 25         }
 26         return map;
 27     }       
 28     public Message toMessage(final Object object, final Session session) throws JMSException, MessageConversionException {
 
 29         if (!(object instanceof Map)) {
 30             throw new MessageConversionException("Object isn't a Map");
 
 31         }
 32         Map<String, String> map = (Map<String, String>) object;
 33         MapMessage message = session.createMapMessage();
 34         for (Map.Entry<String, String> e : map.entrySet()) {
 
 35             message.setString(e.getKey(), e.getValue());
 36         }
 37         return message;
 38     }           
 39 }              
 
 40 
 

Next I created a class that listened for Spring application event and converted into JMS message and published it:

  1 import javax.jms.JMSException;
 
  2 import javax.jms.Session;
  3 import javax.jms.Message;
  4 import java.util.Map;
 
  5 import java.util.UUID;
  6 import org.apache.log4j.Logger;                            
  7 import org.springframework.jms.core.JmsTemplate;                            
 
  8 import org.springframework.jms.core.MessageCreator;               
  9 import org.springframework.jms.support.converter.MessageConverter;         
 10 import org.springframework.context.ApplicationEvent;
 
 11 import org.springframework.context.ApplicationListener;
 12     
 13 public class PublisherAdapter implements ApplicationListener {
 
 14     private static final Logger logger = Logger.getLogger(PublisherAdapter.class);
 15     private JmsTemplate jmsTemplate;
 16     private MessageConverter converter;
 
 17             
 18     //      
 19     public void setMessageConverter(final MessageConverter converter) {
 20         this.converter = converter;
 
 21     }
 22         
 23     public void setJmsTemplate(final JmsTemplate jmsTemplate) {
 24         this.jmsTemplate = jmsTemplate;
 
 25     }
 26         
 27     public void publish(final Map<String, String> map) {
 
 28         jmsTemplate.send(new MessageCreator() {
 29                 public Message createMessage(final Session session) throws JMSException {
 
 30                     return converter.toMessage(map, session);
 31                 }
 32             });
 33     }   
 34     
 35     public void onApplicationEvent(final ApplicationEvent event) {
 
 36        if (event.getSource() != converter && event instanceof MessageEvent) {
 37            publish(((MessageEvent)event).getProperties());
 38        }        
 39     }           
 
 40 }
 41 
 42 
 

Then, I created a JMS listener that listened for messages on Topic and converted those into Spring application event:

  1 import org.apache.log4j.Logger;
 
  2 import java.util.Map;
  3 import javax.jms.MessageListener;
  4 import javax.jms.Message;
 
  5 import javax.jms.MapMessage;
  6 import javax.jms.JMSException;
  7 import org.springframework.jms.support.converter.MessageConverter;
 
  8 import org.springframework.context.ApplicationContext;
  9 import org.springframework.context.ApplicationContextAware;
 10 import org.springframework.beans.BeansException;
 
 11 
 12 
 13 public class ListenerAdapter implements MessageListener, ApplicationContextAware {
 14     private static final Logger logger = Logger.getLogger(ListenerAdapter.class);
 
 15     private MessageConverter converter;
 16     private ApplicationContext applicationContext;
 17             
 18     public void setMessageConverter(final MessageConverter converter) {
 
 19         this.converter = converter;
 20     }   
 21     public void setApplicationContext(final ApplicationContext applicationContext) throws BeansException {
 
 22         this.applicationContext = applicationContext;
 23             
 24     }       
 25             
 26     public void onMessage(final Message message) {
 
 27         Map<String, String> map = (Map<String, String>) converter.fromMessage(message);
 28         applicationContext.publishEvent(new MessageEvent(this, map));
 29     }   
 
 30 }       
 31 
 32 
 

Next, here is Spring configuration to bootstrap these listeners:

  1 <?xml version="1.0" encoding="UTF-8"?>
 
  2 <beans>
  3    <bean id="mapMessageConverter" class="com.amazon.jasper.messaging.spring.MapMessageConverter"/>
 
  4   <bean id="springTopic" class="org.apache.activemq.command.ActiveMQTopic">
  5       <constructor-arg index="0" value="springTopic"/>
 
  6    </bean>
  7    <bean id="springJmsTemplate" class="org.springframework.jms.core.JmsTemplate" scope="prototype">
 
  8         <property name="connectionFactory" ref="jmsConnectionFactory"/>
  9         <property name="deliveryPersistent" value="true"/>
 
 10         <property name="messageConverter" ref="mapMessageConverter"/>
 11         <property name="defaultDestination" ref="springTopic"/>
 
 12    </bean>
 13    <bean id="publisherAdapter" class="com.amazon.jasper.messaging.spring.PublisherAdapter" scope="prototype">
 
 14         <property name="jmsTemplate" ref="springJmsTemplate"/>
 15         <property name="messageConverter" ref="mapMessageConverter"/>
 
 16    </bean>
 17    <bean id="springTopicListener" class="com.amazon.jasper.messaging.spring.ListenerAdapter" scope="prototype">
 
 18         <property name="messageConverter" ref="mapMessageConverter"/>
 19    </bean>
 
 20    <bean class="org.springframework.jms.listener.DefaultMessageListenerContainer" init-method="start" destroy-method="stop" scope="prototype">
 
 21         <property name="connectionFactory" ref="jmsConnectionFactory"/>
 22         <property name="destination" ref="springTopic"/>
 
 23         <property name="messageListener" ref="springTopicListener"/>
 24         <property name="transactionManager" ref="jmsTransactionManager"/>
 
 25         <property name="concurrentConsumers" value="10"/>
 26    </bean>
 
 27 </beans>
 28 
 29 
 

Finally, here is how you will actually use this plumbing in your code:

  1 import org.springframework.context.ApplicationContext;
 
  2 import org.springframework.context.ApplicationContextAware;
  3 import org.springframework.context.ApplicationEvent;
  4 import org.springframework.context.ApplicationListener;
 
  5 import com.amazon.jasper.workflow.WorkflowContext;
  6 public class Myclass implements ApplicationListener, ApplicationContextAware {
 
  7     private ApplicationContext ctx;
  8 
  9     //  ...
 10             ctx.publishEvent(new MessageEvent(this, SYNC_ID, syncId, SYNC_XPDL, "true"));
 
 11 
 12     public void setApplicationContext(ApplicationContext applicationContext) {
 13         this.ctx = applicationContext;
 14     }
 15 
 
 16     public void onApplicationEvent(ApplicationEvent event) {
 17             if (event instanceof MessageEvent) {
 18                 MessageEvent msgEvt = (MessageEvent)event;
 
 19                 // do cache coherence
 20             }
 21     }
 22 
 23 
 

All you need is to add your class to your Spring configuration file, it will automatically be registered as listener for spring events. All this is fairly simple, but I hope it helps you for similar uses in your code.

April 22, 2009

Waterfall model of Hardware Acquisition

Filed under: Computing — admin @ 5:22 pm

I have been ordering a bunch of machines for a new project we are launching in a month and despite the fact that I work for biggest provider of cloud computing, we still use old fashioned acquisition process where the purchase of some machines can take months. So, I have been struggling to find the right size for different services we will be deploying. I find such acquision model similar to waterfall projects where you try to come up with requirements in beginning and in a lot of cases you end up with more requirements you need (because you won’t be able change them) or the requirements that are not suitable in the end. On the other hand, I find hardware based on “cloud computing” or “virtualization” like agile methodologies where you start with just enough resources you need and then add more servers as you grow. Also, as retail business is highly seasonal you can grow and shrink as needed. This is why I like services like EC2, S3 or SimpleDB, but due to security and other corporate policies we can’t use them for internal applications, I feel like child of cobbler who can’t get the shoes even though we provide these services to thousands of other companies.

March 25, 2009

When in Rome, code like how Romans code

Filed under: Languages — admin @ 12:11 pm

I have been programming for over twenty years and I have learned a number of programming languages over the years. One of recurring behavior I have seen in a lot of programmers is that they take a lot of programming habbits (good or bad) from old language(s) to the new language. This could be how you design the application, style of coding, naming conventions, etc. I remember when I switched from C to C++, I was used to procedural thinking and had to learn how to break the problem into classes and how to assign responsibility to different classes. Similarly, when I starting using Java back in 95-95, I had to learn about Java’s peculiar style. For example, I used to declare public methods in C++ at top and all private methods including attributes at the bottom. I also tended to use underscores to prefix member attributes. I slowly learned Java’s style of declaring class attributes at top, using all uppercase for constants, camel case, etc.

In early 2000s, I learned Ruby from PicAxe Ruby book that taught me Ruby from object oriented style and I missed all its functional or meta-programming features. I slowly learned more functional style of programming and meta programming. I even had to switched back to underscores as opposed to camel case. I read Ruby code of other programmers to learn how they code and what conventions they use. I did similar excercises when I learned Python, Erlang, Scala, Objective-C, etc, i.e., I tried to learn not only language itself, its core and third party libraries but how people write the code, package applications or create libraries. Though, I think it helps if there are examples of good usage or style for that language. For example, I have seen plenty of abuses of Javascript that misunderstood its prototype or functional roots and used it as either procedural or class oriented language.

At my work, we use code reviews before any code checkin and I see conventions and styles of other languages mixed in all the time. I think learning different styles of programming makes you a better programmer. For example, I learned from functional programming how immutability can make sure programs safer and I tend to use it more in other object oriented or multi-paradigm langauges that don’t enforce immutability. Though, in other cases it’s hard to force yourself to use features from one language to another when that feature isn’t available inherently. For example, I like mixins feature of Ruby or traits of Scala but I can’t really use them in langauges that support only single inheritance such as Java. So instead of jumping over hoops to use features from other language, I try to use the style suitable for that specific langauge such as using multiple interfaces. I have been learning iphone development and been reading iphone SDK book by Jonathan Zdziarski. One of peculiar thing about his coding examples is that he does not use Interface Builder and creates all UI components from the code. Though, such style is acceptable in many situations but I would prefer to use Interface Builder and follow path of least resistence.

In practice, you will often find multiple styles or approaches of doing a thing in a single language. For example, Ruby encourage multiple ways to do things that can be quite confusing. Though, I like Python’s philolsophy of only one way to do things, but there are plenty of divergent opinions in that language as well. Another somewhat related topic is how to pick a language as languages vary in their core areas of strength. For example, Java was originally marketed as language for Web platform but these days I tend to use Ruby or Python for web development and Java for system development. Also, I tend to use Erlang for network oriented or concurrent applications and use C/C++ where performance is critical. Last year, there was big hoopla over Erlang’s aweful performance for search engine that sparked WideFinder benchmarks but it missed the point that Erlang’s core strength is distributed/concurrent applications and not text searching. So in nutshell, I think it helps to pick a language based on the problem and take advantage of its strengths. Finally, stick to general style of coding and conventions of the language especially when working with large codebase or large number of programmers.

February 24, 2009

Does software quality matters?

Filed under: Methodologies — admin @ 9:24 pm

In last couple of weeks, I followed interesting debate between Bob Martin and Joel Spolsky/Jeff Attwood (listen Hanselminutes) on the issue of SOLID principles. The SOLID acronym stands for five principles, i.e.,

  • Single Responsibility Principle – this principle is based on the principle of Cohesion from Tom Demarco’s Structured Analysis and Design book and mandates that a class should have one, and only one, reason to change.
  • Open Closed Principle – this principle is based on Bertrand Meyer’s book on Object Oriented Software Construction that says that a class should be open to extend without modifying it.
  • Liskov Substitution Principle – this principle was introduced by Barbara Liskov that says derived classes must be substitutable for their base classes.
  • Dependency Inversion Principle – states that your should depend on abstractions, not on concrete implementation.
  • Interface Segregation Principle – states that you should make fine grained interfaces that are client specific.

I learned these principles many years ago and attended Bob Martin in Best Practices 2004 conference, where he talked about these principles. These principles sound good, though in reality they should be a broader guidelines rather than principles. But the heat of the debate was the way these principles have been evangelized by Bob Martin who insisted on using these all the time. Over the years, I have seen similar debates between Cedric Beust and Bob Martin over the use of test driven development. There is also debate on topic of TDD and architecture between Bob Martin and Jim ‘O Coplien. Overall, I find that the issues from these debates boil down to the following:

Usable software vs High quality but unused software

One of the controversial point that Jeff Attwood raised was that the quality does not matter if no one is using your application. I find there is lot of truth to that. In fact, this is the route that most startup takes when developing a new idea and trying something new. For example, everyone was blaming Twitter for choosing Rails when it had scalability and reliability issues. However, Twitter would not have existed if it was written with most scalable and reliable language or framework that took two more years or if the application itself wasn’t as useful. I tend to follow the age old advice: First make it work, then make it right and then make it fast. There is a reason why rapid prototyping frameworks such as Rails, Django, Zend framework, etc. are popular because they allow you to focus more on business functionality and to reduce time to market. So, I agree the first goal of the software should be to make the software that solves real problems or add value. Nevertheless, if first implementation is horrible then it takes hercules effort to make it right and some companies like Friendster never recover.

Customer Experience vs Internal design

One of the earliest advice I got on software development was to write manual before writing the code. It focuses you to solve business problem of the customer rather than writing with top down architecture, which is in a way similar to behavioral driven design in spirit. I find that most useful software developed bottom up and driven by the users. Kent Beck often says that you can’t hide a bad architecture with a good GUI. Nevertheless, for an average user, the usability matters a lot. I remember back in early 90s, IBM OS/2 was superior operating system than Windows but largely loss the market due to the usability (and marketing) issues. The same could be told why people think Macs are better than PC. Rails is also a good example that became popular because you could whip up a webapp in ten minutes despite the fact that its code has been plagued with maintenance issues from monolithic architecture and tricks like chain of alias methods. Other examples include WordPress and Drupal both written in PHP and are the leader in the blogging and content management area due to their usability rather than quality of the code. Again as your software crosses some threshold of number of users it may have to be redesigned, e.g. Rails recently announced that it will merge with another framework Merb in 3.0 because Merb has been designed with micro-kernel and plugable architecture. This also reminds me of merge between Struts and WebWork that turned out to be failure. Joel Spolsky cautions about software rewrites in his blogs and book and I have also blogged on Software rewrites earlier. In the end, you want to extend your application incrementally using Strangler Fig model, which is not an easy thing to do. Ultimately, people matters more than technology, processes or best practices in software development as good people can ship a good software regardless of the language or tools you use.

Evolutionary architecture vs Up front architecture

This has been brought up in debate between Jim Coplien and Bob Martin, where Jim took the position of more upfront design and architecture and Bob took position of evolutionary architecture. I have a lot of respect for Jim Coplien, I still have a copy of Advanced C++ I bought back in ’92 and it introduced me to the concepts of abstraction, body/handle, envelop/letter principles which are sort of DIP. In the interview with Bob Martin, Jim Coplien raised a lot of good points that YAGNI and test-driven based bottom design can create architecture meltdown. Though, I believe good software is developed bottom up, but I do believe some architecture and domain analysis beneficial. I am not necessarily suggesting BDUF (big design up front) or BRUF (big requirements up front), but iteration 0 style architecture and domain analysis when solving a complex problem. For example, the domain driven design by Eric Evans or Responsibility driven design by Rebecca Wirfs-Brock require working closely with the domain experts to analyze the business problems and capturing essential knowledge that may not be obvious. Any investment in proper domain analysis simplifies rest of development and make your application more extensible. There are a number of agile methodologies such as feature driven development and DSDM that encourage some upfront architecture and domain analysis, which I find beneficial.

Extensibility and maintenance

Once your product is hit and loved by users, your next task becomes extending it and adding more features. At this stage, all -ilities such as scalability, performance, security, extensibility becomes more important. Each team can decide on what practices and principles are appropriate and follow them. Agile methodologies encourage collective ownership and pair programming that can spread knowledge and skills, though there are some alternatives such as code reviews or mentoring. I think, having a technical lead who ensures overall quality and keeps the bar up for rest of developers can help with extensibility and maintenance.

Test driven Development

Bob Martin has been adamant proponent of test driven development with his three laws of TDD. I blogged about original debate between Cedric Beust and Bob Martin back in June, 2006 and showed how Bob Martin’s position was not pragmatic. This reaction has also been echoed by Joel, Jeff, Cedric and Jim, who agree 100% coverage is unrealistic. Lately, more and more people are joining this group. I watched recently an interview of Luke Francl who expressed similar sentiments. In my earlier blog, I wrote various types of testing required for writing a reliable and enterprise level software. One of the selling point of unit testing has been ease of refactoring with confidence but I have also found too many unit tests stifle refactoring because they require changing both the code and unit tests. I have found that testing only public interfaces or a bit high level testing without dependency on internal implementation can produce reliable software and also is less prone to breakage when doing refactoring. Though, no one refuses value of testing, but it needs to be practical.

Dependency management

A number of SOLID principles and extended principles such as reuse principle, dependency principles mandates removing or minimizing dependencies between packages. Those principles were made in C++ environment that historically had problems with managing dependencies. Again, the problem comes from religious attitude with these principles. For example, I have worked in companies where different teams shared their code with java jar files and it created dependency hell. I have used jDepend in past to reduce such dependency but it’s not needed in all situation. I am currently building a common platform that will be used by dozens of teams and I intentionally removed or minimized static dependencies and instead used services.

Conclusion

Unfortunately in software industry, new buzzwords often appear every few years that are often used to foster consulting or selling books. There are lots of books on principles, best practices, etc. and the entire agile movement has turned into prescriptive methodologies as opposed to being pragmatic and flexible based on the environment. People like uncle bob use these principles in draconian fashion such as always stressing on 100% test coverage or always keeping interfaces separate from implementation (DIP). Such advice may bring more money from consulting work or publication, but is disingenuous for practical use. In the end, people matters more in software development than arbitrary technology or principles and you need good judgement to pick what practices and solutions suit the problem at hand.

February 23, 2009

Software Estimation

Filed under: Project Management,Technology — admin @ 6:04 pm

Software estimation is a difficult art that I am still learning despite developing software for more than twenty years. I have worked on a number of projects that started with some broad vision and manager asked me how many man-months will it take. You feel like a guy who is asked how long will it take you to survey a cave without going inside (see Software Estimates and the Parable of the Cave). So based on some initial requirements, you make up some numbers. But, often that number translates into commitment and some target date. This issue has been also brought up by Software Estimation by Steve McConnell, Manage It by Johanna Rothman, Lean Software Development by Mary Poppendieck and a number of other people. So it must be made clear that your estimate is not the target date.

As a project is always constrained by iron triangle of schedule/cost/functionality or sometime referred to as cost/quality/schedule or cost/resourcs/schedule. It is crucial to find what’s driving the project as also suggested by Johanna Rothman in her book Manage It. I have seen a number of cases where dates were arbitrary picked, sometime referred to as “happy date”. Though, at other times, dates may depend on marketing campaign, seasons, tax time, Olympics, etc. So, you can negotiate between functionality and schedule based on what’s driving the project. Following are some of techniques that I have found useful with estimation:

  • Get the vision and requirements straight – It’s important about the charter, constraints and requirements for the project as any misdirection here would lead to disaster. Luke Hohmann in his book Beyond Software Architecture recommends starting with good vision and mission statement. Johanna Rothman also recommends creating a project charter before starting the project.
  • Probablistic based estimation – Despite the fact, you are often pressured to produce more precise estimates even though they would be inaccurate, it is better to give estimate with some probablity. Both Johanna Rothman and Steve McConnell cite cone of uncertainty, where your estimate becomes more accurate as project progresses.

  • Based on best/worst/most-likely case – use following formula from Steve McConnell’s book can be used when estimates are more accurate:
expected_case = (best_case + (4 * most_likely) + worse_case) / 6

If estimates are not accurate, then Steve McConnell recommends

expected_case = (best_case + (3 * most_likely) + (2 * worse_case)) / 6

Bob Martin also similar formula from his article PERT, CPM, and Agile Project Management:

Mean     = (best_case + worst_case + (4 * most_likely) ) / 6

Variance = ((worst_case_best_case) / 6) ^ 2
  • Iterative development – No matter if you are working on small or large project, the only way to bring some reality and feedback on initial estimate is to develop iteratively starting with highest valued features.
  • T-shirt based estimation – I find t-shirt based estimation useful when estimating with minimal information available. For example, you may have to estimate projects that you can deliver in Q1, Q2, etc and you can order them in small, medium, large and compare them against their business value.
  • Spiking can also help in areas that are new to the team and spending a little time creating walking skeleton or tracer bullet can give you some idea on the size of the effort for the project.
  • Delphi estimation – where PM and team prepares task list, assumptions and estimate in private and reviews them together.
  • Divide and conquer/Decomposition/WBS – as with any large effort, breaking a project into smaller subsystems, components, services and tasks will help estimate better. In general any errors in estimation of smaller tasks will cancel each other.
  • Estimate fine grained tasks – I can rarely estimate with some accuracy for tasks that are longer than a few days so it’s important to estimate only fine grained tasks. XP has a concept of inch pebble and story points that can help in this case. The idea is that each task is either done or not done.
  • Planning poker a technique from Agile Estimating and Planning by Mike Cohn, where each member of the team pick an estimate for a story based on fibonacci numbers, but don’t show until everyone selects some number. The members then pick some average or may ask member with highest or lowest estimates to explain.
  • Historical data – though I rarely see PM track estimates but tracking them can help future projects and new projects can use LOC, man-months, function-points, # of services, files, interfaces, bugs from prior projects for estimation.
  • Schedule chicken – Kent Beck often talks about schedule chicken where you have some some meeting about who is ontrack and you hope there is someone who is behind so that you don’t have to admit you are behind as well. Integrity is big part of the XP and agile methodologies so it encourages transparency and honesty instead of schedule chicken.
  • Better to overestimate than underestimate – programmers often underestimate and though there is risk of student syndrome or Parkinson’s law but it’s better to overestimate.
  • Don’t question developer’s estimate – even though developers tend to underestimate, some managers still question them, which is not a good idea.
  • In XP or Scrum, you use story points, which can be ideal hours or based on some multiplier. These numbers are generally follow fibonacci sequence such as 1, 2, 3, 5, 8, 13, 21.
  • Function points use number of external input/output/queries, internal logical files/external interface files and it can be used as unit of measurements similar to story points.
  • Estimation quality factory (EQF) as proposed by Tom Demarco in his paper A Defined Process For Project Postmortem Review can be used to check how accurate estimates are.
  • Include vacation, sick, holidays as well as non-development activities such as testing, deployment, configuration, migration, etc in your project plan.
  • Scheduling is all about ordering with highest value features. I find rolling-wave scheduling based on milestones useful when planning iterations.

Summary

I often find projects turn into death march projects due to overly optimistic estimates and “queen of denial” manager who holds developers’ estimates as commitment and refuses to accept the reality. One way to overcome bad estimation is to adopt iterative development that delivers small features based on the value proposition, which creates biggest value for the business. Another way is to use advice from the Rational Unified Process that uses risk management to prioritize the highest risk tasks first. Though, some managers are keen to accept more work than the team can handle in order to aim high but it takes a courage to say NO. In the end, under-promise and over deliver as it can save your credibility and not to mention unnecessary overtime and stress on the team.

February 5, 2009

IP addresses of Spammers from my Honeypot

Filed under: SPAM — admin @ 3:59 pm

I have an old guest book application that was originally written in J2EE in ’98 and then moved to Rails a couple of years ago. Though, I don’t get much guest entries, but it does get a lot of SPAM. It has become sort of honeypot and I keep IP addresses of those spammers and in case anyone is interested you can download that list from http://plexobject.com/spammers_ip_addresses.txt.

« Newer PostsOlder Posts »

Powered by WordPress