Shahzad Bhatti

I came across an interesting blog on Programmers: Before you turn 40, get a plan B, where the author sites age discrimination in programming career. The author shows how in IT industry, technology shift every 10 years levels the plainfield so there is a little difference between junior and senior programmer. The author also asserts that more experienced programmer will have a lot of irrelevent experience that employer may not be willing to consider when hiring or may not pay higher salary for those skills. I have been programming for over twenty two years and working professionally for over seventeen years and I certainly have seen effects of technology shift. In early 90s, I worked for a physics lab that had plenty of VAX computers using FORTRAN, and they were migrating most of the system to UNIX so people with VMS skills had to adopt to UNIX. I then had to work on mainframe system for another company and used COBOL and CICS. I saw plenty of people who worked on mainframe system for over 10-15 years and knew a little outside mainframe technology. When that company also migrated to NT and UNIX platform, it was quite stressful to many people. Though, there are still companies who are willing to pay top dollars for mainframe experience but opportunities are very few. I myself had to adopt and learn C++ and object-oriented design and found other opportunties that used those skills. I was young and unmarried so switching to another technology was not hard. In mid 90s I switched to Java, CORBA and Web development and there was some learning and adopting involved. In late 90s and early 2000s, I started using Java Enterprise Beans (EJBs) and SOAP based Web services. From 2003-onward, I started using more open source technologies like Spring, Hibernate, Tomcat, REST, etc and I had to abandoned my past CORBA or EJB expertise. This was followed by rise of dynamic languages and I started using Ruby on Rails and built a couple of sites using it. After that I became interested functional programming such as Erlang. These days, I have been spending my spare time in iPhone development.

I completely agree that in programming career, the past experience becomes irrelevant and most companies won’t pay me for the irrelevant experience such as VMS, mainframe, CORBA or EJBs. Also, as I became older and started family, switching jobs just for technology became a lot harder. I believe experienced programmers who have worked with varied technologies bring useful perspective and a bit of pessimism that is often needed. I also believe most experienced programmers have better design skills and pay more attention to writing maintainable or scalable code, but it probably doesn’t matter in may places that just want to get shit done.

Another aspect that differentiates programming career from other professions is lack of predictability. I have found that in our industry, project scope, requirements and estimates are poorly defined and most companies expect programmers to be heros or martyrs who will abandoned their life and dedicate completely to the work. IT people also have to pay for the global culture of 24×7 service and have to be oncall when things break. This becomes a lot harder with the family so many companies prefer young and unmarried folks who are more willing to be heros and those young programmers like the extra attention.

So what is my plan B? It is to continually learn new skills and be a little better than the competition. Many of my past colleagues have moved to the dark site and now are directors or senior managers at other companies. And truth is that in most companies managers carry more respect and authority so no wonder most programmers aspire to become one. I on the other hand, just can’t give up programming and usually spend my spare time on a number of small projects. So I will continue to be a lowly code monkey for the rest of my life.

I have been learning iPhone development lately and needed to add some persistence capability to my application. There are varied options available for persistence such as using “User Defaults” for small user specific settings, serialization similar to serialization or pickle features of other languages and builtin support of embeded Sqlite. I found Sqlite more performant, memory efficient and flexible than other options so I chose it. Using Sqlite with iPhone is fairly straight forward and there are tons of examples such Creating Todo list using Sqlite. However, when I looked for some O/R mapping framework for iPhone, I found that iPhone SDK unlike Mac development didn’t have any support and neither I could find any other solution elsewhere. So, I started writing a simple O/R mapping library based on Active Object pattern. This is similar to how Rails and Django implement O/R mapping. Based on convention over configuration, it simply maps object properties to the database table fields. At this time, this is very basic O/R mapping library and does not support relations, validation, database integrity support, etc. Nevertheless, it met my simple needs and I have released it as open source project under OCActiveObjects on GitHub.

The OCActiveObjects library is fairly small and consists of following classes:

ActiveObject

This is the base class that you extend in order to add automatic O/R support. You have to override following Class methods to specify name of the database and table:

     + (NSString *) getTableName;
   

Above method defines name of table where instances of the object will be stored.

     + (NSString *) getDatabaseName;
   

Above method defines name of the database to be used. You will then be able to call following methods to interact with the Sqlite database:

   + (void) openDatabase;
   

Above method must be called once before any other methods, usually at the start of your application.

   + (void) closeDatabase;
   

Above method must be called once before you shutodnw your application.

  
   - (void) save;
 

Above method saves a new object or updates an existing object. Each subclass of ActiveObject is automatically assigned a unique database id with a property named “objectId”. This is another example of convention where all tables will use a numeric surrogate key to identify each row. If that property is nil then it assumes this is new object and inserts a new row in the database, otherwise it updates an existing row in the database. It assumes that name of database fields are same as property names, though you can override that behavior by overriding _getPropertyNamesAndTypes Class method.

   + (ActiveObject *) findByPrimaryKey:(NSNumber *)objId;
 

Above method queries an object in the database matching objectId property, which identifies each object in the database.

   + (NSArray *) findWithCriteria:(NSDictionary *)criteria;
 

Above method returns an array of objects that match criteria. The criteria at this time is simple dictionary, i.e., pair of name and values that are joined by “AND” clause. There is a immediate need to extend this to support more flexible queries.

   + (NSArray *) findAll;
 

Above method returns all objects, which may not be good for iPhone application due to limited amount of memory. This is another area that needs immediate attention.

   + (int) removeAll;
 

Above method removes all rows in the table so be careful with this.

   + (int) removeWithCriteria:(NSDictionary *)criteria;
 

Above method removes only methods matching criteria. Again criteria consists of name/value pairs.

   + (int) countWithCriteria:(NSDictionary *)criteria;
 

Above method counts the number of rows in the database matching criteria.

   + (int) countAll;
 

Above method returns count of all rows in the table.

Exension Methods in ActiveObject

There are number of extension methods to customize SQLs or behavior of the object such as

 - (void) _insert;
 

Above method inserts an object into the database.

 - (void) _update;
 

Above method updates an existing object into the database.

 + (NSDictionary *) _getPropertyNamesAndTypes;
 

You can override above method to change properties that needs to be persisted.

 + (NSString *) _getCreateSQL;
 

Above method generates an SQL for creating table.

 + (NSMutableString *) _getInsertSQL;
 

Above method generates an SQL for inserting a row in the table.

 + (NSMutableString *) _getUpdateSQL;
 

Above method generates an SQL for updating a row in the table.

 + (NSMutableString *) _getSelectSQL;
 

Above method generates an SQL for selecting fields from the database.

 + (void) _createTable;
 

Above method creates database table.

IntrospectHelper

The OCActiveObjects library uses some Objective C magical runtime support to query for properties and this class encapsulates those methods.

SqliteHelper

This class some helper methods for Sqlite3.

How to use

In order to test it, let’s define a simple Person class that extends ActiveObject, e.g.

 #import <Foundation/Foundation.h>
 #import "ActiveObject.h"
 
 @interface Person : ActiveObject {
         NSString *name;
         short age;
         int rank;
         long votes;
         char sex;
         double income;
         BOOL active;
         NSInteger flags;
         NSNumber *rating;
         NSDate *birthdate;      
 }
 
 
 @property (nonatomic, retain) NSString *name;
 @property (nonatomic, assign) short age;
 @property (nonatomic, assign) int rank;
 @property (nonatomic, assign) long votes;
 @property (nonatomic, assign) char sex;
 @property (nonatomic, assign) double income;
 @property (nonatomic, assign) BOOL active;
 @property (nonatomic, assign) NSInteger flags;
 @property (nonatomic, retain) NSNumber *rating;
 @property (nonatomic, retain) NSDate *birthdate;
 
 - (BOOL)isEqualToPerson:(Person *)aPerson;
 
 @end
 

Implemention of Person.m looks like:

 #import "Person.h"
 
 @implementation Person
 @synthesize name;
 @synthesize age;
 @synthesize rank;
 @synthesize votes;
 @synthesize sex;
 @synthesize income;
 @synthesize active;
 @synthesize flags;
 @synthesize rating;
 @synthesize birthdate;
 
 - (BOOL)isEqual:(id)other {
     if (other == self)
         return YES;
     if (!other || ![other isKindOfClass:[self class]])
         return NO;
     return [self isEqualToPerson:other];
 }
 
 - (BOOL)isEqualToPerson:(Person *)aPerson {
         if (self == aPerson)
         return YES;
     if (![(id)[self name] isEqual:[aPerson name]])
         return NO;
     return YES;
 }
 
 - (NSUInteger)hash {
         NSUInteger hash = 0;
         hash += [[self name] hash];
         return hash;
 }
 
 
 
 - (NSString *)description {
         return [NSString stringWithFormat:@"id %@, name %@", self.objectId, self.name);
 }
 
 
 - (void) dealloc {
         [name release];
         [birthdate release];
         [super dealloc];
 }
 
 + (NSString *) getTableName {
         return @”persons”;
 }
 
 + (NSString *) getDatabaseName {
         return @”personsdb”;
 }
 
 @end
 

Then you can first open the database, e.g.

 
         [Person openDatabase];
 

Then create a new person object, e.g.

         Person *person = [[[Person alloc] init] autorelease];
         person.birthdate = [[NSDate alloc]init];
         int random = [person.birthdate timeIntervalSince1970];
         person.age =  random % 30;
         person.rank = random % 20;
         person.votes = random % 10;
         person.sex = ‘M’;
         person.name = [NSString stringWithFormat:@"Joe #%d", random % 1000];
         person.income = random % 3000;
         person.active = YES;
         person.flags = random % 30 + 0.5;
         person.rating = [NSNumber numberWithInt:20.5];  
         return person;
 

You will then be able to save the person object as

         [person save];
 

You can see how many rows are in the database by

         int count = [Person countAll];
 

And then retrieve the object that we saved as

         Person *person2 = (Person *) [Person findByPrimaryKey:person.objectId];
 

When you are done, you can then close the database:

         [Person closeDatabase];
 

One of the frustrating aspect of iPhone development has been lack of good unit testing support. Though, XCode comes with OCUnit, but it is hard to install and use with iPhone. I kept getting weird errors like:

 exited abnormally with code 139
 

 failed tests for architecture 'i386'
 

Though, there are some basic tutorials like Test Driving Your Code with OCUnit or OCUnit: Integrated Unit Testing In Xcode, but they didn’t help. I also tried adding google-toolbox-for-mac but macro errors are extremely frustrating. Besides better testing, OCActiveObjects needs a lot of help to add better support of criteria, paging, relational mapping and validation. I also had hard time figuring out how to create a static library until I found Building static libraries with the iPhone SDK though I still need help in adding framework level support. Hopefuly, other people can contribute to the open source project. You can send me your suggestions and comments as well via email “bhatti AT plexobject DOT com” or tweet me at bhatti_shahzad.

I was recently tracking memory leak in one of our applicatios, however the problem was only occuring in our production environment and the network latency between production subnet and my local subnet was too high to run profiler (I tried). So, I had to use some other tools to look at the snapshot of memory that was in use. I found jmap and jhat that are shipped with JDK (1.6 in our case) pretty handy. When memory usage was high, I dumped the memory using jmap, e.g.

 jmap -F -dump:file=/tmp/dump.out 
 

I then copied the file to my local machine and ran jhat using

 jhat dump.out
 

Though, jhat is fairly slow, but it finally starts the web server on port 7000 and you can then point your browser to

 http://mymachine:7000
 

The jhat can also read data from hprof that you can dump by adding following vm option:

 -agentlib:hprof=heap=sites
 

and then read the file using jhat. In addition to above tool, you can run garbage collection in verbose mode using following vm option:

 -verbose:gc -verbose:class -Xloggc:gc.out -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
 

Finally, you can use jconsole to look at memory graph by adding support of jmx to your application server, .e.g.

 -Dcom.sun.management.jmxremote.port=9003 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote
 

Despite all these options, I find profiler is much better for finding memory leaks, but these tools helped a bit.

I have been ordering a bunch of machines for a new project we are launching in a month and despite the fact that I work for biggest provider of cloud computing, we still use old fashioned acquisition process where the purchase of some machines can take months. So, I have been struggling to find the right size for different services we will be deploying. I find such acquision model similar to waterfall projects where you try to come up with requirements in beginning and in a lot of cases you end up with more requirements you need (because you won’t be able change them) or the requirements that are not suitable in the end. On the other hand, I find hardware based on “cloud computing” or “virtualization” like agile methodologies where you start with just enough resources you need and then add more servers as you grow. Also, as retail business is highly seasonal you can grow and shrink as needed. This is why I like services like EC2, S3 or SimpleDB, but due to security and other corporate policies we can’t use them for internal applications, I feel like child of cobbler who can’t get the shoes even though we provide these services to thousands of other companies.

I have been programming for over twenty years and I have learned a number of programming languages over the years. One of recurring behavior I have seen in a lot of programmers is that they take a lot of programming habbits (good or bad) from old language(s) to the new language. This could be how you design the application, style of coding, naming conventions, etc. I remember when I switched from C to C++, I was used to procedural thinking and had to learn how to break the problem into classes and how to assign responsibility to different classes. Similarly, when I starting using Java back in 95-95, I had to learn about Java’s peculiar style. For example, I used to declare public methods in C++ at top and all private methods including attributes at the bottom. I also tended to use underscores to prefix member attributes. I slowly learned Java’s style of declaring class attributes at top, using all uppercase for constants, camel case, etc.

In early 2000s, I learned Ruby from PicAxe Ruby book that taught me Ruby from object oriented style and I missed all its functional or meta-programming features. I slowly learned more functional style of programming and meta programming. I even had to switched back to underscores as opposed to camel case. I read Ruby code of other programmers to learn how they code and what conventions they use. I did similar excercises when I learned Python, Erlang, Scala, Objective-C, etc, i.e., I tried to learn not only language itself, its core and third party libraries but how people write the code, package applications or create libraries. Though, I think it helps if there are examples of good usage or style for that language. For example, I have seen plenty of abuses of Javascript that misunderstood its prototype or functional roots and used it as either procedural or class oriented language.

At my work, we use code reviews before any code checkin and I see conventions and styles of other languages mixed in all the time. I think learning different styles of programming makes you a better programmer. For example, I learned from functional programming how immutability can make sure programs safer and I tend to use it more in other object oriented or multi-paradigm langauges that don’t enforce immutability. Though, in other cases it’s hard to force yourself to use features from one language to another when that feature isn’t available inherently. For example, I like mixins feature of Ruby or traits of Scala but I can’t really use them in langauges that support only single inheritance such as Java. So instead of jumping over hoops to use features from other language, I try to use the style suitable for that specific langauge such as using multiple interfaces. I have been learning iphone development and been reading iphone SDK book by Jonathan Zdziarski. One of peculiar thing about his coding examples is that he does not use Interface Builder and creates all UI components from the code. Though, such style is acceptable in many situations but I would prefer to use Interface Builder and follow path of least resistence.

In practice, you will often find multiple styles or approaches of doing a thing in a single language. For example, Ruby encourage multiple ways to do things that can be quite confusing. Though, I like Python’s philolsophy of only one way to do things, but there are plenty of divergent opinions in that language as well. Another somewhat related topic is how to pick a language as languages vary in their core areas of strength. For example, Java was originally marketed as language for Web platform but these days I tend to use Ruby or Python for web development and Java for system development. Also, I tend to use Erlang for network oriented or concurrent applications and use C/C++ where performance is critical. Last year, there was big hoopla over Erlang’s aweful performance for search engine that sparked WideFinder benchmarks but it missed the point that Erlang’s core strength is distributed/concurrent applications and not text searching. So in nutshell, I think it helps to pick a language based on the problem and take advantage of its strengths. Finally, stick to general style of coding and conventions of the language especially when working with large codebase or large number of programmers.

In last couple of weeks, I followed interesting debate between Bob Martin and Joel Spolsky/Jeff Attwood (listen Hanselminutes) on the issue of SOLID principles. The SOLID acronym stands for five principles, i.e.,

• Single Responsibility Principle - this principle is based on the principle of Cohesion from Tom Demarco’s Structured Analysis and Design book and mandates that a class should have one, and only one, reason to change.
• Open Closed Principle - this principle is based on Bertrand Meyer’s book on Object Oriented Software Construction that says that a class should be open to extend without modifying it.
• Liskov Substitution Principle - this principle was introduced by Barbara Liskov that says derived classes must be substitutable for their base classes.
• Dependency Inversion Principle - states that your should depend on abstractions, not on concrete implementation.
• Interface Segregation Principle - states that you should make fine grained interfaces that are client specific.

I learned these principles many years agao and also watched Bob Martin in Best Practices 2004 conference, where he talked about these principles. These principles sound good, though in reality they should be general guidelines rather than principles. But the heat of the debate was the way these principles have been evangilized by Bob Martin who insisted on using these all the time. Over the years, I have seen similar debates between Cedric Beust and Bob Martin over the use of test driven development. There is also debate on topic of TDD and architecture between Bob Martin and Jim ‘O Coplien. Overall, I find that the issues from these debates boil down to the following:

Usable software vs quality

One of the controversial point that Jeff Attwood raised was that the quality does not matter if no one is using your application. I find there is lot of truth to that. In fact, this is the route that most startup takes when developing a new idea and trying something new. For example, everyone was blaming Twitter for choosing Rails when it had scalability and reliability issues. However, Twitter would not have existed if it were written with most scalable and reliable language or framework that took two more years or if the application itself wasn’t as useful. I tend to follow the age old advice: First make it work, then make it right and then make it fast. There is a reason why rapid prototyping frameworks such as Rails, Django, Zend framework, etc. are popular because they allow you to focus more on business functionality and to reduce time to market. So, I agree the first goal of the software should be to make the software that solves real problems or add value. Nevertheless, if first implementation is horrible then it takes hercules effort to make it right and some companies like Friendster never recover.

External design vs internal design

One of the earliest advice I got on software development was to write manual before writing the code. It focuses you to solve business problem of the customer rather than writing with top down architecture, which is in a way similar to behavioral driven design in spirit. I find that most useful software developed bottom up and driven by the users. Kent Beck often says that you can’t hide a bad architecture with a good GUI. Nevertheless, for an average user, the usability matters a lot. I remember back in early 90s, IBM OS/2 was superior operating system than Windows but largely loss the market due to the usability (and marketing) issues. The same could be told why people think Macs are better than PC. Rails is also a good example that became popular because it could whip up a webapp in ten minutes despite the fact that its code has been plagued with maintenance issues from monolithic architecture and tricks like chain of alias methods. Other examples include Wordpress and Drupal both written in PHP and are the leader in the blogging and content management area due to their usability rather than quality of the code. Again as your software crosses some threshold of number of users it may have to be redesigned, e.g. Rails recently announced that it will merge with another framework Merb in 3.0 because Merb has been designed with micro-kernel and pluggability based architecture. This also reminds me of merge between Struts and WebWork that turned out to be failure. Joel Spolsky cautions about software rewrites in his blogs and book and I have also blogged on Software rewrites earlier. In the end you want to extend your application incrementally, which is not an easy thing to do. Ultimately, I think software development is all about people rather than technology, principles or best practice. If you have good people then can take your application to the next step regardless if it was written in PHP, Java or Erlang.

Evolutionary architecture vs up front architecture

This has been brought up in debate between Jim Coplien and Bob Martin, where Jim took the position of more upfront design and architecture and Bob took position of evolutionary architecture. I have a lot of respect for Jim Coplien, I still have a copy of Advanced C++ I bought back in ‘92 and it introduced me to the concepts of abstraction, body/handle, envelop/letter principles which are sort of DIP. In the interview with Bob Martin, Jim Coplien raised a lot of good points that YAGNI and test-driven based bottom design can create architecture meltdown. Though, I believe good software is developed bottom up, but I do believe some architecture and domain analysis beneficial. I am not necessarily suggesting BDUF (big design up front) or BRUF (big requirements up front), but iteration 0 style architecture and domain analysis when solving a complex problem. For example, the domain driven design by Eric Evans or Responsibility driven design by Rebecca Wirfs-Brock require working closely with the domain experts to analyze the business problems and capturing essential knowledge that may not be obvious. Any investment in proper domain analysis simplifies rest of development and make your application more extensible. There are a number of agile methodologies such as feature driven development and DSDM that encourage some upfront architecture and domain analysis, which I find beneficial.

Extensibility and maintenance

Once your product is hit and loved by users, your next task becomes extending it and adding more features. At this stage, all -ilities such as scalability, performance, security, extensibility becomes more important. This is also more common in corporate environment and most of the developers live their entire lives maintaining the code written by others. Nevertheless, each team should decide on what practices and principles are appropriate and follow them. Agile methodologies encourage collective ownership and pair programming that can spread knowledge and skills, though there are some alternatives such as code reviews or mentoring. I also think, having a technical lead who ensures overall quality and keeps the bar up for rest of developers can help with extensibility and maintenance.

Test driven Development

Bob Martin has been adament proponent of test driven development with his three laws of TDD. I blogged about original debate between Cedric Beust and Bob Martin back in June, 2006 and showed how Bob Martin’s position was unpragmatic. This reaction has also been echoed by Joel, Jeff, Cedric and Jim, who agree 100% coverage is unrealistic. Lately, more and more people are joining this group. I watched recently an interview of Luke Francl who expressed similar sentiments. In my earlier blog, I wrote various types of testing required for writing a reliable and enterprise level software. One of the selling point of unit testing has been ease of refactoring with confidence but I have also found too many unit tests stifle refactoring because they require changing both the code and unit tests. I have found that testing only public interfaces or a bit high level testing can produce reliable software and also is less prone to breakage when doing refactoring. Though, no one refuses value of testing, but it needs to be practical.

Dependency management

A number of SOLID principles and extended principles such as reuse principle, dependency principles mandates removing or minimizing dependnecies between packages. Those principles were made in C++ environment that historically had problems with managing dependencies. Again, the problem comes from religious attitude with these principles. For example, I have worked in companies where different teams shared their code with java jar files and it created dependency hell. I used jDepend to reduce such dependency but it’s not needed in all situation. I am currently building a common platform that will be used by dozens of teams and I intentionally removed or minimized static dependencies and instead used services.

Conclusion

Unfortunately, in software industry buzzwords are often used to foster consulting or selling books. There are lots of books on principles, best practices, etc. The entire agile movement has turned into prescriptive methodologies as opposed to being pragmatic and flexible based on the environment. People like uncle bob use these principles in draconian fashion such as always stressing on 100% test coverage or always keeping interfaces separate from implementation (DIP). Such advice may bring more money from consulting work or publication, but is disingenuous for practical use. In the end, software development is about people and not about technology or principles and you need good judgement to pick what practices and solutions suit the problem.

Software estimation is a difficult art that I am still learning despite developing software for more than twenty years. I have worked on a number of projects that started with some broad vision and manager asked me how many man-months will it take. You feel like a guy who is asked how long will it take you to survey a cave without going inside (see Software Estimates and the Parable of the Cave). So based on some initial requirements, you make up some numbers. But, often that number translates into commitment and some target date. This issue has been also brought up by Software Estimation by Steve McConnell, Manage It by Johanna Rothman, Lean Software Development by Mary Poppendieck and a number of other people. So it must be made clear that your estimate is not the target date.

As a project is always constrained by iron triangle of schedule/cost/functionality or sometime referred to as cost/quality/schedule or cost/resourcs/schedule. It is crucial to find what’s driving the project as also suggested by Johanna Rothman in her book Manage It. I have seen a number of cases where dates were arbitrary picked, sometime referred to as “happy date”. Though, at other times, dates may depend on marketing campaign, seasons, tax time, olympics, etc. So, you can negotiate between functionality and schedule based on what’s driving the project. Following are some of techniques that I have found useful with estimation:

• Get the vision and requirements straight - It’s important about the charter, constraints and requirements for the project as any misdirection here would lead to disaster. Luke Hohmann in his book Beyond Software Architecture recommends starting with good vision and mission statement. Johanna Rothman also recommends creating a project charter before starting the project.
• Probablistic based estimation - Despite the fact, you are often pressured to produce more precise estimates even though they would be inaccurate, it is better to give estimate with some probablity. Both Johanna Rothman and Steve McConnell cite cone of uncertainty, where your estimate becomes more accurate as project progresses.
  
  
• Based on best/worst/most-likely case - use following formula from Steve McConnell’s book can be used when estimates are more accurate:

   expected case = (best-case + (4 * most-likely) + worse-case) / 6
   

  If estimates are not accurate, then Steve McConnell recommends

   expected case = (best-case + (3 * most-likely) + (2*worse-case)) / 6
   

  Bob Martin also similar formula from his article PERT, CPM, and Agile Project Management:

    Mean = (best-case+worst-case+(4*most-likely))/6
    Variance = ((worst-case-best-case)/6)^2
   

• Iterative development - No matter if you are working on small or large project, the only way to bring some reality and feedback on initial estimate is to develop iteratively starting with highest valued features.
• T-shirt based estimation - I find t-shirt based estimation useful when estimating with minimal information available. For example, you may have to estimate projects that you can deliver in Q1, Q2, etc and you can order them in small, medium, large and compare them against their business value.
• Spiking can also help in areas that are new to the team and spending a little time creating walking skeleton or tracer bullet can give you some idea on the size of the effort for the project.
• Delphi estimation - where PM and team prepares task list, assumptions and estimate in private and reviews them together.
• Divide and conquer/Decomposition/WBS - as with any large effort, breaking a project into smaller subsystems, components, services and tasks will help estimate better. In general any errors in estimation of smaller tasks will cancel each other.
• Estimate fine grained tasks - I can rarely estimate with some accuracy for tasks that are longer than a few days so it’s important to estimate only fine grained tasks. XP has a concept of inch pebble and story points that can help in this case. The idea is that each task is either done or not done.
• Planning poker a technique from Agile Estimating and Planning by Mike Cohn, where each member of the team pick an estimate for a story based on fibonacci numbers, but don’t show until everyone selects some number. The members then pick some average or may ask member with highest or lowest estimates to explain.
• Historical data - though I rarely see PM track estimates but tracking them can help future projects and new projects can use LOC, man-months, function-points, # of services, files, interfaces, bugs from prior projects for estimation.
• Schedule chicken - Kent Beck often talks about schedule chicken where you have some some meeting about who is ontrack and you hope there is someone who is behind so that you don’t have to admit you are behind as well. Integrity is big part of the XP and agile methodologies so it encourages transparency and honesty instead of schedule chicken.
• Better to overestimate than underestimate - programmers often underestimate and though there is risk of student syndrome or Parkinson’s law but it’s better to overestimate.
• Don’t question developer’s estimate - even though developers tend to underestimate, some managers still question them, which is not a good idea.
• In XP or Scrum, you use story points, which can be ideal hours or based on some multiplier. These numbers are generally follow fibonacci sequence such as 1, 2, 3, 5, 8, 13, 21.
• Function points use number of external input/output/queries, internal logical files/external interface files and it can be used as unit of measurements similar to story points.
• Estimation quality factory (EQF) as proposed by Tom Demarco in his paper A Defined Process For Project Postmortem Review can be used to check how accurate estimates are.
• Include vacation, sick, holidays as well as non-development activities such as testing, deployment, configuration, migration, etc in your project plan.
• Scheduling is all about ordering with highest value features. I find rolling-wave scheduling based on milestones useful when planning iterations.

I often find projects turn into death march projects due to overly optimistic estimates and “queen of denial” manager who holds developers’ estimates as commitment and refuses to accept the reality. I don’t find any good solution to estimation except iterative development starting with highest value, which creates biggest value for the business. Though, often agile methodologies recommend starting with simplest thing possible but I like Rational Unified Process for risk management that suggests doing highest risk tasks first. Also, I find multi-tasking diminishes productivity so it’s better to assign people to only one project if possible. Finally, I have seen many managers accept more work than the team can handle in order to appear more committed in front of upper management. It takes courage to say NO and in the end it can save your credibility and not to mention unnecessary overtime.

I have an old guest book application that was originally written in J2EE in ‘98 and then moved to Rails a couple of years ago. Though, I don’t get much guest entries, but it does get a lot of SPAM. It has become sort of honeypot and I keep IP addresses of those spammers and in case anyone is interested you can download that list from http://plexobject.com/spammers_ip_addresses.txt.

I just finished Clean Code: A Handbook of Agile Software Craftsmanship. The book is compilation of patterns on writing clean/maintainable code and is written by uncle Bob and other folks from his consulting company Object Mentors including Tim Ottinger, Michael Feathers (author of Working Effectively with Legacy Code. This book is similar to Implementation Patterns by Kent Beck I also read recently. As opposed, this book is a lot more thick with seventeen chapters, however there are plenty of pages filled with tedious listing of Java code.

The first chapter gathers thoughts on good code from a number innovators and authors such as Kent Beck, Bjarne Stroustrup, Grady Booch, Dave Thomas, etc. They mention various attributes of good code such as easy to read, efficient, DRY, focused, literate, minimal, error handling and warn of bad code that leads to messy code or broken windows mentality.

The chapter 2 talks about golden advice of using intention-revealing names to improve the readability and following principle of least surprise. This chapter denounces use of hungarian notation or member prefix such as _ or m_.

The chapter 3 describes functions and methods and advices to use small, cohesive functions. It suggests using one level of abstraction to facilitate reading code from top to bottom. It also recommends using polymorphic methods as opposed to switch statement, if-else or functions that take flag/boolean arguments. The chapter discourages functions with side effects or the one that create temporarl coupling. This chapter also mentions an aged old advice of command query separation, though it skips its roots from Bertrand Meyer’s design by contract. Finally, this chapter urges use of exceptions as opposed to error codes.

The chapter 4 describes writing good comments that focus on intent and dissuades against redundant and misleading comments.

The chapter 5 illustrates use of good formatting such as identation, horizontal, vertical spacing, etc.

The chapter 6 describes use of objects and data structures. It encompasses advice on polymorphism, law of demeter, encapsulation, DTO/value objects, etc.

The chapter 7 discusses error handling. Again it encourages use of exceptions rather than return codes or error codes. It prohibits use of checked exceptions as it violates open/closed principle. It also discourages returning or passing null and recommends exceptions or empty collection instead of returning null.

The chapter 8 describes boundaries between components.

The chapter 9 describes advice on unit testing such as TDD, keeping tests clean, one assert per test, single concept per test, etc.

The chapter 10 is similar to chapter 6 and contains recommendations on writing classes such as encapsulation, classes should be small, single responsibility principle, cohesion, dependency inversion principle, etc.

The chapter 11 encompasses advice on building systems and managing complexity. It suggests dependency injection and use of factories. It also advocates use of AOP for managing cross cutting concerns.

The chapter 12 talks about emergent design, the concept I first heard from Andy Hunt. This chapter describes advice from Kent Beck: run all tests, refactoring, no duplication, express intents of the programmer and minize the number of classes and methods. This chapter advocates use of template methods to remove duplication and command/visitor patterns to express design to other developers.

The chapter 13 discusses concurrency. It suggests use of single-responsbility principle to keep concurrent code separate from other code, limiting scope of data, keeping threads independent, and use of immutable oobjects or copies of data. It recommends keeping critical section small. The chapter also holds advice on testing threaded code and suggests making threaded code plugable.

The chapter 14 encompasses an excercise on how to incrementally improve code.

The chapter 15 describes JUnit framework and walks reader through improving tests.

The chapter 16 walks reader through another refactor excercise.

The chapter 17 encompasses a number of smells, heuristics and antipatterns. It deters use of poolrly written comments, builds/tests that require more than one step. The chapter prohibits passing too many arguments to functions or use of output/flag arguments to functions. It encourages testing boundrary conditions and respecting overridden safeties. It also proscribes writing code at wrong level of abstraction, i.e., exposing low-level logic through interface. The chapter also forbids exposing derivatives to base classes, having too much information in interface. Other smells include artifical coupling (sharing constants), feature envy (coupling formating logic), flag arguments. This chapter also holds advice of Kent Beck about using explanatory (local/temporary) variables. The chapter recommends use of constants, structure over convention, encapsulate conditionals, avoiding temporal couplings, keeping functions at same level of abstraction and keeping configurable data at high levels. The chapter also contains Java specific is advice such as avoiding wildcards in imports, avoiding use of interface to inherit constants, etc. Finally, the chapter cautions against insufficient tests, use of coverage tool, testing boundary conditions, etc.

In conclusion, this book contains advice from wide range of authors on writing good maintainable code. From being programmer myself for over 20 years, I can attest that writing good code is hard and requires a lot of attention for continually improving and refactoring code. I see too often projects have incredible pressure to deliver and programmers end up using short cuts or not able to refactor the code. Kent often talks about courage in XP and programmers needs to have courage to fight for writing maintainable code and take time to refactor existing code. Finally, I would also caution against using these practices mindlessly as another favorite rule of mine is that every practice or pattern depends on the situation. Unfortunately, there are lot of people in software industry including uncle bob who use these rules in draconian fashion such as always stressing on 100% test coverage or always keeping interfaces separate from implementation (DIP). Such advice may bring more money from consulting work or publication, but is disingenuous for practical use.