Shahzad Bhatti Welcome to my ramblings and rants!

Agile Methodologies Under the Hood
Based on research and Scott Ambler’s article, agile methodologies have crossed
chasm and moved to mainstream. But I find most companies seems to follow
practices without understanding or adopting underlying foundations for these methodologies.

Agile manifesto describes four principles such as:
Individuals and interactions over processes and tools; Working software over
comprehensive documentation; Customer collaboration over contract negotiation;
Responding to change over following a plan. But how do we adopt these
principles. Most people focus on practices, for example they may follow
practices of XP such as
Pair Programming, TDD, Refactoring, Small Releases, Continuous
Integration, Collective Ownership, Sustainable Work Pace, etc. without
accepting all of its values. Crystal Clear has similar properties, though some
of they are like practices such as Frequent Delivery, Reflective Improvement,
and Automated Testing, whereas others are like principles such as
Osmotic Communication, Collaboration, Personal Safety.
Scrum is PM based process that
practices such as Frequent delivery, Daily Scrums and principles such as
Customer focus, Transparency, Energized work hours.
Lean Software Development is
largely based on practices such as Delivery Fast and TDD (Quality) and
principles such as Eliminate Waste, Knowledge, Defer Commitment and Respect.
ASD don’t have practices and is
largely based on principles such as Speculate, Collaborate and Learning.
I don’t include RUP,
though there are a number of agile variations.
Similarly, FDD is also iterative
based process, but it’s a bit more BDUF similar to RUP.
Though, I have not looked closely at DSDM and PRINCE2, but they seem to be
general purpose PM based methodologies.

As you can see there are common themes across them such as:

• Short Iterative Development
• Up front Quality using Testing, Pairing
• Communication/Reflective Environment
• Transparency/Big Visible Charts
• Respect/Personal Safety
• Collaboration

However, at deeper level agile methodologies are about two things: egalitarian
culture and giving up control. In agile organizations, everyone works
for a common goal without any selfish motives. Principles such as collaboration,
honest communication, self-organizing teams, responsibility, accountability
requires people will do the right thing without any kind of control. This is
probably biggest reason, why agile methodologies are adopted only at skin
level because most people are driven by personal success, ambition and power.
At many companies, people use information as a tool of control and power. Some
of the practices such as respect, collaboration and honest communication
are hard.
Agile processes assume developers, customers and core stake holders have
same commitment. However, in practice it often falls short. For example,
XP recommends collective ownership of code that can be really difficult in
teams where a few developers are not responsible and don’t believe in
excellence.

Second, Agile methodologies are all about giving control: the management
gives control of planning, deliverables to the development team.
Most companies are run based on command-control and don’t give up such control.
The
development in turn gives control to the customers for dictating what needs
to be delivered. The development team work on features defined by customers,
they can’t just spent months on big architectures or frameworks without
providing real software. Obviously, it works only if everyone is collaborating,
communicating and have a reward system that favors these principles (instead of usual hero based culture or firefighting culture).

In most places, clients don’t trust software companies, that is why they
work on big requirement specification because otherwise they will be
cornered into expensive change-request cycles. These companies create RFPs
and software/consulting companies respond by providing services for entire
project. On the other hand agile projects cannot be run based on BRUF,
they need continuous collaboration and the best form of delivery is piece
meal. It means the software developers may risk being fired if they
can’t deliver in first few iterations.

Rules for development in large IT shops
According to Conway’s law, the architecture should match organization’s style
of business and culture. I find working with large number of developers
and integrating with large number of components imposes limitation on
what technologies you can use or the processes you can adapt. Here are
few rules that I have found to be useful in these situation:

• Start with a few good people: Software development
  is still very much art, which is learned through actual experience and
  schools don’t prepare for real life development environment. So before
  starting any project, choose good people who are experienced,
  responsible and care about success for the project. I must also add that stay
  away from prima donnas (or big titles like enterprise architects).

• Minimize dependencies: If there is a single rule in software development
  that should be religously followed it’s loosely coupled components. It
  does not matter if you use object-oriented or any other methodology, if
  your system is not loosely coupled it becomes nightmare to maintain.
  Dependency Inversion Principle ( DIP) or separation of interfaces from
  implementation is a classic way to do it. However, I have seen it break
  down when interfaces churn too much. So, I will add defining generic
  interfaces. Clearly approaches like REST or applying functional language
  mechanism (things like closures or command pattern) are way to do it. Though,
  they may not be applicable to all interfaces, but any interfaces that
  have a lot dependencies or clients should use these approaches. These
  approaches also minimize any compile time dependencies, so you don’t have
  to update all clients when update your software.

• Physical Isolated Deployment: Besides using loosely
  coupled design, it’s also important that different services especially
  written by different teams are deployed on separate processes. J2EE
  application servers have pretty good isolation support via per application
  class loader, but it’s important to isolate malfunction or misconfiguration
  of one application from another. It also helps trouble shooting because
  you can look at log files for just one application, instead of searching
  through big log file for multiple applications.

• Minimize Team dependencies: Other than software dependencies, teams
  should be created so that there is minimal interaction needed among different
  teams. Through, Tom Demarco’s Peopleware, Fredrick Brook’s Mythical Man month
  and Alistair Cockburn’s Crystal methodologies, we learn that as you add more
  people to team you add overhead of communication and though communication
  between a single team can be managed (especially if they are colocated),
  however communication between departments or groups add much more overhead.
  This overhead becomes much more when these teams have different
  priorities and agenda.

  Generally, organization structure is of three type:
  horizontal or specialized teams such as GUI team, backend team, database
  team, systems team, etc. Vertical or functional teams where each team has
  mixture of specialization and combination of these two. On top of it,
  management structure can also influence coordination between the teams
  such as pyramids, cross functional or matrix organizations. The goal should
  be to keep the team as cross functional as possible and all domain experts,
  and technical experts are within the team so that there is minimal time
  wasted in coordination. Though, with large organization this will not be
  entirely possible as there will always be separate systems group or
  database group, but with matrix organizations those people can be assigned
  to specific projects. The downside of matrix organization is that often
  people have to report to more than one project manager in addition to
  their own manager and they will have to do a lot of multitasking. Both of
  these downsides can severely impact effectiveness of cross functional
  teams and should be avoided if all possible. Finally, colocate the team
  in one area as it adds osmotic communication and builds team gellness. One
  more thing on gellness, I have found that it takes several months for a
  new team to really work as a team or create gelled team. However, another
  drawback of matrix organizations is that it breaks team after each project,
  which should be avoided if all possible.

• Which technology to use?
  There are three types of technologies, first is backed by industry or open
  standards, second is backed by defacto standards and third is proprietary.
  The safest choice for large corporation is open technology,
  which is backed by industry consortiums or task forces such as ISO, ITU,
  OMG, IETF, Oasis, W3C, etc. The downside of open technology is that often
  process of approval is very slow and often result of agreement is least
  common denominator. Though, interoperability is biggest strength of
  open technology, but often it is not realized and each vendor adds in their
  own hooks. The defacto technologies are popular way of doing things that
  may be backed by one vendor or open source effort and adopted by others
  such as Spring, Hibernate, RoR, and even J2EE to some extent.
  Third type is proprietary technology that are controlled by a single vendor
  such as .Net, Macromedia Flash/Flex, Tangosol Coherence, etc. Often large
  companies have a single technology policy such as Java or .Net shop, which
  can be good because there are skilled people to tackle any problem, on the
  other hand solving every problem with same tool or technology may not be
  great idea.

  After working on a number of J2EE projects for over five years, I have been
  working on open source technologies such as Spring, Hibernate, Acegi, etc for a couple of years.
  And though, I like them because of their support for POJO style programming,
  which are easier to test. However, I don’t call them simple or lightweight
  as they are often referred. Spring has become huge and it now pretty much
  covers everything and kitchen sink. I found biggest problems with spring was
  managing configuration files especially between teams. When you have to
  load dozens of spring files from different projects into same
  class loader, it becomes really hard to trouble shoot. You also have to be
  very careful with bean-ids so that there is no unexpected
  behavior when you are loading a lot spring files from different teams.
  Similarly, Hibernate despite being very powerful is immensley
  complicated and things like caching, lazy loading often are hard to debug.
  Spring also covers security (Acegi), remoting and transactions. However,
  one thing that it does not cover as well as JavaEE is application isolation
  and clustering.
  When running multiple services from different groups under one VM using
  JavaSE is also very hard especially if each service has different versions
  of dependent jar files and in large companies these cases are too common.
  So, I suggest use J2EE as a service container instead of Spring, Rmi, Jini
  or alike when you have a lot of interacting services. It takes care of
  security, class loading, clustering, etc (not to mention huge GC overhead
  caused by RMI or non-JERI JINI). And with EJB 3.0, there is virtually a little
  difference between Spring and EJB and you can take advantage of POJO style
  coding and testing.

• Which language to choose? Similar to technologies, it’s better to choose
  open or defacto based language such as C++, C#, Java. Again, certain problems
  might require special technologies and languages so these rules should not
  be hard. However, in large companies often people skills vary and it’s
  easier to use popular languages. I find that with small group, you can
  control quality and communication, but with large community, your weakest
  link are average developers. So, you must choose technology that is well
  understood by mass developers and not niche technology understood by handful
  people. There is a lot of debate of static language vs dynamic language.
  However, I find dynamic languages a bit dangerous with large number of
  developers. I have seen that the average quality of code in most
  organizations is pretty low and tests are not widely used. So, I will
  be scared to leave a lot of errors at runtime. Also, despite the claims,
  most dynamic languages are still too slow for enterprise level scalability
  and I caution their use or at least minimize
  their use. For example, strength of a number of dynamic language based
  frameworks such as RoR or Django is web development so they can be used
  for that purpose leaving backend in other languages.

• Keep it simple and Evolve: I have seen too many
  sophisticated frameworks that are designed top down and are just too
  complicated and difficult to use despite being powerful. The
  bottom line is that any framework that interferes with development is just
  not worth it despite how powerful it is. The framework and tools are built
  to ease development and best one stay out of developer’s way. So, don’t use
  complicated XML/XPath/Schema based configuration if simple property files
  or database can work. Don’t define too many facades and layers that adds
  overhead of development. Best frameworks start small and evolved based on
  actual requirements. Keep framework simple and low-level and don’t try
  to do too much in them.

• Evolutionary and Emergent Design: I have been big fan
  of agile methodologies and I have personally seen that best systems start
  small and evolve. The only thing I would add is that with this evolution,
  existing system should be refactored continuously because otherwise they
  become useless.

• Simplify development: Development often has a cycle of: design, write
  failing test, write code, test, make it pass, promoting to configuration
  management and finally releasing the software. Though, often people
  only do unit tests, but many things require integration testing and require
  deploying services. In this regard, statically typed languages such as Java
  can be very slow. It might take 15 minutes to compile and deploy a single
  change. In this regard, I like dynamic languages that make this as painless
  as possible. However, if you are using Java then use hot deployment to
  reduce any startup time.

• Take pride with Quality work: People who take pride
  on their work feel responsible for the success. And their work will show
  that they care about quality whether it is with good testing or better
  quality code. Again, this shows that fundamental for any project is good
  people.

• Deployment support: Services should be simple to deploy and should not
  have to be restarted for simple configuration changes. So, use automatic
  reload of configuration files or database based properties. Also, everything
  needed for deployment should packaged in a single deployment unit (again
  J2EE simplifies it), because relying on external file structure is always
  problematic. Often deployment
  goes through different environments such as Integration, Staging, and
  Production and some of the properties needs to be changed based on
  environment, so provide a simple way of override. You can use simple
  override property files or database for this.

• Debug and Trouble shooting support: Once services are deployed, often
  you have to trouble shoot issues. So allow logging additional information
  based on some configuration such as JMX. In a large distributed system,
  debugging is often a challenge because you have to dig through multiple log
  files on different machine. So it’s generally good idea to define
  a transaction-id for each user request and write tools to search these
  log files easily. See my previous blog for more information.
  You can also provide tools to notify and send you alerts
  when certain error conditions occur.

• Monitoring: Enterprise applications require carefully monitoring. Often
  different matrices are collected needed for support or business folks. So,
  design the application with these monitoring in mind.

• Security: Most application will require security capabilities such
  as permissions, role based security, instance or data based security.

• Standardized Toolset: Most large companies write
  customized development libraries and tools for deployment, build,
  dependency management, monitoring, etc. However, often there is “Not Invented Here” syndrome and countless effort is devoted when similar tools are available
  in open source or commercial space. These companies ignore the learning
  curve each new hire goes through and even the experienced developers don’t
  understand those tools. Again, this goes along with using popular
  technologies so that there isn’t any learning curve required and off the
  shelf solutions are available.

• Introspection: One of the thing that most agile methodologies promote
  is introspection and it’s very necessary that after each release cycle, some
  soul searching is done and an effort is made to find where unneeded time
  is spent. You may use Theory of Constraints to find the biggest time waster
  and iteratively eliminate the waste.

• Conclusion: Above rules and guidelines are lesson learned from my
  my 15 years of experience. On a final note, I should indicate that not all
  of these guidelines indicate my ideal workspace or preferences. For example,
  I like dynamic languages despite their slow performance. In fact, ideally I
  would like to work with no more than five people and with good people you can
  solve any problem that may require 100+ people. It all depends on narrowing
  scope to essential requirements based on 20/80 rule. However, in big
  organizations, you have to find a solution within the constraints of
  organization environment, which may not be optimal but is best suited for
  that organization.