Shahzad Bhatti

October 3, 2006

Rules for development in large IT shops

Filed under: Computing — admin @ 10:10 pm

Rules for development in large IT shops
According to Conway’s law, the architecture should match organization’s style
of business and culture. I find working with large number of developers
and integrating with large number of components imposes limitation on
what technologies you can use or the processes you can adapt. Here are
few rules that I have found to be useful in these situation:

  • Start with a few good people: Software development
    is still very much art, which is learned through actual experience and
    schools don’t prepare for real life development environment. So before
    starting any project, choose good people who are experienced,
    responsible and care about success for the project. I must also add that stay
    away from prima donnas (or big titles like enterprise architects).

  • Minimize dependencies: If there is a single rule in software development
    that should be religously followed it’s loosely coupled components. It
    does not matter if you use object-oriented or any other methodology, if
    your system is not loosely coupled it becomes nightmare to maintain.
    Dependency Inversion Principle ( DIP) or separation of interfaces from
    implementation is a classic way to do it. However, I have seen it break
    down when interfaces churn too much. So, I will add defining generic
    interfaces. Clearly approaches like REST or applying functional language
    mechanism (things like closures or command pattern) are way to do it. Though,
    they may not be applicable to all interfaces, but any interfaces that
    have a lot dependencies or clients should use these approaches. These
    approaches also minimize any compile time dependencies, so you don’t have
    to update all clients when update your software.

  • Physical Isolated Deployment: Besides using loosely
    coupled design, it’s also important that different services especially
    written by different teams are deployed on separate processes. J2EE
    application servers have pretty good isolation support via per application
    class loader, but it’s important to isolate malfunction or misconfiguration
    of one application from another. It also helps trouble shooting because
    you can look at log files for just one application, instead of searching
    through big log file for multiple applications.

  • Minimize Team dependencies: Other than software dependencies, teams
    should be created so that there is minimal interaction needed among different
    teams. Through, Tom Demarco’s Peopleware, Fredrick Brook’s Mythical Man month
    and Alistair Cockburn’s Crystal methodologies, we learn that as you add more
    people to team you add overhead of communication and though communication
    between a single team can be managed (especially if they are colocated),
    however communication between departments or groups add much more overhead.
    This overhead becomes much more when these teams have different
    priorities and agenda.

    Generally, organization structure is of three type:
    horizontal or specialized teams such as GUI team, backend team, database
    team, systems team, etc. Vertical or functional teams where each team has
    mixture of specialization and combination of these two. On top of it,
    management structure can also influence coordination between the teams
    such as pyramids, cross functional or matrix organizations. The goal should
    be to keep the team as cross functional as possible and all domain experts,
    and technical experts are within the team so that there is minimal time
    wasted in coordination. Though, with large organization this will not be
    entirely possible as there will always be separate systems group or
    database group, but with matrix organizations those people can be assigned
    to specific projects. The downside of matrix organization is that often
    people have to report to more than one project manager in addition to
    their own manager and they will have to do a lot of multitasking. Both of
    these downsides can severely impact effectiveness of cross functional
    teams and should be avoided if all possible. Finally, colocate the team
    in one area as it adds osmotic communication and builds team gellness. One
    more thing on gellness, I have found that it takes several months for a
    new team to really work as a team or create gelled team. However, another
    drawback of matrix organizations is that it breaks team after each project,
    which should be avoided if all possible.

  • Which technology to use?
    There are three types of technologies, first is backed by industry or open
    standards, second is backed by defacto standards and third is proprietary.
    The safest choice for large corporation is open technology,
    which is backed by industry consortiums or task forces such as ISO, ITU,
    OMG, IETF, Oasis, W3C, etc. The downside of open technology is that often
    process of approval is very slow and often result of agreement is least
    common denominator. Though, interoperability is biggest strength of
    open technology, but often it is not realized and each vendor adds in their
    own hooks. The defacto technologies are popular way of doing things that
    may be backed by one vendor or open source effort and adopted by others
    such as Spring, Hibernate, RoR, and even J2EE to some extent.
    Third type is proprietary technology that are controlled by a single vendor
    such as .Net, Macromedia Flash/Flex, Tangosol Coherence, etc. Often large
    companies have a single technology policy such as Java or .Net shop, which
    can be good because there are skilled people to tackle any problem, on the
    other hand solving every problem with same tool or technology may not be
    great idea.

    After working on a number of J2EE projects for over five years, I have been
    working on open source technologies such as Spring, Hibernate, Acegi, etc for a couple of years.
    And though, I like them because of their support for POJO style programming,
    which are easier to test. However, I don’t call them simple or lightweight
    as they are often referred. Spring has become huge and it now pretty much
    covers everything and kitchen sink. I found biggest problems with spring was
    managing configuration files especially between teams. When you have to
    load dozens of spring files from different projects into same
    class loader, it becomes really hard to trouble shoot. You also have to be
    very careful with bean-ids so that there is no unexpected
    behavior when you are loading a lot spring files from different teams.
    Similarly, Hibernate despite being very powerful is immensley
    complicated and things like caching, lazy loading often are hard to debug.
    Spring also covers security (Acegi), remoting and transactions. However,
    one thing that it does not cover as well as JavaEE is application isolation
    and clustering.
    When running multiple services from different groups under one VM using
    JavaSE is also very hard especially if each service has different versions
    of dependent jar files and in large companies these cases are too common.
    So, I suggest use J2EE as a service container instead of Spring, Rmi, Jini
    or alike when you have a lot of interacting services. It takes care of
    security, class loading, clustering, etc (not to mention huge GC overhead
    caused by RMI or non-JERI JINI). And with EJB 3.0, there is virtually a little
    difference between Spring and EJB and you can take advantage of POJO style
    coding and testing.

  • Which language to choose? Similar to technologies, it’s better to choose
    open or defacto based language such as C++, C#, Java. Again, certain problems
    might require special technologies and languages so these rules should not
    be hard. However, in large companies often people skills vary and it’s
    easier to use popular languages. I find that with small group, you can
    control quality and communication, but with large community, your weakest
    link are average developers. So, you must choose technology that is well
    understood by mass developers and not niche technology understood by handful
    people. There is a lot of debate of static language vs dynamic language.
    However, I find dynamic languages a bit dangerous with large number of
    developers. I have seen that the average quality of code in most
    organizations is pretty low and tests are not widely used. So, I will
    be scared to leave a lot of errors at runtime. Also, despite the claims,
    most dynamic languages are still too slow for enterprise level scalability
    and I caution their use or at least minimize
    their use. For example, strength of a number of dynamic language based
    frameworks such as RoR or Django is web development so they can be used
    for that purpose leaving backend in other languages.

  • Keep it simple and Evolve: I have seen too many
    sophisticated frameworks that are designed top down and are just too
    complicated and difficult to use despite being powerful. The
    bottom line is that any framework that interferes with development is just
    not worth it despite how powerful it is. The framework and tools are built
    to ease development and best one stay out of developer’s way. So, don’t use
    complicated XML/XPath/Schema based configuration if simple property files
    or database can work. Don’t define too many facades and layers that adds
    overhead of development. Best frameworks start small and evolved based on
    actual requirements. Keep framework simple and low-level and don’t try
    to do too much in them.

  • Evolutionary and Emergent Design: I have been big fan
    of agile methodologies and I have personally seen that best systems start
    small and evolve. The only thing I would add is that with this evolution,
    existing system should be refactored continuously because otherwise they
    become useless.

  • Simplify development: Development often has a cycle of: design, write
    failing test, write code, test, make it pass, promoting to configuration
    management and finally releasing the software. Though, often people
    only do unit tests, but many things require integration testing and require
    deploying services. In this regard, statically typed languages such as Java
    can be very slow. It might take 15 minutes to compile and deploy a single
    change. In this regard, I like dynamic languages that make this as painless
    as possible. However, if you are using Java then use hot deployment to
    reduce any startup time.

  • Take pride with Quality work: People who take pride
    on their work feel responsible for the success. And their work will show
    that they care about quality whether it is with good testing or better
    quality code. Again, this shows that fundamental for any project is good
    people.

  • Deployment support: Services should be simple to deploy and should not
    have to be restarted for simple configuration changes. So, use automatic
    reload of configuration files or database based properties. Also, everything
    needed for deployment should packaged in a single deployment unit (again
    J2EE simplifies it), because relying on external file structure is always
    problematic. Often deployment
    goes through different environments such as Integration, Staging, and
    Production and some of the properties needs to be changed based on
    environment, so provide a simple way of override. You can use simple
    override property files or database for this.

  • Debug and Trouble shooting support: Once services are deployed, often
    you have to trouble shoot issues. So allow logging additional information
    based on some configuration such as JMX. In a large distributed system,
    debugging is often a challenge because you have to dig through multiple log
    files on different machine. So it’s generally good idea to define
    a transaction-id for each user request and write tools to search these
    log files easily. See my previous blog for more information.
    You can also provide tools to notify and send you alerts
    when certain error conditions occur.

  • Monitoring: Enterprise applications require carefully monitoring. Often
    different matrices are collected needed for support or business folks. So,
    design the application with these monitoring in mind.

  • Security: Most application will require security capabilities such
    as permissions, role based security, instance or data based security.

  • Standardized Toolset: Most large companies write
    customized development libraries and tools for deployment, build,
    dependency management, monitoring, etc. However, often there is “Not Invented Here” syndrome and countless effort is devoted when similar tools are available
    in open source or commercial space. These companies ignore the learning
    curve each new hire goes through and even the experienced developers don’t
    understand those tools. Again, this goes along with using popular
    technologies so that there isn’t any learning curve required and off the
    shelf solutions are available.

  • Introspection: One of the thing that most agile methodologies promote
    is introspection and it’s very necessary that after each release cycle, some
    soul searching is done and an effort is made to find where unneeded time
    is spent. You may use Theory of Constraints to find the biggest time waster
    and iteratively eliminate the waste.

  • Conclusion: Above rules and guidelines are lesson learned from my
    my 15 years of experience. On a final note, I should indicate that not all
    of these guidelines indicate my ideal workspace or preferences. For example,
    I like dynamic languages despite their slow performance. In fact, ideally I
    would like to work with no more than five people and with good people you can
    solve any problem that may require 100+ people. It all depends on narrowing
    scope to essential requirements based on 20/80 rule. However, in big
    organizations, you have to find a solution within the constraints of
    organization environment, which may not be optimal but is best suited for
    that organization.

No Comments

No comments yet.

RSS feed for comments on this post. TrackBack URL

Sorry, the comment form is closed at this time.

Powered by WordPress