Shahzad Bhatti Welcome to my ramblings and rants!

Nexus for next generation languages
It’s said every ten to fifteen years you need a new programming language.
Today, Java is over ten years old and one of the most popular language.
Though, C# is another important language, but its design for most part is
similar to Java. A lot of people lately have been trying to find next
generation language. I bought Bruce Tate’s book “Beyond Java” hoping to
get some insight, but was somewhat disappointed as he only covered Ruby and
despite the fact he mentioned other languages like Lisp, Python, Smalltalk,
he didn’t really elaborated. Also, he ignored many other neat languages like
Haskell, Scala, PHP, Groovy. To some extent, a lot of these languages don’t
present fundamentally new idea, but extend a lot of ideas from 70s that were
first used in Lisp and later Smalltalk such as data-driven paradigm using
closures, functional programming, meta programming, and generics programming.
One of the battle going on between widely popular languages like Java/C# and
scripting languages is acceptance of scripting languages in large organizations
or in enterprise environment. Both James Gosling and James McGovern have been
trying to rip scripting languages for their lack of speed and lack of support
for enterprise environment. To some extent, there is truth to that, but as
most of the evangelists for the new scripting languages have been saying is
that not every organization needs sophisticated application servers, messaging
middlewares, EIS or ESB. One of interesting revelation came out in recent
symposium of theserverside, where Dian Almaer took survey of Rails in
production and only found one. Though, according to Mike Clark who was also
there said there were about four. It clearly shows that most people are
looking for bridge to connect with scripting languages.
I think one of the biggest effect of scripting
languages is emphasis on simplicity. It doesn’t matter if Fortune 500 can’t
use these scripting languages due to their complexed environment, but rest of
tens of thousand organizations from small and medium size can greatly get
benefit from them.

One of biggest barrier in acceptance of these scripting languages is lack
of Nexus. If we look at last twenty years, the new languages become widely
popular only when they created powerful nexus between existing language and
the new language. It’s sort of like hand-off from old-generation Star Trek
to new-generation Star Trek. For example, despite the fact C++ was object-oriented, but it accepted C syntax and allowed programmers to slowly adapt to it.
Similarly, Java’s syntax was very similar to C++ and despite the fact that
it was complete object-oriented, it used primitive types to give some compfort
to C++ programmers. This is why next generation language needs a nexus. I
thought Groovy might create that nexus for Java, but was very disappointed.
I like Ruby a lot, but I wish JRuby could be fully compatible and would be
able to run all libraries and frameworks like Rails. This is one of the reason,
I like Scala a lot, because it offers a lot of benefits of data-flow based
programming similar to Lisp, Ruby or Smalltalk, functional programming
similar to Haskell and full Generics support. On top of it, it can run both
on .NET’s CLR and Java’s VM. Unfortunately, it hasn’t got much attention as
other scripting languages like Ruby or Python.

There is a heating debates going on between scripting languages such as Python and Ruby against corporate friendly languages like Java and C#.
First, Gostling Claimed on JDJ that scripting languages are very simple and limited their use for simple
web pages. Then he derided scripting languages for lack of performance on Artima’s weblog.
Another flaming post by James McGovern suggested that Ruby is train reck waiting for happen.
Gosling tried to clarify in his blog, but it does not help much. He still tries to corner
scripting languages into domain specific. Despite the performance issues with the scripting languages like Ruby, Python, PHP, and Scheme, they are gaining popularity due to simplification and programmer productivity.
As hardware has been becoming more and more powerful, the performance issues would become less important. There are also plans for Java like VMs in Ruby, which can make it more acceptable to corporate world.
There has been growing trend in solutions that are open source and easier to develop, and many corporations resist for the fear of loosing market. The time is right for scripting languages and I hope languages like
Ruby, Python, Haskell, Scala, Smalltalk or Lisp variants become more acceptable in most places.

Design by Committee
Most large organizations have an architecture committee, platform team,
center of excellence or other forms of review committees that determine
vision and direction of software projects along with technology. It is
generally good idea to have a uniform vision and a fewer set of technologies
used across projects in order to minimize maintenance and learning curve.
One of the things that these committees are responsible is the enterprise
architecture. The enterprise architecture includes hardware platform,
software platform, tools, etc. In addition, it also includes application
architecture that determines how the system will be broken into subsystems.
In some respect these committees work similar to industry standard bodies or
consortium of organizations. Depending on size and diversity of these
committees, often the design process becomes somewhat bureaucratic. The
situation can be worse if you follow strict IT governance practices or ITIL
process.
The disagreement often results in lowest common denominator or low
quality solution.
Another frequest observation is that solution is often overly complexed
and over-engineered for the problem domain. Worse, due to diversity, the
architecture is not uniform and consistent. One of the reason for
such inconsistent or complexed architure is pissing contest. In general,
software architects don’t work well with other architects and the design
meetings turn into show-offs for coming up with most clever solution.
On the other hand, an architecture by a single experienced person is often
simple and consistent. The open source community offers many examples
of creating a simple solutions that works bettern than design
by committee’s. The open source community often uses the term “benevolent
dictator” who is incharge of unified vision and architecture. So,
the software project should probably be only designed by a single architect.
Software architecture is still an art that is learned through apprenticeship.
So, it helps if there is an apprentice who can do the grunt work such as
documenting designs and taking care of details. However, I don’t mean an
architect who does not do hands on work, but rather similar to
James O’ Coplien’s pattern “Architects-Also-Implements”.

One of best ideas from agile methodologies is the balance of work and life. In the original XP book, Kent Beck laid down 40-hour week as one of the core practices of XP. However, in the second edition he started the term “Energized Work”, which is a bit vague. I guess if you want to be less controversial then choose more vague terms. Anyway, the idea is pretty much same that knowledge work such as developing software needs a relaxed mind. I have seen everywhere that when you are tired you make stupid mistakes and spend next day or days making up for it.

I read another interesting article Be smarter at work, slack off that reinforces same idea. It quotes Peter Capelli, a professor of management at Wharton, “You can turn a smart person into an idiot just by overworking him.”

Nevertheless, most of the real work still requires 50-60 hours a week and I have seen plenty of death-march projects and fear of loss of jobs to offshore has made things much worse.

Log locally and query globally I have found the rule “log locally and query globally” quite handy in large distributed applications. However, I still find many projects that try to create a centralized logging server for receiving all log messages. In many cases this adds too much overhead for logging. Though, asynchronous logging can reduce direct overhead, but it is still problematic. This also results in loss of log messages or confusion when the log server crashes. A single logging server often becomes bottleneck for resources such as network, disk and CPU. I admit I have designed some systems where I setup centralized logging service. In some instances I used wrapper on top of UNIX’s syslog server. One of the drawback of syslog was that it drops messages under heavy load. Though, syslog service provides useful logging for system level logs where you don’t want to log system related messages locally especially security violations because if the local machine is compromised you will loose all logging information. However, I have found for application logging, local logging works much better. On the other hand this requires that a centralized service exist that can search logs from all machines. One of
the difficulty with local logging is creating a complete picture of logs. For example a user request goes throught various services running on different servers and for debugging it is important to trace complete business transaction information. In such cases, I have found a transaction correlation id quite useful. For example, each business transaction id can be stored as part of the log messages or user’s session information can be used, which is later
used to create complete picture. Also, it is important that all servers use NTP service and have their clocks synchronized so that the log messages are in right time order. For query, you can create a local searching agent on each machine which is contacted by centralized query system. This way you can query multiple log files simultaneously and return results much more
efficiently.

Based on Jim O’ Coplien’s book on organization patterns, you can not give responsibility to anyone. Only someone can accept it, however you can hold someone accountable. According to Kent Beck, accountability entails reporting back on what happened and responsibility means sticking with the problem in trying to solve it. However, you cannot hold someone accountable if you don’t provide enough resources to him to complete his job. I think this last statement is specially true and I have dealt with numerous situations when management never provided tools or resources to finish the job or do the job right. For example, recently I was trying to track an application bug that caused some financial damage. However, due to complexity of the system, the log files were of terabytes. Worse, there isn’t any data-mining tools to query the logs instead you have to do zgrep on huge compressed files and it takes days to search. In the meantime management keeps bugging when will the financial data be ready. This is another example of an organization where daily firefighting prevents anyone to think about any long term solutions to common problems.

Glory of Firefighting
I have observed in a few dysfunctional organizations where firefighting
is the norm and is rewarded. In such organization normal process of software
development is viewed as mundane and people do poor quality job. However,
they use this low quality process to become heros when crisis emerge. In
one of the organization, they created special roles and teams like SWAT
team that handles crisis or firefighting. In these organizations, emphasis
is on firefighting rather than fire-prevention.

A few old usenet postings:
I guess not many people usenet anymore, here are links to some of my past
postings:

• majordomo installation problem Dec 3 1996, 2:00 am
• ObjectInput/OutputStream with Sockets Jul 24 1997, 1:00 am
• setting cookies in java application (URL) Aug 24 1998, 1:00 am
• RMI question

Software rewrites
I just read a great Blog entry of Jim Shore about Software Rewrite. I like many have been in situation when I just wished that if we could redo, the application would have been much better. As some people say always do things twice
and throw away your first try. However, when you are working on a large
complicated software, despite the temptation it can be enormous effort.
According to Jim, If your platform or language does not change, dont’ throw
away your code instead refactor or redesign to work with existing software.
I have heard similar advince from Joel Spolsky. I have been involved with a few
projects with complete rewrites. In early 90s, there was widespread use of
the term “Re-engineering”, that related with business re-engineering and
encompassed a lot of rewrites of software. Later, I was involved with an
ITS project for Illinois Department of Transportation, where we rewrote old
CTIC project, which was written in C/UNIX and wrote new system
GCM Travel in C++/Java/CORBA. It
turned out to be major effort and over budget and years late. Unlike
private company, the government had plenty of tax dollars to keep spending
on the project. When I was at United, I worked on prettty badly written
software and I had to decide whether to continue with the software or rewrite.
I opted for incremental refactoring because unlike CTIC project, I had to
not only write new version, but had to maintain old software. It is terribly
risky strategy to maintain two versions of software. Martin Fowler also has
Strangler
Pattern he describes for rewriting software. Unfortunately, I am
currently involved with another grand rewrite project (it was started
before I joined). It also has requirement to maintain two different
versions of the software and I can see a lot of time will be spend on
integrating two different versions. I have already seen people taking
short cuts to finish the project and before you would know it, it is
going to follow same road that previous project did.

Finished Personal and Company’s Web Redesign
Since I moved my personal website from my school to my own domain back in 1998,
I did little touch up. So, I finally spent some time to change the looks. I
then realized my company’s website can take a new look too as that has not been
changed since around the same time. I wanted to keep both sites easy to
maintain without any dynamic content management, so I used same technique
that I have been using over ten years, i.e., wrote some templates and created
Ruby scripts to generate html files. It’s not state of the art, but given
my server isn’t too powerful, I think it will work.