Shahzad Bhatti Welcome to my ramblings and rants!

Agile Methodologies Under the Hood
Based on research and Scott Ambler’s article, agile methodologies have crossed
chasm and moved to mainstream. But I find most companies seems to follow
practices without understanding or adopting underlying foundations for these methodologies.

Agile manifesto describes four principles such as:
Individuals and interactions over processes and tools; Working software over
comprehensive documentation; Customer collaboration over contract negotiation;
Responding to change over following a plan. But how do we adopt these
principles. Most people focus on practices, for example they may follow
practices of XP such as
Pair Programming, TDD, Refactoring, Small Releases, Continuous
Integration, Collective Ownership, Sustainable Work Pace, etc. without
accepting all of its values. Crystal Clear has similar properties, though some
of they are like practices such as Frequent Delivery, Reflective Improvement,
and Automated Testing, whereas others are like principles such as
Osmotic Communication, Collaboration, Personal Safety.
Scrum is PM based process that
practices such as Frequent delivery, Daily Scrums and principles such as
Customer focus, Transparency, Energized work hours.
Lean Software Development is
largely based on practices such as Delivery Fast and TDD (Quality) and
principles such as Eliminate Waste, Knowledge, Defer Commitment and Respect.
ASD don’t have practices and is
largely based on principles such as Speculate, Collaborate and Learning.
I don’t include RUP,
though there are a number of agile variations.
Similarly, FDD is also iterative
based process, but it’s a bit more BDUF similar to RUP.
Though, I have not looked closely at DSDM and PRINCE2, but they seem to be
general purpose PM based methodologies.

As you can see there are common themes across them such as:

• Short Iterative Development
• Up front Quality using Testing, Pairing
• Communication/Reflective Environment
• Transparency/Big Visible Charts
• Respect/Personal Safety
• Collaboration

However, at deeper level agile methodologies are about two things: egalitarian
culture and giving up control. In agile organizations, everyone works
for a common goal without any selfish motives. Principles such as collaboration,
honest communication, self-organizing teams, responsibility, accountability
requires people will do the right thing without any kind of control. This is
probably biggest reason, why agile methodologies are adopted only at skin
level because most people are driven by personal success, ambition and power.
At many companies, people use information as a tool of control and power. Some
of the practices such as respect, collaboration and honest communication
are hard.
Agile processes assume developers, customers and core stake holders have
same commitment. However, in practice it often falls short. For example,
XP recommends collective ownership of code that can be really difficult in
teams where a few developers are not responsible and don’t believe in
excellence.

Second, Agile methodologies are all about giving control: the management
gives control of planning, deliverables to the development team.
Most companies are run based on command-control and don’t give up such control.
The
development in turn gives control to the customers for dictating what needs
to be delivered. The development team work on features defined by customers,
they can’t just spent months on big architectures or frameworks without
providing real software. Obviously, it works only if everyone is collaborating,
communicating and have a reward system that favors these principles (instead of usual hero based culture or firefighting culture).

In most places, clients don’t trust software companies, that is why they
work on big requirement specification because otherwise they will be
cornered into expensive change-request cycles. These companies create RFPs
and software/consulting companies respond by providing services for entire
project. On the other hand agile projects cannot be run based on BRUF,
they need continuous collaboration and the best form of delivery is piece
meal. It means the software developers may risk being fired if they
can’t deliver in first few iterations.

Rules for development in large IT shops
According to Conway’s law, the architecture should match organization’s style
of business and culture. I find working with large number of developers
and integrating with large number of components imposes limitation on
what technologies you can use or the processes you can adapt. Here are
few rules that I have found to be useful in these situation:

• Start with a few good people: Software development
  is still very much art, which is learned through actual experience and
  schools don’t prepare for real life development environment. So before
  starting any project, choose good people who are experienced,
  responsible and care about success for the project. I must also add that stay
  away from prima donnas (or big titles like enterprise architects).

• Minimize dependencies: If there is a single rule in software development
  that should be religously followed it’s loosely coupled components. It
  does not matter if you use object-oriented or any other methodology, if
  your system is not loosely coupled it becomes nightmare to maintain.
  Dependency Inversion Principle ( DIP) or separation of interfaces from
  implementation is a classic way to do it. However, I have seen it break
  down when interfaces churn too much. So, I will add defining generic
  interfaces. Clearly approaches like REST or applying functional language
  mechanism (things like closures or command pattern) are way to do it. Though,
  they may not be applicable to all interfaces, but any interfaces that
  have a lot dependencies or clients should use these approaches. These
  approaches also minimize any compile time dependencies, so you don’t have
  to update all clients when update your software.

• Physical Isolated Deployment: Besides using loosely
  coupled design, it’s also important that different services especially
  written by different teams are deployed on separate processes. J2EE
  application servers have pretty good isolation support via per application
  class loader, but it’s important to isolate malfunction or misconfiguration
  of one application from another. It also helps trouble shooting because
  you can look at log files for just one application, instead of searching
  through big log file for multiple applications.

• Minimize Team dependencies: Other than software dependencies, teams
  should be created so that there is minimal interaction needed among different
  teams. Through, Tom Demarco’s Peopleware, Fredrick Brook’s Mythical Man month
  and Alistair Cockburn’s Crystal methodologies, we learn that as you add more
  people to team you add overhead of communication and though communication
  between a single team can be managed (especially if they are colocated),
  however communication between departments or groups add much more overhead.
  This overhead becomes much more when these teams have different
  priorities and agenda.

  Generally, organization structure is of three type:
  horizontal or specialized teams such as GUI team, backend team, database
  team, systems team, etc. Vertical or functional teams where each team has
  mixture of specialization and combination of these two. On top of it,
  management structure can also influence coordination between the teams
  such as pyramids, cross functional or matrix organizations. The goal should
  be to keep the team as cross functional as possible and all domain experts,
  and technical experts are within the team so that there is minimal time
  wasted in coordination. Though, with large organization this will not be
  entirely possible as there will always be separate systems group or
  database group, but with matrix organizations those people can be assigned
  to specific projects. The downside of matrix organization is that often
  people have to report to more than one project manager in addition to
  their own manager and they will have to do a lot of multitasking. Both of
  these downsides can severely impact effectiveness of cross functional
  teams and should be avoided if all possible. Finally, colocate the team
  in one area as it adds osmotic communication and builds team gellness. One
  more thing on gellness, I have found that it takes several months for a
  new team to really work as a team or create gelled team. However, another
  drawback of matrix organizations is that it breaks team after each project,
  which should be avoided if all possible.

• Which technology to use?
  There are three types of technologies, first is backed by industry or open
  standards, second is backed by defacto standards and third is proprietary.
  The safest choice for large corporation is open technology,
  which is backed by industry consortiums or task forces such as ISO, ITU,
  OMG, IETF, Oasis, W3C, etc. The downside of open technology is that often
  process of approval is very slow and often result of agreement is least
  common denominator. Though, interoperability is biggest strength of
  open technology, but often it is not realized and each vendor adds in their
  own hooks. The defacto technologies are popular way of doing things that
  may be backed by one vendor or open source effort and adopted by others
  such as Spring, Hibernate, RoR, and even J2EE to some extent.
  Third type is proprietary technology that are controlled by a single vendor
  such as .Net, Macromedia Flash/Flex, Tangosol Coherence, etc. Often large
  companies have a single technology policy such as Java or .Net shop, which
  can be good because there are skilled people to tackle any problem, on the
  other hand solving every problem with same tool or technology may not be
  great idea.

  After working on a number of J2EE projects for over five years, I have been
  working on open source technologies such as Spring, Hibernate, Acegi, etc for a couple of years.
  And though, I like them because of their support for POJO style programming,
  which are easier to test. However, I don’t call them simple or lightweight
  as they are often referred. Spring has become huge and it now pretty much
  covers everything and kitchen sink. I found biggest problems with spring was
  managing configuration files especially between teams. When you have to
  load dozens of spring files from different projects into same
  class loader, it becomes really hard to trouble shoot. You also have to be
  very careful with bean-ids so that there is no unexpected
  behavior when you are loading a lot spring files from different teams.
  Similarly, Hibernate despite being very powerful is immensley
  complicated and things like caching, lazy loading often are hard to debug.
  Spring also covers security (Acegi), remoting and transactions. However,
  one thing that it does not cover as well as JavaEE is application isolation
  and clustering.
  When running multiple services from different groups under one VM using
  JavaSE is also very hard especially if each service has different versions
  of dependent jar files and in large companies these cases are too common.
  So, I suggest use J2EE as a service container instead of Spring, Rmi, Jini
  or alike when you have a lot of interacting services. It takes care of
  security, class loading, clustering, etc (not to mention huge GC overhead
  caused by RMI or non-JERI JINI). And with EJB 3.0, there is virtually a little
  difference between Spring and EJB and you can take advantage of POJO style
  coding and testing.

• Which language to choose? Similar to technologies, it’s better to choose
  open or defacto based language such as C++, C#, Java. Again, certain problems
  might require special technologies and languages so these rules should not
  be hard. However, in large companies often people skills vary and it’s
  easier to use popular languages. I find that with small group, you can
  control quality and communication, but with large community, your weakest
  link are average developers. So, you must choose technology that is well
  understood by mass developers and not niche technology understood by handful
  people. There is a lot of debate of static language vs dynamic language.
  However, I find dynamic languages a bit dangerous with large number of
  developers. I have seen that the average quality of code in most
  organizations is pretty low and tests are not widely used. So, I will
  be scared to leave a lot of errors at runtime. Also, despite the claims,
  most dynamic languages are still too slow for enterprise level scalability
  and I caution their use or at least minimize
  their use. For example, strength of a number of dynamic language based
  frameworks such as RoR or Django is web development so they can be used
  for that purpose leaving backend in other languages.

• Keep it simple and Evolve: I have seen too many
  sophisticated frameworks that are designed top down and are just too
  complicated and difficult to use despite being powerful. The
  bottom line is that any framework that interferes with development is just
  not worth it despite how powerful it is. The framework and tools are built
  to ease development and best one stay out of developer’s way. So, don’t use
  complicated XML/XPath/Schema based configuration if simple property files
  or database can work. Don’t define too many facades and layers that adds
  overhead of development. Best frameworks start small and evolved based on
  actual requirements. Keep framework simple and low-level and don’t try
  to do too much in them.

• Evolutionary and Emergent Design: I have been big fan
  of agile methodologies and I have personally seen that best systems start
  small and evolve. The only thing I would add is that with this evolution,
  existing system should be refactored continuously because otherwise they
  become useless.

• Simplify development: Development often has a cycle of: design, write
  failing test, write code, test, make it pass, promoting to configuration
  management and finally releasing the software. Though, often people
  only do unit tests, but many things require integration testing and require
  deploying services. In this regard, statically typed languages such as Java
  can be very slow. It might take 15 minutes to compile and deploy a single
  change. In this regard, I like dynamic languages that make this as painless
  as possible. However, if you are using Java then use hot deployment to
  reduce any startup time.

• Take pride with Quality work: People who take pride
  on their work feel responsible for the success. And their work will show
  that they care about quality whether it is with good testing or better
  quality code. Again, this shows that fundamental for any project is good
  people.

• Deployment support: Services should be simple to deploy and should not
  have to be restarted for simple configuration changes. So, use automatic
  reload of configuration files or database based properties. Also, everything
  needed for deployment should packaged in a single deployment unit (again
  J2EE simplifies it), because relying on external file structure is always
  problematic. Often deployment
  goes through different environments such as Integration, Staging, and
  Production and some of the properties needs to be changed based on
  environment, so provide a simple way of override. You can use simple
  override property files or database for this.

• Debug and Trouble shooting support: Once services are deployed, often
  you have to trouble shoot issues. So allow logging additional information
  based on some configuration such as JMX. In a large distributed system,
  debugging is often a challenge because you have to dig through multiple log
  files on different machine. So it’s generally good idea to define
  a transaction-id for each user request and write tools to search these
  log files easily. See my previous blog for more information.
  You can also provide tools to notify and send you alerts
  when certain error conditions occur.

• Monitoring: Enterprise applications require carefully monitoring. Often
  different matrices are collected needed for support or business folks. So,
  design the application with these monitoring in mind.

• Security: Most application will require security capabilities such
  as permissions, role based security, instance or data based security.

• Standardized Toolset: Most large companies write
  customized development libraries and tools for deployment, build,
  dependency management, monitoring, etc. However, often there is “Not Invented Here” syndrome and countless effort is devoted when similar tools are available
  in open source or commercial space. These companies ignore the learning
  curve each new hire goes through and even the experienced developers don’t
  understand those tools. Again, this goes along with using popular
  technologies so that there isn’t any learning curve required and off the
  shelf solutions are available.

• Introspection: One of the thing that most agile methodologies promote
  is introspection and it’s very necessary that after each release cycle, some
  soul searching is done and an effort is made to find where unneeded time
  is spent. You may use Theory of Constraints to find the biggest time waster
  and iteratively eliminate the waste.

• Conclusion: Above rules and guidelines are lesson learned from my
  my 15 years of experience. On a final note, I should indicate that not all
  of these guidelines indicate my ideal workspace or preferences. For example,
  I like dynamic languages despite their slow performance. In fact, ideally I
  would like to work with no more than five people and with good people you can
  solve any problem that may require 100+ people. It all depends on narrowing
  scope to essential requirements based on 20/80 rule. However, in big
  organizations, you have to find a solution within the constraints of
  organization environment, which may not be optimal but is best suited for
  that organization.

Enums and Lookups in Ruby and Java
Though, Java inherited many of its syntax and features from C++, but it
shunned enums until 1.5 release. Similarly, there is no support of enums in
Ruby, though there is
RCR 178 to add this feature. Another need most software need
is lookups such as list of states, airports, range of ages, etc. Neither
language has built-in support for lookups. Though, both languages have
very good collection support and simple applications can use Map (Java)
or Hash (Ruby). Though, such arrangement can work for small teams where
code is shared by relatively few developers, but when such shared definition
of constants and lookups is shared by hundreds of programmers, it can be
hard to maintain. Thus many large organizations use database or xml files
to store files. Such arrangement in Java can result in cumbersome synax
and maintenance. For example, though we use Java 1.5 at work, but we
have our own way of defining enums similar to how Java’s enum class works,
i.e.,

• Define an inteface that returns name and description.
• For each enum type, define a class that implements this interface.
• Store values of enum in an xml file.
• The runtime environment loads these classes and xml files and provides
  lookup and caching.

This can be a lot of work for programmers, who are defining these enums or
using it. Part of the problem is Java is not dynamic and does not has
features like to change types at runtime. For example, here is a small
example, how Ruby can provide type safe enums or lookups:

Define method_missing in Hash

The method_missing is called when method does not exist. In this case,
the key name is used as a method name, and this method will return hash
value.

1 class Hash
2   def method_missing(key, *args)
3     self[key]
4   end
5 end
6 

Define Storage mechanism for enums

Here I am using simply Hash to store enums and lookup data, though database
can be used as well (with some operator overloading to provide hash like
syntax)

 1 
 2  #
 3  ###
 4  #
 5  module LookupMemoryStorage
 6    @@ChargeTypeCode = {
 7         :ChargeBack             => :CB,
 8         :ServiceFee             => :SR,
 9         :ExchangeFee            => :EX,
10         :HighTouchFee           => :HT,
11         :ExecutiveHighTouchFee  => :ET,
12         :CarServiceFee          => :CF,
13         :HotelServiceFee        => :HF,
14         :TelesalesServiceFee    => :TS,
15         :ShippingFee            => :SH,
16         :BookingFee             => :BK,
17         :ExchangeAddCollectFee  => :XA,
18         :CustomerRefund         => :CR,
19         :MerchantHotel          => :HM,
20         :MerchantCar            => :CM,
21         :MechantInsurance       => :IN,
22         :MerchantAir            => :MA,
23    }
24 
25    @@ProductTypeCode = {
26         :AIR                    => :AIR,
27         :HOTEL                  => :HOT,
28         :CAR                    => :CAR,
29    }
30 
31    @@FeeTypeCode = {
32         :AirlinePaperTicketFee  => :APF,
33         :ServiceFee             => :SV,
34    }
35 
36    @@PostAuthTypeCode = {
37         :FAILED                 => :F,
38         :SUCCESS                => :S,
39    }
40 
41    @@storage = {
42      : ChargeTypeCode           =>      @@ChargeTypeCode,
43      : ProductTypeCode          =>      @@ProductTypeCode,
44      : FeeTypeCode              =>      @@FeeTypeCode,
45      : PostAuthTypeCode         =>      @@PostAuthTypeCode,
46    }
47  end

Lookup Class

Lookup class is helper class for looking up enums and key/value pair, e.g.

 1  class Lookup
 2    include LookupMemoryStorage 
 3 
 4    def self.method_missing(sym, *args)
 5      hash = @@storage[sym]
 6      raise "Could not find category #{sym}" unless hash
 7      hash
 8    end
 9  end
10 

Lookup Class

Lookup class is helper class for looking up enums and key/value pair, e.g.

Client Code

Finally, following code shows how client will use Lookup class. The method
after Lookup is used as a type of lookup and method after type is used as
name of lookup key.

1 puts Lookup.ChargeTypeCode.PreAuth
2 puts Lookup.ChargeTypeCode.InvalidPreAuth
3 puts Lookup.ProductTypeCode.AIR
4 puts Lookup.ProductTypeCode.HOTEL
5 

In this case, when type is not found an exception is raised, howevery when
key is not found it returns nil.

#!/usr/bin/env ruby -w

require ‘rubygems’
require ‘stomp’

#class Publisher < ActiveMessaging::Processor # publishes_to :ServerSideQ # #end #pub = Publisher.new #pub.publish :message => “Go Sox!”

client = Stomp::Client.open nil, nil, “localhost”, 61613
client.subscribe(“/queue/clientSideReplyQ”, {
“persistent” => true,
“client-id” => “rubyClient”,
}) do |message|
puts “Got Reply: #{message.body} on #{message.headers[‘destination’]}”
client.ack message
end

for i in 1..5 do
m = “Go Sox #{i}!”
puts m
client.send(“/queue/ServerSideQ”, m, {
“persistent” => true,
“priority” => 4,
“reply-to” => “/queue/clientSideReplyQ”,
}
)
end
puts “Waiting for response on /queue/clientSideReplyQ”
gets
client.close #disconnect

Message-Driven Pojos with ActiveMQ in Java
ActiveMQ is a popular open source messaging middleware, that obviates the need
for application server just for the sake of messaging. It offers Message
Driven Pojos, which is alternative to Message Driven Beans. You can still use JMS APIs and even transactions through Jencks (lightweight JCA container).

Here is what you need:

Download Java SE 1.5

Download Java SE 1.5 from http://java.sun.com/javase/downloads/index.jsp (Though, this would work with Java SE 1.4, but 1.5 has very good JMX support that helps debugging as we will see later).

Set JAVA_HOME environment variable that points to the installation directory for Java SE.

Download and install ActiveMQ 4.0.1

set ACTIVEMQ_HOME environment variable to point to installation directory of ActiveMQ.

Copy ActiveMQ JCA (rar) file to JBoss’s deploy directory, e.g.

cp $ACTIVEMQ_HOME/lib/optional/optional/activemq-ra-4.0-RC2.rar $JBOSS_HOME/server/all/deploy

Start Active MQ Server

cd $ACTIVEMQ_HOME/bin
sh activemq

Download Jencks from jencks.org

Download Spring 2.0 from springframework.org

Download Spring-XBean 2.5 from springframework.org

Define Message Driven Pojo, in src/mdp/MessageReceiver.java e.g.

 1
 2  package mdp;
 3  import org.apache.log4j.Logger;
 4  import java.io.Serializable;
 5  import javax.jms.Message;
 6  import javax.jms.ObjectMessage;
 7  import javax.jms.JMSException;
 8  import javax.jms.ExceptionListener;
 9  import javax.jms.MessageListener;
10  import javax.jms.Session;
11  import javax.jms.Queue;
12  import javax.jms.QueueConnection;
13  import javax.jms.QueueConnectionFactory;
14  import javax.jms.QueueSender;
15  import javax.jms.QueueSession;
16  import org.springframework.jms.support.converter.SimpleMessageConverter;
17
18  import org.springframework.beans.BeansException;
19  import org.springframework.beans.factory.NoSuchBeanDefinitionException;
20  import org.springframework.context.ApplicationContext;
21  import org.springframework.context.ApplicationContextAware;
22  import org.springframework.context.support.ClassPathXmlApplicationContext;
23
24  public class MessageReceiver implements MessageListener, ApplicationContextAware
25   {
26      final Logger log = Logger.getLogger(getClass());
27
28      private ApplicationContext applicationContext;
29      private QueueConnectionFactory connectionFactory;
30      private ExceptionListener exceptionListener;
31
32      private SimpleMessageConverter converter = new SimpleMessageConverter();
33
34      public final void onMessage(Message jmsMessage){
35          try {
36              ObjectMessage objectMessage = (ObjectMessage) jmsMessage;
37              Serializable response = handleMessage(objectMessage.getObject());
38              reply((Queue) jmsMessage.getJMSReplyTo(), response);
39          } catch (JMSException jmsException){
40              log.error("Error handling " + jmsMessage, jmsException);
41              if (exceptionListener != null) exceptionListener.onException(jmsException);
42          }
43      }
44
45      public void setApplicationContext(ApplicationContext applicationContext) throws Beans
46  Exception {
47          this.applicationContext = applicationContext;
48          this.connectionFactory = (QueueConnectionFactory) applicationContext.getBean("jms
49  Factory");
50      }
51
52      private void reply(Queue queue, Serializable data)
53          throws JMSException {
54          if (queue == null) {
55              log.warn(" No reply queue specified for data " + data);
56              return;
57          }
58          QueueConnection connection = connectionFactory.createQueueConnection();
59          QueueSession session = connection.createQueueSession(false, Session.AUTO_ACKNOWLE
60  DGE);
61
62          QueueSender sender = session.createSender(queue);
63
64          connection.start();
65          ObjectMessage m = null;
66          try {
67              m = session.createObjectMessage(data);
68              sender.send(m);
69              log.info(" replying -------- -> Message: " + data);
70          } finally {
71              connection.close();
72          }
73     }
74
75      public void setExceptionListener(ExceptionListener el) {
76          this.exceptionListener = el;
77      }
78
79      protected Serializable handleMessage(Serializable message) {
80          Integer n = (Integer) message;
81          return new Integer(n.intValue() * n.intValue());
82      }
83  }

Define Message Sender in src/client/MessageSender.java, e.g.

 1  package client;
 2  import org.apache.log4j.Logger;
 3  import javax.jms.Session;
 4  import java.io.Serializable;
 5 
 6  import javax.jms.JMSException;
 7  import javax.jms.QueueReceiver;
 8  import javax.jms.Session;
 9  import javax.jms.ObjectMessage;
10  import javax.jms.Queue;
11  import javax.jms.QueueConnection;
12  import javax.jms.QueueConnectionFactory;
13  import javax.jms.QueueSender;
14  import javax.jms.QueueSession;
15 
16  public class MessageSenderImpl implements MessageSender {
17      static final Logger log = Logger.getLogger(MessageSender.class);
18 
19      private long timeout;
20      private QueueConnectionFactory connectionFactory;
21 
22      public MessageSenderImpl(QueueConnectionFactory connectionFactory, long timeout) {
23          this.connectionFactory = connectionFactory;
24          this.timeout = timeout;
25      }
26 
27      public Serializable sendReceive(
28                  String queueName,
29                  Serializable data) throws JMSException {
30          QueueConnection connection = connectionFactory.createQueueConnection();
31          QueueSession session = connection.createQueueSession(false, Session.AUTO_ACKNOWLE
32  DGE);
33          Queue queue = session.createQueue(queueName);
34          QueueSender sender = session.createSender(queue);
35 
36          Queue tempQ = session.createTemporaryQueue();
37          connection.start();
38          ObjectMessage m = null;
39          try {
40              m = session.createObjectMessage(data);
41              m.setJMSReplyTo(tempQ);
42              sender.send(m);
43              m = null;
44              //session.commit();
45              log.info(" sending -------- -> Message: " + data);
46              QueueReceiver receiver = session.createReceiver(tempQ);
47 
48              m = (ObjectMessage) receiver.receive(timeout);
49 
50          } finally {
51              connection.close();
52          }
53          if (m == null) {
54              throw new JMSException("Failed to receive response from " + queueName + " within " + timeout + " millis for request " + data);
55              //throw new TimeoutException("Failed to receive response from " + queueName + " within " + timeout + " millis for request " + data);
56          }
57          log.info(" received -------- -> Message: " + m.getObject());
58          return m.getObject();
59     }
60 
61      public void setTimeout(long timeout) {
62          this.timeout = timeout;
63      }
64  }

Define application-context.xml as follows:

 <?xml version="1.0" encoding="UTF-8"?>

 <!-- START SNIPPET: spring -->
 <!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
 <beans>

   <!--
   || ActiveMQ Broker
     <bean id="broker" class="org.apache.activemq.xbean.BrokerFactoryBean">
       <property name="config" value="classpath:tbs-activemq.xml" />
       <property name="start" value="true" />
     </bean>
   -->

     <bean id="brokerURL" class="java.lang.String">
         <constructor-arg>
             <value>tcp://localhost:61616</value>
         </constructor-arg>
     </bean>

     <bean id="activeMQContainer" class="org.jencks.JCAContainer" singleton="true">
         <property name="bootstrapContext">
                 <bean class="org.jencks.factory.BootstrapContextFactoryBean">
                         <property name="threadPoolSize" value="25" />
                 </bean>
          </property>
         <!-- the JCA Resource Adapter -->
         <property name="resourceAdapter">
             <bean id="activeMQResourceAdapter"
                   class="org.apache.activemq.ra.ActiveMQResourceAdapter">
                 <property name="serverUrl" ref="brokerURL"/>
             </bean>
         </property>
     </bean>

     <!-- plain connection factory -->
     <bean id="jmsFactory"  class="org.apache.activemq.ActiveMQConnectionFactory">
         <property name="brokerURL" ref="brokerURL"/>
     </bean>

     <bean id="MessageSender" class="com.orbitz.tbs.host.txn.messaging.MessageSenderImpl">
         <constructor-arg>
             <ref bean="jmsFactory"/>
         </constructor-arg>
         <constructor-arg>
             <value>30000</value>
         </constructor-arg>
     </bean>

   <!--
     || an inbound message connector using a stateless, thread safe MessageListener
     -->
   <!-- START SNIPPET: inbound -->
   <bean id="inboundConnectorA" class="org.jencks.JCAConnector">

     <property name="jcaContainer" ref="activeMQContainer" />

     <!-- subscription details -->
     <property name="activationSpec">
       <bean class="org.apache.activemq.ra.ActiveMQActivationSpec">
         <property name="destination" value="testQ"/>
         <property name="destinationType" value="javax.jms.Queue"/>
       </bean>
     </property>
     <property name="ref" value="MessageReceiver"/>
   </bean>
   <!-- END SNIPPET: inbound -->

   <bean id="MessageReceiver" class="mdp.MessageReceiver">
   </bean>

Write Unit test/MessageSenderTest.java as follows:

 1  package com.orbitz.tbs.host.txn.messaging;
 2 
 3  import junit.framework.TestCase;
 4  import com.orbitz.tbs.host.txn.messaging.MessageSender;
 5  import org.springframework.context.support.ClassPathXmlApplicationContext;
 6 
 7  public class MessageSenderTest extends TestCase {
 8      protected static ClassPathXmlApplicationContext appContext = new ClassPathXmlApplicationContext(new String[]{"TBSMessagingOrbitzBeans.xml"});
 9      MessageSender sender;
10 
11    public MessageSenderTest(String name) {
12      super(name);
13    }
14 
15    protected void setUp() throws Exception {
16      super.setUp();
17      sender = (MessageSender) appContext.getBean("MessageSender", MessageSender.class);
18    }
19 
20    protected void tearDown() throws Exception {
21      sender = null;
22      super.tearDown();
23    }
24 
25    public void testSendReceive() throws Exception {
26      for (int i=0; i<10; i++) {
27        Integer data = (Integer) sender.sendReceive("CancellationQ", new Integer(i));
28        assertEquals("Data didn't match", new Integer(i*i), data);
29      }
30    }
31 
32    public static void main(String[] args) {
33      junit.textui.TestRunner.run(MessageSenderTest.class);
34    }
35  }

Compile your code as follows:

javac -d build -classpath $ACTIVEMQ_HOME/lib/optional/spring-1.2.4.jar:\
$ACTIVEMQ_HOME/lib/optional/activemq-optional-4.0.jar:$ACTIVEMQ_HOME/lib/activeio-core-3.0.jar:\
$ACTIVEMQ_HOME/lib/activemq-core-4.0.jar:$ACTIVEMQ_HOME/lib/activemq-console-4.0.jar src/*/*java

Start activemq server:

cd $ACTIVEMQ_HOME/bin
./activemq

Run Test:

java -classpath build:$ACTIVEMQ_HOME/lib/optional/spring-1.2.4.jar:\
$ACTIVEMQ_HOME/lib/optional/activemq-optional-4.0.jar:$ACTIVEMQ_HOME/lib/activeio-core-3.0.jar:\
$ACTIVEMQ_HOME/lib/activemq-core-4.0.jar:$ACTIVEMQ_HOME/lib/activemq-console-4.0.jar client.MessageSender

Start jConsole (require Java 1.5) to verify messages using JMX

.
Select Queue and then see # of messages

Voila

Finally, you can also run embedded ActiveMQ, though it is commented in the spring configuration file.

Integrating Ruby with Java using ActiveMQ and Stomp
Though, Ruby is a great dynamic language, and it is slowly gaining on Java
and .Net, but often folks from large corporations snub at Ruby for lack of
enterprise features. This criticism is somewhat valid as Ruby and Rails have to play
nice with existing legacy applications. Here, I am showing how Ruby and
Java can be integrated using ActiveMQ messaging middleware. ActiveMQ is a very
popular open source messaging middleware and is highly scalable, robust,
clusterable and peformant piece of software. In an ideal world, we can
use Ruby/Rails, where its strength lies, i.e., web development and
Java can be used on the serverside, where its strength lies.

Here is the recipe for integrating Java with Ruby:

Install Java SE 1.5 and Java EE 1.4

• Download Java SE 1.5 or above from http://java.sun.com/javase/downloads/index.jsp (Though, this would work with Java SE 1.4, but 1.5 has very good JMX support that helps debugging as we will see later).
• Set JAVA_HOME environment variable that points to the installation directory for Java SE.
• Download Java EE 1.4 SDK from http://java.sun.com/javaee/downloads/index.jsp.
• Set J2EE_HOME environment variable that points to the installation directory for Java EE.

Install ActiveMQ 4.0.1

• Download ActiveMQ 4.0.1 from http://www.activemq.org/site/download.html.
• Set ACTIVEMQ_HOME environment variable that points to the installation directory for ActiveMQ.

Install Ruby 1.8 and Stomp 1.0.1

• Install Ruby using One-Click from http://rubyforge.org/projects/rubyinstaller/ or use other distribution for your platform.
• Install Stomp Gem by typing “gem install stomp” or download it from http://stomp.codehaus.org/.

Start ActiveMQ

• If you are using older version of ActiveMQ, you may need to edit conf/activemq.xml and add

       <transportConnectors>
         <transportConnector uri="tcp://localhost:61616"/>
         <transportConnector uri="stomp://localhost:61613"/>
       </transportConnectors>

  However, if you are using 4.0.1 then it comes with built-in support for
  stomp and you don’t need to do anything.

• cd to ActiveMQ’s bin directory and type activemq or activemq.bat depending on your operating system.

Debugging ActiveMQ through JMX

• start jconsole from Java 1.5’s bin directory.
• It will show activemq’s process, select it as shown below
• We will get back to jConsole later.

Create a Ruby Client (ruby-stomp.rb) that sends a message and then receives one

 1 #!/usr/bin/env ruby -w

 2 
 3 require 'rubygems'
 4 require 'stomp'
 5 

 6 #class Publisher < ActiveMessaging::Processor
 7 # publishes_to :ServerSideQ

 8 #
 9 #end
10 #pub = Publisher.new

11 #pub.publish :message => "Go Sox!"
12 
13 client = Stomp::Client.open nil, nil, "localhost", 61613

14 client.subscribe("/queue/clientSideReplyQ", {
15   "persistent" => true,
16   "client-id" => "rubyClient",

17 }) do |message|
18   puts "Got Reply: #{message.body} on #{message.headers['destination']}"

19   client.ack message
20 end
21 
22 
23 for i in 1..5 do

24   m = "Go Sox #{i}!"
25   puts m
26   client.send("/queue/ServerSideQ", m, {
27     "persistent" => true,

28     "priority" => 4,
29     "reply-to" => "/queue/clientSideReplyQ",
30   }

31   )
32 end
33 puts "Waiting for response on /queue/clientSideReplyQ"
34 gets
35 client.close 

Create Java Server that listens and then responds

On the server-side, ActiveMQ will use BytesMessage if “content-length”
header is present, otherwise it uses TextMessage. Ruby stomp client sets
this parameter, so on the server side it will be BytesMessage.

 1 import java.io.PrintStream;

 2 import javax.jms.BytesMessage;
 3 import javax.jms.Connection;
 4 import javax.jms.Destination;

 5 import javax.jms.ExceptionListener;
 6 import javax.jms.JMSException;
 7 import javax.jms.Message;

 8 import javax.jms.MessageConsumer;
 9 import javax.jms.MessageEOFException;
10 import javax.jms.MessageListener;

11 import javax.jms.Queue;
12 import javax.jms.QueueConnection;
13 import javax.jms.QueueSender;
14 import javax.jms.QueueSession;

15 import javax.jms.Session;
16 import javax.jms.TextMessage;
17 import org.apache.activemq.ActiveMQConnection;
18 import org.apache.activemq.ActiveMQConnectionFactory;

19 
20 public class Server implements MessageListener, ExceptionListener {
21     Session session;
22     public Server() throws Exception {

23         // I found it simple to do everything here, but production code will
24         // need to be cleaned up and resources management needs to be better
25         // handled.
26         ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(

27                 ActiveMQConnection.DEFAULT_USER,
28                 ActiveMQConnection.DEFAULT_PASSWORD,
29                 ActiveMQConnection.DEFAULT_BROKER_URL);
30         Connection connection = connectionFactory.createQueueConnection();
31         connection.setClientID("clientID");

32         connection.setExceptionListener(this);
33         connection.start();
34         session = connection.createSession(true, Session.AUTO_ACKNOWLEDGE);
35         Destination destination = session.createQueue("ServerSideQ");

36         MessageConsumer consumer = session.createConsumer(destination);
37         consumer.setMessageListener(this);
38     }
39     
40     // I am not closing all resources, which should change for production code.

41     private void close() throws Exception {
42         if (session != null) session.close();

43         //if (connection != null) connection.close();
44     }
45     
46     public static void main(String[] args) throws Exception {

47         new Server();
48         Thread.currentThread().join();
49     }
50     
51     public void onMessage(Message message) {

52         try {
53             if (message instanceof BytesMessage) {
54                 BytesMessage bytMsg = (BytesMessage) message;
55                 StringBuffer msg = new StringBuffer();

56                 int c;
57                 try {
58                     while ((c=bytMsg.readByte()) != -1) {

59                         msg.append((char) c);
60                     }
61                 } catch (javax.jms.MessageEOFException e) {}
62                 System.out.println("Received: " + msg);

63             } else {
64                 System.out.println("Unknown Message: " + message);
65             }
66             session.commit();

67             reply(message);
68         } catch (Exception e) {
69             e.printStackTrace();
70         }
71     }

72     
73     synchronized public void onException(JMSException ex) {
74         System.out.println("JMS Exception occured. " + ex);

75     }
76     
77     public void reply(Message msg) throws Exception {
78         ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(

79                 ActiveMQConnection.DEFAULT_USER,
80                 ActiveMQConnection.DEFAULT_PASSWORD,
81                 ActiveMQConnection.DEFAULT_BROKER_URL);
82         QueueConnection replyConnection = connectionFactory.createQueueConnection();
83         replyConnection.setExceptionListener(this);

84         replyConnection.start();
85         QueueSession replySession = (QueueSession) replyConnection.createQueueSession(
86                 true, Session.AUTO_ACKNOWLEDGE);
87         Queue queue = (Queue) msg.getJMSReplyTo();
88         QueueSender sender = replySession.createSender(queue);
89         BytesMessage responseMessage = replySession.createBytesMessage();

90         responseMessage.writeUTF("Response to " + msg);
91         responseMessage.setStringProperty("KEY", "key");
92         sender.send(responseMessage);
93         replySession.commit();

94         sender.close();
95         replyConnection.close();
96     }
97 }

Compile Server.java

When compiling Java file, make sure you include following jar files in your CLASSPATH:

• activemq-core-4.0.1.jar
• j2ee.jar
• incubator-activemq-4.0.1.jar

For example,

 javac -classpath lib/activemq-core-4.0.1.jar;lib/j2ee.jar; -d classes/ Server.java

Start Server

java -classpath lib/incubator-activemq-4.0.1.jar;lib/activemq-core-4.0.1.jar;lib/j2ee.jar;classes/ Server

Run Ruby client

ruby ruby-stomp.rb

Verify jConsole

You should see two queues in jConsole, i.e., “/queue/ServerSideQ”, and
“/queue/clientSideReplyQ”. Look at the # of enqueues and dequeues count.

Voila

Note that we are using persistent queues and subscribers by setting persistent and client-id headers. Refer to ActiveMQ and Stomp for more information on these headers.

Caveat Emptor: I found subscribe functionality of stomp gem a bit shaky and it often missed publications.

Finally, Rails is planning to add support for ActiveMessaging in future, which
is based on Stomp and will make this a bit easier. Following is how it will
look in rails:

 class MyProcessor < ActiveMessaging:: Processor
   subscribes_to :clientSideReplyQ
   def on_message(message)
     puts "received: " + message
   end
 end

Download ruby-stomp.rb and Server.java.

import java.io.PrintStream;
import javax.jms.BytesMessage;
import javax.jms.Connection;
import javax.jms.Destination;
import javax.jms.ExceptionListener;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageEOFException;
import javax.jms.MessageListener;
import javax.jms.Queue;
import javax.jms.QueueConnection;
import javax.jms.QueueSender;
import javax.jms.QueueSession;
import javax.jms.Session;
import javax.jms.TextMessage;
import org.apache.activemq.ActiveMQConnection;
import org.apache.activemq.ActiveMQConnectionFactory;

public class Server implements MessageListener, ExceptionListener {
Session session;
public Server() throws Exception {
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(ActiveMQConnection.DEFAULT_USER, ActiveMQConnection.DEFAULT_PASSWORD, ActiveMQConnection.DEFAULT_BROKER_URL);
Connection connection = connectionFactory.createQueueConnection();
connection.setClientID(“clientID”);
connection.setExceptionListener(this);
connection.start();
session = connection.createSession(true, Session.AUTO_ACKNOWLEDGE);
Destination destination = session.createQueue(“ServerSideQ”);
MessageConsumer consumer = session.createConsumer(destination);
consumer.setMessageListener(this);
//if (session != null) session.close();
//if (connection != null) connection.close();
}

public static void main(String[] args) throws Exception {
new Server();
Thread.currentThread().join();
}

public void onMessage(Message message) {
try {
if (message instanceof TextMessage) {
TextMessage txtMsg = (TextMessage) message;
String msg = txtMsg.getText();
System.out.println(“Received: ” + msg);
} else if (message instanceof BytesMessage) {
BytesMessage bytMsg = (BytesMessage) message;
StringBuffer msg = new StringBuffer();
int c;
try {
while ((c=bytMsg.readByte()) != -1) {
msg.append((char) c);
}
} catch (javax.jms.MessageEOFException e) {}
System.out.println(“Received: ” + msg);
}
else {
System.out.println(“Unknown Type ” + message.getClass().getName() + ” Got: ” + message);
}
session.commit();
reply(message);
/*
if (++count % dumpCount == 0) {
dumpStats(connection);
}
*/
}
catch (Exception e) {
System.out.println(“Caught: ” + e);
e.printStackTrace();
}
}

synchronized public void onException(JMSException ex) {
System.out.println(“JMS Exception occured. Shutting down client.”);
}

public void reply(Message msg) throws Exception {
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(ActiveMQConnection.DEFAULT_USER, ActiveMQConnection.DEFAULT_PASSWORD, ActiveMQConnection.DEFAULT_BROKER_URL);
QueueConnection qconnection = connectionFactory.createQueueConnection();
qconnection.setExceptionListener(this);
qconnection.start();
QueueSession qsession = (QueueSession) qconnection.createQueueSession(true, Session.AUTO_ACKNOWLEDGE);
String subject = msg.getJMSReplyTo().toString();
subject = subject.substring(subject.lastIndexOf(‘/’)+1);
//Queue queue = (Queue) session.createQueue(subject);
Queue queue = (Queue) msg.getJMSReplyTo();
QueueSender sender = qsession.createSender(queue);
TextMessage responseMessage = qsession.createTextMessage(“hello back”);
//BytesMessage responseMessage = qsession.createBytesMessage();
//responseMessage.writeUTF(“Response to ” + msg);
responseMessage.setStringProperty(“KEY”, “key”);
sender.send(responseMessage);
qsession.commit();
sender.close();
qconnection.close();
System.out.println(“—Sent ” + responseMessage.toString().substring(0, 20) + ” to ” + subject + “—” + queue);
}
}

RailsConf 2006 Day 3
I started today with Beyond DHTML: Introducing Laszlo on Rails by Mike Pence. It was interesting talk, but speaker could not give proper demo due to Internet connection. Overall, I am not convinced why I would need Lazlo, if I can use Ajax especially it is more resource intensive. I wish, I had attended Selinium track because I caught last few minutes and it seemed pretty useful.

Second session was Just the Facts (and Dimensions) — using Rails with your OLAP data model by Ken Kunz. It was slighly useful talk, but I didn’t learn a whole a lot.

Third session was Using Ruby on Rails to Succeed in Selling Music in the 21st Century by Benjamin Curtis. It was by far the best session of the conference and I got a lot of practical information on eCommerce. Though, I have been working for a very large eCommerce, but you don’t always get to see all areas. He also has a book called “Money Train”, though I saw it only had 18 pages. This is one area that I have not found a lot of information in books and tutorials so it might be worth buying it.

Then I attended Deploying Rails Applications by  James Duncan Davidson. Clearly, deployment has been more complicated piece of Rails development and there aren’t any clean solutions available. So, there was good discussion of using Apache/FastCGI, Lighttpd+fcgi. He recommended Capistrano and start small. He also suggested RSS/Campfire to keep track of deployment notifications. He also cautioned with log files, database sockets (firewall), file permissions (public directory). The other solution as few other people mentioned is Mongrel. It also reminded me of proxy servers and reverse proxy servers I used in a number of projects and helped setup in some consulting gigs (and a number of ugly issues along with it [keep-alive] ). I was surprised he ended the talk with using “Container” to tackle these deployment issues. Does this mean that Rails will turn into J2EE?

Final thoughts

This conference was not very well organized and simple things like lack of slides or Internet were bothersome. Also, there were no birds of a feather sessions. I found most of the speakers were not experienced and many of top level leaders of Rails such as DHH, Dave Thomas, Chad Fowler, David Black didn’t have any sessions. The only exceptions were Mike Clark, James Duncan Davidson and Justin Getland and except Justin, the talks of Mike Clark and JDD were very high level (sort of like keynotes). I found many of those sessions were light on content as a number of them had much higher agendas in their talk summaries, but speakers could not finish the APIs, demo, research or work needed. So, overall I found the conference useful, but it didn’t meet expectations. It was sort of brown bags, but didn’t have high caliber speakers like other conference. Also, I am not sure why they didn’t bother inviting Bruce Tate or Jim Weirich.

Since boom of Ruby/Rails, I find that biggest beneficiaries of all this revolution have been either 37signals, Dave Thomas (et al), Thoughtworks and a number of other startups, book publishers and trainers. I have bought almost all of Ruby/Rails books of Pragmatic series (along with other) and in this conference I heard a number of upcoming books from their series (and yes I will buy their books as well). Also, Dave Thomas, Mike Clark and Chad Fowler have pretty good training program going where they charge about $1800 for 3-day class (but they didn’t offer any short sessions from their training program). So, clearly there is a lot of money in it for all these people, but, I hope some of this windfall goes to Yukihiro Matz Matsumoto, the guy behind all this.

Another observation, I found that 90% of people at conference used Mac and I was one of very few people who didn’t have Mac and received a non-conformity certificate:

You can visit flicker for this picture as well and other pictures are available at pictures from Rails conference.

RailsConf 2006 Day 2
I started today with Ajax on Rails by Justin Gehtland. It was excellent talk, though most of it was nothing new. But Justin
gave demo of his upcoming project Streamlined that has scaffold to create views and has builtin
filter and allows managing relationships. It’s going to be very exciting
framework.

I then sat on Lessons from Blinksale and IconBuffet by Scott Raymond. Scott has worked as Rails contractor and he started
with a bit history of Vitruvius and described three tenants of architecture, i.e., firmitas (firmness), utilitas (usefulness) and venustas (beauty/delight). He presented a
number of interesting lessons such as starting project with templates,
mocking templates and then incrementally adding functionality with
very closed loop. He also emphasized testing markups before adding any
code.

Third session, I saw was Putting the BBC’s Programme Catalogue on Rails by Matt Biddulph. That talk was not as good
as I expected, I was hoping to get more practical information on
managing large database, but most of the talk covered history of BBC.
Also, I was shocked that he used ActiveRecord to migrate 2Gig data from
old database from legacy database to mysql. Clearly, an SQL script is
more suitable for such (I guess I would have used stored procedure,
but I am not sure which version of MySQL he used). He admitted, 99%
of database load was spent in creating ActiveRecords.

Then I attended an excellent talk on Rails Deployment by Ezra Zygmuntowicz. He showed a lot of practical tips on deploying
applications such as Apache/FastCGI, Lighttpd, Mongrel. A lot of tips
were from his article Rails Stack. He also plugged his upcoming Pragmatic book on Rails deployment.
Though, his talk summary included BackgrounDRb, but he didn’t talk
about it. Anyway, I look forward to his book.

I then attended talk on Testing Migrations by Glenn Vanderburg. There was a lot of good information on testing
migrations, though a lot of his approaches are still half-baked.

My last talk was by Mike Clark on
Testing Rails Apps
. I didn’t get much new from his talk, it pretty much boiled to:
Rails/Rake has excellent support for testing such as unit, integration,
functional, features and mocks. I don’t understand, why people still
inviting people to use TDD in conferences. By now, most develoeprs
understand benefits of TDD. I would be more interested in new support of
TDD in Rails or tackling difficults issues of testing. For example, I found a couple of articles of
Bruce Tate on Functiona/Integration testing much more useful, but Mike Clark’s
session was very high level sort of like keynote.

Finally, I could not connect to Internet for whole day. I don’t understand,
why RailsConf/Wyndham hotel has such poor Wifi support. My laptop connected to Wifi,
but never could get IP address from DHCP server. And one more thing,
I was forced to pay $50.0 for slides/audio sessions. I wish that was
available freely as we already paid for the conference.

RailsConf 2006 Day 1
I finally was able to find tickets for RailsConf 2006 in Chicago. Though,
I could not find tickets back in Feb., but I managed to find someone who
could not make it. Anyway, I missed most of sessions due to work, but I
was able to checkout “Rails Application Optimization – Techniques and Tools” by Stefan Kaes.
It was excellent talk on profiling Rails applications and improving
performance. He also posted most of tips at A Look at Common Performance Problems in Rails about 1-2 weeks ago. Though, I am disappointed most of tips encourage
bypassing rails framework. I think we need to tackle these problems
at language and framework level and not go around and write ugly hacks.
This also shows that Rails is not really ready for real businesses despite
such claims by rails community.

Second session I attended was Rails Deployment on Shared Hosts by Geoffrey Grosenbach. It was big disappointed, if he only had said “Don’t use shared hosts”, that would have saved me an hour.

I also enjoyed Martin Fowler’s keynote. He admitted, he has never
used Rails, but he is an avid proponent of Ruby. He basically ravished
design principles of Rails, i.e., opinionated software, simplicity,
quick and clean setup (which encourages agile dev. — tracer bullet — fire/aim/fire/aim). He talked about postmodern programming which was interesting. It just means that world isn’t perfect so languages or frameworks that
try to think nothing exist outside their world fails. Clearly, Ruby has
clean design and has a lot of influence from Lisp/Smalltalk, but it also
takes tips from Perl and Python by integrating with other languages and
external environment.

Finally, I found this conference not very well organized. For example,
for popular sessions, there weren’t enough seats and people had to stand
in the doorways. Also, there are no slides handout or conference proceeding.
Worst, organizers are going to charge $50.0 for session slides and audio.
At very least, slides should have made freely available.