Shahzad Bhatti Welcome to my ramblings and rants!

December 29, 2008

Leaving the industry (IT)

Filed under: Computing — admin @ 1:14 pm

I read an interesting reply by Joel Spolsky to a post on Thinking of leaving the industry (IT) by someone who is thinking about leaving software industry and going for a business degree. Joel reply can be summed up to following points:

  • great programmers are not effected by the economic downturn.
  • pay is great with starting salary of $75,000 and most earn six figures salary.
  • programmers are treated better than people from other occupation.
  • most programmers would do programming even if they don’t get paid.

I have been programming for over twenty one years and love programming. For long time, the way I distinguished between great programmers and average programmers was passion and one way to see the passion is to ask them of their side projects. Though, I agree with first and last point and second point is a stretch, but I disagree with the third point. I have worked at ten or more companies as an employee and consultant and like answer to every programming design question: “it depends,” the treatment of programmers also varies.

I have spoken on this topic earlier [ 1 , 2 , 3]. I have found most IT places to be sweatshops, for example, in most companies Taylorism rules and programmers are treated like another dumb workers and managers are responsible for whipping them to get the work done. In fact, death march projects are norm in most places, where managers think setting impossible deadline will motivate people to put a lot of hours and finish the project earlier. Given, that there isn’t any overtime compensation in our industry, there is a little problem with such mentality. Such attitude is common in not only average companies but at top companies like Microsoft and Google where you get your so called 20% time for personal projects after you spend 60+ hours on actual project. Though, computers have added convenience to our life with a 7×24 culture, the cost of such culture is being oncall, which can curtail your social life significantly. Finally, offshoring has also effected job market and though most offshore people can’t compete with local great programmers and I will take 4-5 local great programmers over 100 offshore programmers any day, but it provides a very attractive alternative against average programmers.

So, despite higher salary, I think most programmers (great) work much harder than most other professionals and it can be a stressful job when you continuously have to work with impossible deadlines. Though, during dot com boom I saw plenty of store clerks and script kiddies making six figures in IT but they are long gone. These days, you cannot survive IT industry unless you are a top notch. And if you are not then, you’d better off move to management, which is the trend I have seen in most companies. In conclusion, I agree with the last point of Joel and the only way to survive in IT is passion.

November 22, 2008

Implementing Actor-based message passing using Object-oriented methods

Filed under: Computing — admin @ 11:01 am

I wrote about use of actor based programming for parallel and concurrent programming previously [ 1 , 2 ], where an actor keeps its own thread of execution with a mailbox. Actors communicates to other actors by sending messages that are delivered to its mailbox. In this model, the message passing takes literal form rather than method invocation. I studied Actor based parallel computing in my post graduate research on parallel and distributed field. I learned message passing systems such as Actors, MPI, MPL were much more scalable than shared memory. This describes popularity of languages like Erlang and Scala that has adopted Actor model to tackle concurrency (object-oriented nature in Erlang). According to Ralph Johnson concurrent Erlang is object oriented. Alan Kay has also argued that message passing is more important than objects in object-oriented programming and regrets that he chose the term “object-oriented” instead of “message-passing”. Even Joe Armstrong, who generally dislikes object oriented programming admits similarity between object-oriented programming and Erlang’s message passing.

However, despite effectiveness of message passing in new era of concurrency, I find that pure message passing is a bit intimidating to an average programmer. In this blog, I will show how an object-oriented methods can be used to abstract actor-based message passing. The primary benefit of message passing alongside with immutability is its support of concurrency so the following example will demonstrate use of message passing in multiple threads. Let’s suppose, we are designing a simple chat service to send and receive messages and the message looks like following:

package actor;

public class ChatMessage {
    public final String fromUser;
    public final String toUser;
    public final String message;

    public ChatMessage(String fromUser, String toUser, String message) {
        this.fromUser = fromUser;
        this.toUser = toUser;
        this.message = message;
    }
}

Following is an interface for the chat service:

package actor;

import java.util.Collection;

public interface ChatService {
    /**
     * sends a message to another user * @param message - message object
     */
    @Asynchronous
    public void sendMessage(ChatMessage message);

    /**
     * retrieves messages that were sent to the user. * @param user - target user who received the messages * @return - collection of messages
     */
    public Collection<ChatMessage> getMessages(String user);

    /**
     * remove all messages
     */
    @Asynchronous
    public void clear();

    /**
     * return number of messages
     */
    public long countMessages();

    /**
     * @throws IllegalArgumentException
     */
    public void raiseError() throws IllegalArgumentException;
}

Note that above interface uses “@Asynchronous” annotation for “sendMessage” and “clear” methods. One of the benefit of message-passing is that you can optionally send asynchronous messages. So I defined an annotation to declare a particular method as asynchronous, e.g.

package actor;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
public @interface Asynchronous {
}
Following code implements the chat service in non-thread-safe fashion:
package actor;

import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.Map;

/**
 * Simple non-thread-safe implementation of ChatService
 */
public class ChatServiceImpl implements ChatService {
    private long count;
    private Map<String, Collection<ChatMessage>> messagesByUsers;

    public ChatServiceImpl() {
        messagesByUsers = new HashMap<String, Collection<ChatMessage>>();
    }

    /**
     * sends a message to another user * @param message - message object
     */
    public void sendMessage(ChatMessage message) {
        getMessages(message.toUser).add(message);
        count++;
    }

    /**
     * retrieves messages that were sent to the user. * @param user - target user who received the messages * @return - collection of messages
     */
    public Collection<ChatMessage> getMessages(String user) {
        Collection<ChatMessage> messages = messagesByUsers.get(user);
        if (messages == null) {
            messages = new ArrayList<ChatMessage>();
            messagesByUsers.put(user, messages);
        }
        return messages;
    }

    /**
     * remove all messages
     */
    public void clear() {
        messagesByUsers.clear();
    }

    /**
     * return number of messages
     */
    public long countMessages() {
        return count;
    }

    /**
     * @throws IllegalArgumentException
     */
    public void raiseError() throws IllegalArgumentException {
        throw new IllegalArgumentException();
    }
}

Note that above code cannot be used in multi-threading environment. If we were building thread-safe service using Java, we would have to add synchronization or locking such as:

package actor;

import java.util.Collection;

/**
 * Simple thread-safe implementation of ChatService
 */
public class SafeChatServiceImpl implements ChatService {
    private ChatService delegate;

    public SafeChatServiceImpl(ChatService delegate) {
        this.delegate = delegate;
    }

    /**
     * sends a message to another user * @param message - message object
     */
    public synchronized void sendMessage(ChatMessage message) {
        delegate.sendMessage(message);
    }

    /**
     * retrieves messages that were sent to the user. * @param user - target user who received the messages * @return - collection of messages
     */
    public synchronized Collection<ChatMessage> getMessages(String user) {
        return delegate.getMessages(user);
    }

    /**
     * remove all messages
     */
    public synchronized void clear() {
        delegate.clear();
    }

    /**
     * return number of messages
     */
    public synchronized long countMessages() {
        return delegate.countMessages();
    }

    /**
     * @throws IllegalArgumentException
     */
    public synchronized void raiseError() throws IllegalArgumentException {
        delegate.raiseError();
    }
}
Above code demonstrates use of composition to convert a non-thread-safe class into a thread-safe class.

Now the meat of this library is following factory class that converts any Java object into an actor:

package actor;

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;

/**
 * Factory class to convert any POJO class into Actor.
 */
public class ActorFactory {
    private static final int MAIL_BOX_SIZE = 100;

    private static class MethodRequest {
        private final Object object;
        private final Method method;
        private final Object[] args;
        private final BlockingQueue<MethodResponse> replyQueue;

        private MethodRequest(Object object, Method method, Object[] args, BlockingQueue<MethodResponse> replyQueue) {
            this.object = object;
            this.method = method;
            this.args = args;
            this.replyQueue = replyQueue;
        }
    }

    private static class MethodResponse {
        private Object response;
        private Exception exception;

        private MethodResponse(Object response) {
            this.response = response;
        }

        private MethodResponse(Exception exception) {
            this.exception = exception;
        }

        Object getResponse() throws Exception {
            if (exception != null) throw exception;
            return response;
        }
    }

    private static class ActorThread extends Thread {
        private volatile boolean shutdown;
        private BlockingQueue<MethodRequest> queue;

        private ActorThread(BlockingQueue<MethodRequest> queue) {
            this.queue = queue;
            setDaemon(true);
        }

        public void run() {
            while (!shutdown) {
                try {
                    MethodRequest request = queue.take();
                    try {
                        Object response = request.method.invoke(request.object, request.args);
                        if (request.replyQueue != null) {
                            request.replyQueue.put(new MethodResponse(response));
                        }
                    } catch (InvocationTargetException e) {
                        if (request.replyQueue != null) {
                            request.replyQueue.put(new MethodResponse((Exception) e.getCause()));
                        } else {
                            e.printStackTrace();
                        }
                    } catch (Exception e) {
                        if (request.replyQueue != null) {
                            request.replyQueue.put(new MethodResponse(e));
                        } else {
                            e.printStackTrace();
                        }
                    }
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupted();
                }
            }
        }
    }

    public static Object create(final Class ifaceClass, final Object object) {
        final BlockingQueue<MethodRequest> queue = new ArrayBlockingQueue<MethodRequest>(MAIL_BOX_SIZE);
        new ActorThread(queue).start();
        final InvocationHandler handler = new InvocationHandler() {
            public Object invoke(final Object proxy, final Method method, final Object[] args) throws Throwable {
                if (method.isAnnotationPresent(Asynchronous.class)) {
                    queue.put(new MethodRequest(object, method, args, null));
                    return null;
                } else {
                    final BlockingQueue<MethodResponse> replyQueue = new ArrayBlockingQueue<MethodResponse>(1);
                    queue.put(new MethodRequest(object, method, args, replyQueue));
                    return replyQueue.take().getResponse();
                }
            }
        };
        return ifaceClass.cast(Proxy.newProxyInstance(ifaceClass.getClassLoader(), new Class<?>[]{ifaceClass}, handler));
    }

    public static Object createWithoutActor(final Class ifaceClass, final Object object) {
        final InvocationHandler handler = new InvocationHandler() {
            public Object invoke(final Object proxy, final Method method, final Object[] args) throws Throwable {
                Object response = method.invoke(object, args);
                return response;
            }
        };
        return ifaceClass.cast(Proxy.newProxyInstance(ifaceClass.getClassLoader(), new Class<?>[]{ifaceClass}, handler));
    }
}
The ActoryFactory uses reflection and proxy features of Java. The client code creates an actor by invoking
public static Object create(final Class ifaceClass, final Object object) {
  ...
}

ChatService service = (ChatService) ActorFactory.create(ChatService.class, new ChatServiceImpl());
      
The create method instantiates a blocking queue that serves as mailbox for the actor and then starts a thread. The create method then instantiates InvocationHandler that intercepts all methods. Finally, the proxy object is returned to the client. When the client invokes any method it is intercepted by the proxy, which then sends the message to the mailbox of the actor. The proxy also checks if the method i defined as asynchronous then it does not wait for the reply. The thread simply waits until there is a message in the mailbox. The message for the actor is defined in MethodRequest that encapsulates reflection information to invoke a method. The actor thread invokes the method using reflection and puts back reply in the queue supplied by the proxy. The proxy waits for the reply and then sends back the reply back to the client. As an actor serves one request at a time, there is no need for locking or synchronization.

Following code demonstrate client test code:

package actor;

import junit.framework.*;

import java.util.Collection;

public class ActorTest extends TestCase {
    private static ChatService service;
    private static final int NUM_THREADS = 100;
    private static final int NUM_MESSAGES = 100;

    protected void setUp() throws Exception {
        super.setUp();
    }

    public void testSendGetMessages() throws Exception {
        for (int i = 0; i < NUM_MESSAGES; i++) {
            final String suffix = Thread.currentThread().getName() + "--" + i;
            ChatMessage janeToAdamMessage = new ChatMessage("jane" + suffix, "adam" + suffix, "hi there" + suffix);
            service.sendMessage(janeToAdamMessage);
            ChatMessage adamToJaneMessage = new ChatMessage("adam" + suffix, "jane" + suffix, "hi there too" + suffix);
            service.sendMessage(adamToJaneMessage);
            Collection<ChatMessage> janeMessages = service.getMessages("jane" + suffix);
            assertEquals(1, janeMessages.size());
            assertEquals("hi there too" + suffix, janeMessages.iterator().next().message);
            Collection<ChatMessage> adamMessages = service.getMessages("adam" + suffix);
            assertEquals(1, adamMessages.size());
            assertEquals("hi there" + suffix, adamMessages.iterator().next().message);
        }
    }

    public void testError() throws Exception {
        try {
            service.raiseError();
            fail("should have thrown IllegalArgumentException");
        } catch (IllegalArgumentException e) {
        }
    }

    public static Test suite() {
        TestSuite suite = new TestSuite(ActorTest.class);
        return suite;
    }

    private static void runTestsWith(ChatService svc, String logMessage) throws InterruptedException {
        service = svc;
        long started = System.currentTimeMillis();
        Thread[] threads = new Thread[NUM_THREADS];
        for (int i = 0; i < NUM_THREADS; i++) {
            threads[i] = new Thread(new Runnable() {
                public void run() {
                    junit.textui.TestRunner.run(suite());
                }
            });
            threads[i].start();
        }
        for (int i = 0; i < NUM_THREADS; i++) {
            threads[i].join();
        }
        assertEquals(NUM_THREADS * NUM_MESSAGES * 2, service.countMessages());
        long elapsed = System.currentTimeMillis() - started;
        System.out.println("Completed " + logMessage + " in " + elapsed + " millis");
    }

    public static void main(String[] args) throws Exception {
        runTestsWith(new SafeChatServiceImpl(new ChatServiceImpl()), "thread-safe direct");
        runTestsWith((ChatService) ActorFactory.create(ChatService.class, new ChatServiceImpl()), "Actor model");
        runTestsWith((ChatService) ActorFactory.createWithoutActor(ChatService.class, new SafeChatServiceImpl(new ChatServiceImpl())), "thread-safe reflection");
    }
}
Above class demonstrates a variety of way you can instantiate service class, e.g. by simply invoking the constructor, using reflection and using actors. Note that you cannot simply run non-thread-safe implementation because it would produce incorrect results. For each type of service, the test creates 100 threads that try to send and receive messages. By the end of each test, 200,000 messages are sent and retrieved. Following are runtimes for the service invocation:
 Completed thread-safe direct in 286 millis
 Completed Actor model in 1412 millis
 Completed thread-safe reflection in 693 millis

As expected, using builtin synchronization yields the fastest response, reflection adds some overhead and finally actor based communication adds considerable more overhead. One of the feature of Erlang is its support of seamless networking support. Using an abstraction layer as above can easily be enhanced to start an actor on remote machine. Though, in that case the service will only exist on the remote server. As threads in Java are natives and map to light-weight process, they are much more heavy weight than process in Erlang, which only consumes 300 bytes per process. I have found that native threads are more suitable for CPU-bound processing and green or user-based threads suit well for IO-bound processing. In the end, biggest gain of above design from programmer’s perspective is that the client code looks exactly like any other object oriented and unlike Erlang you get real type safety in message passing. Another big advantage is use of polymorphism, above technique uses dependency inversion principle to interface and implementation separate, thus actual implementation can be changed as needed.

You can download all code from here

November 5, 2008

Ten things I hate about Enhydra Shark

Filed under: Computing — admin @ 12:17 pm

I am trying to use Enhydra Shark at work for implementing a workflow system, but I have been finding a lot of issues with it. Here are ten things that I found absolutely aweful about the Enhydra Shark:

  1. All methods throw Exception as part of the signature instead of specific exceptions.
  2. It uses constants instead of enums.
  3. Many classes use public fields and underscore as part of the name, e.g.
                      public final class AppParameter {
                                     public String the_actual_name = null;
     

    it reminds me of a quote ‘You can write FORTRAN in any language”.

  4. Every single Shark method requires transactions and many of the methods open and close several transactions per method, e.g. real code from another group using Shark:
                userTxn = getUserTransaction();
                userTxn.setTransactionTimeout(600);
                userTxn.begin();
                                                                                                  
                 // Connect to the shark process repository
                 SharkConnection connection = shark.getSharkConnection();
                 connection.attachToHandle(executor.getSharkSessionHandle());
     
                 WfProcess process = loadProcess(connection, userTxn);
                 userTxn.commit();
     
                 userTxn.begin();
                 WfActivity activity = loadActivity(shark, userTxn, connection);
                 userTxn.commit();
                                                                                                  
                 userTxn.setTransactionTimeout(600);
                 userTxn.begin();
                 WMSessionHandle toolAgentSession = toolAgent.connect(connectionInfo);
                 toolAgent.invokeApplication(session, toolAgentSession.getId(),
                             appInfo, toolInfo, null, processInstanceId, assignmentId,
                             parameters, appMode);
                 connection.disconnect();
                 userTxn.commit();
     

    Worst, some of the methods require both local and XA transactions.

  5. It uses its own connection factory so I can’t use apache commons or other connection pool. It defines connection properties in several files, which don’t work as expected, e.g. some parameters are used for upper bound and other for lower bound.
  6. It uses its own transaction factory so I can’t use declarative transactions using annotations or Spring. It is just not designed to work with any other framework.
  7. It requires each JVM to connect with a unique engine name and another team had to generate that name on startup.
  8. Its community does not answer questions (I never got response of my questions) and last activity on the mailing list was 10 months old.
  9. It does not use latest XPDL 2.0, which has been released a few years ago.
  10. Due to explicit connection management and transaction management several times per method, other groups using Shark found a lot of problems with multi-threading, scalability and reliability.

I like the frameworks or languages that follow “principle of least surprise”, on the other hand Shark follows “principle of voodoo magic” to get things working. In the end, I have not completely given up on Shark engine, but I have very low expectations.

October 25, 2008

My gripes about REST services

Filed under: REST — admin @ 11:17 am

I love REST style services because of their simplicity and ease of testing. I have discussed benefits of REST earlier. I have used a number of distributed technologies over seventeen years such as LU6.2 (CICS) for mainframes, BSD Sockets, RPC, CORBA, RMI, JINI, Messaging Middlewares, SOAP, etc. In most of those technologies, you had to use special libraries to interact with the server. I worked in some organizations where I saw real dependency hell or jar hell, where I had to import dozens of client jars from different groups to talk to those services. REST, on the other hand only relies on HTTP (and security) libraries.

I first wrote XML over HTTP style service in late 90s before I knew about the REST term. At the time, I worked as a consultant for government DOTs and built CORBA based system to provide traffic data to media agencies. However, people were scraping our website for the data so I built an XML over HTTP service to download the data with some credentials. I think a lot of people used simplicity of HTTP to build similar services. And many of them didn’t understand REST as put forth by Roy Fielding. Over past few years, a lot of people are promoting real style of REST such as Sam Ruby, Stefan Tilkov, Steve Vinosky and Roy Fielding himself. At the same time, they are chastising people who diverge from their vision.

I have tried building REST style services over last few years and recently I have started building new services that will be used for entire organization. As, these services will need to support large number of transactions, performance and scalability are critical. Also, these services need to support batch of requests. In my older blog, I discussed commandments for writing service and wrote about importance of batch requests for scalability. That requirement changes a few things, for example instead of taking advantage of request parameters in HTTP, I had to use XML for input request and had to use XML for response. Though, this style is suitable for POST or PUT where you are expecting to read request as a file but is unnatural for GET types.

Another tenet of REST style services is resource. In this style, you interact with the service similar to how users interact with a web site, i.e., you click on links to go to another page, which returns more links and so on. However, this style in service adds network communication. For example, in one of the service for workflow I had to return active tasks for a particular workflow. In true REST style, I would have to return a hyperlink for each task instead of contents of task and the client would have to ask server for task content by hitting the resource for the task. As, you might guess this adds significant network latency so I am being practical instead of purest and as a result, I am returning task contents. Also, the service allows you to pass requests for multiple workflows and returns a single response for all workflows.

I used similar style when using messaging middlewares that allowed serving batches of requests in a single message. I found messaging middlewares offered more flexible options, e.g. you can design an aggregator service that waits for a few seconds for incoming requests and then puts them in a single request to another service that serve them. These kind of batch services are integral for building scalable systems. I have discussed some of these limitations in earlier blog, especially its lack of push based architecture. Though there are some workaround for push such as long-lived HTTP connection that are difficult to use behind firewall, comet style or reverse Ajax based communication, which is limited and more recently reverse http, which is too new to apply in real applications. Perhaps, messaging standards like XMPP can fill this gap.

I find that IT industry is largely driven by a single minded attitude where every new technology, language or process is adopted by bandwagon of people who just repeat each other’s words without understanding. For example, I have seen similar attitude from agile folks, folks from object oriented programming, aspect oriented programming, metaprogramming or dynamic programming, design patterns, etc. In the end, I believe you have to be practical instead of purest and as always answer to every question in IT is “it depends”, i.e., there is no single universal solution.

October 22, 2008

Developing REST based services using JSR 311

Filed under: Java — admin @ 1:28 pm

Recently, I had to write a few services using REST style in Java. Though I have developed such services in past either using Java Servlet APIs, but I wanted to find something better. I had found writing REST style services in Ruby on Rails a breeze and I wanted something as testable and easy to configure. I found a new JSR 311 that does exactly that. Similar to EJB 3.0 and Servlet 3.0 trend, it allows you to write REST services using annotations. I found an open source project that implements JSR 311 and just released 1.0. I found most of the documentation was Maven and Glassfish driven and I wanted to simply try on Tomcat and Ant so here is how I did it:

Download and Install

First, download following jar libraries (you can see that I am using jars from maven repository because Jersey’s wiki was missing jars or pointing to old version):

Developing

My application used models to describe business object, DAOs for database access and I didn’t want to pollute my business objects with REST annotations so I created two new packages for resources and services. Here is a simple example that follows this separation of concerns:

Business Objects (Model)

The model simply consists of a contact class that stores contact information for a person.

 
 package rest;
 
 public class Contact {
     private final String name;
     private final String email;
     private final String address;
 
     public Contact(final String name, final String email, final String address) {
         this.name = name;
         this.email = email;
         this.address = address;
     }
 
 
     public String getName() {
         return name;
     }
 
     public String getEmail() {
         return email;
     }
 
     public String getAddress() {
         return address;
     }
 }
 

Data Access Layer

The data access layer simply uses a hashmap for storing and accessing these contacts, e.g.

 
 package rest;
 
 import java.util.Collection;
 import java.util.HashMap;
 import java.util.Map;
 
 public class ContactDao {
     private static Map repository = new HashMap();
 
     public boolean save(Contact c) {
         boolean created = repository.get(c.getName()) != null;
         repository.put(c.getName(), c);
         return created;
     }
 
     public boolean delete(String name) {
         return repository.remove(name) != null;
     }
 
     public Contact get(String name) {
         return repository.get(name);
     }
 
     public Collection getAll() {
         return repository.values();
     }
 }
 
 

Resources

For resources, I defined ContactResource that adds XML annotations to convert Contact into XML and ContactsResponse for returning complete XML response, e.g.

 
 package rest;
 import javax.ws.rs.core.UriInfo;
 import javax.xml.bind.annotation.XmlAccessType;
 import javax.xml.bind.annotation.XmlAccessorType;
 import javax.xml.bind.annotation.XmlElement;
 import javax.xml.bind.annotation.XmlRootElement;
 
 import org.codehaus.jettison.json.JSONException;
 import org.codehaus.jettison.json.JSONObject;
                                                                                                                                                       
 
 @XmlAccessorType(XmlAccessType.PROPERTY)
 @XmlRootElement(name = "Contact")
 public class ContactResource {
     private UriInfo uriInfo;
     private Contact contact;
 
     public ContactResource(final UriInfo uriInfo, final String name, final String email, final String address) {
         this.uriInfo = uriInfo;
         this.contact = new Contact(name, email, address);
     }
 
 
     public ContactResource(final Contact contact) {
         this.contact = contact;
     }
 
 
     ContactResource() {
     }
 
 
     public JSONObject toJson() throws JSONException {
         return new JSONObject()
             .put("name", contact.getName())
             .put("email", contact.getEmail())
             .put("address",contact.getAddress());
     }
 
     @XmlElement(name = "Name")
     public String getName() {
         return contact.getName();
     }
 
     @XmlElement(name = "Email")
     public String getEmail() {
         return contact.getEmail();
     }
 
     @XmlElement(name = "Address")
     public String getAddress() {
         return contact.getAddress();
     }
 }
 
 
 

 
 package rest;
 import java.util.ArrayList;
 import java.util.Collection;
 import java.util.Collections;
 
 import javax.xml.bind.annotation.XmlAccessType;
 import javax.xml.bind.annotation.XmlAccessorType;
 import javax.xml.bind.annotation.XmlElement;
 import javax.xml.bind.annotation.XmlRootElement;
 
 
 
 @XmlAccessorType(XmlAccessType.PROPERTY)
 @XmlRootElement(name = "ContactsResponse")
 public class ContactsResponse {
     private String uri;
     private String status;
     private Collection contacts;
 
     public ContactsResponse(String uri, String status, Collection contacts) {
         this.uri = uri;
         this.status = status;
         this.contacts = new ArrayList();
         for (Contact contact : contacts) {
             this.contacts.add(new ContactResource(contact));
         }
     }
 
     public ContactsResponse(String uri, String status, Contact contact) {
         this(uri, status, contact == null ? new ArrayList() : Collections.singleton(contact));
     }
 
     ContactsResponse() {
     }
 
     @XmlElement(name = "Uri")
     public String getUri() {
         return uri;
     }
     public void setUri(String uri) {
         this.uri = uri;
     }
     @XmlElement(name = "Status")
     public String getStatus() {
         return status;
     }
     public void setStatus(String status) {
         this.status = status;
     }
 
     @XmlElement(name = "Contacts")
     public Collection getContactResources() {
         return contacts;
     }
     public void setContactResources(Collection contacts) {
         this.contacts = contacts;
     }
 }
 
 

Service

Here is the meat of JSR 311 that defines annotations for the REST based web service, i.e.,

 
 package rest;
 import java.net.URI;
 import java.net.URISyntaxException;
 import java.util.Collection;
 
 import javax.ws.rs.Consumes;
 import javax.ws.rs.DELETE;
 import javax.ws.rs.FormParam;
 import javax.ws.rs.GET;
 import javax.ws.rs.PUT;
 import javax.ws.rs.Path;
 import javax.ws.rs.PathParam;
 import javax.ws.rs.Produces;
 import javax.ws.rs.core.Context;
 import javax.ws.rs.core.Request;
 import javax.ws.rs.core.Response;
 import javax.ws.rs.core.UriInfo;
 
 import org.codehaus.jettison.json.JSONArray;
 import org.codehaus.jettison.json.JSONException;
 import org.springframework.context.annotation.Scope;
 
 import com.sun.jersey.api.spring.Autowire;
 
 @Path("/contacts/")
 @Produces( { "application/json", "application/xml" })
 @Consumes( { "application/json", "application/xml" })
 @Scope("singleton")
 @Autowire
 public class ContactService {
     private final ContactDao contactDao;
     @Context UriInfo uriInfo;
     @Context Request request;
 
     public ContactService(ContactDao dao) {
         this.contactDao = dao;
     }
 
     public ContactService() {
         this(new ContactDao()); // this will be injected in real-app
     }
 
     /**
      * create contact
      */
     @PUT
     @Consumes("*/*")
     @Produces("application/xml")
     @Path("{name}")
     public Response createcontact(
             @PathParam("name") String name,
             @FormParam("email") String email,
             @FormParam("address") String address) {
         Contact contact = new Contact(name, email, address);
         final boolean newRecord = contactDao.save(contact);
         if (newRecord) {
             try {
                 URI uri = uriInfo != null ? uriInfo.getAbsolutePath()
                         : new URI("/contacts/");
                 return Response.created(uri).build();
             } catch (URISyntaxException e) {
                 throw new RuntimeException("Failed to create uri", e);
             }
         } else {
             return Response.noContent().build();
         }
     }
 
     /**
      * deletes contact 
      * 
      */
     @DELETE
     @Consumes("*/*")
     @Path("{name}")
     public Response deletecontact(@PathParam("name") String name) {
         boolean deleted = contactDao.delete(name);
         if (deleted) {
             return Response.ok().build();
         } else {
             return Response.status(404).build();
         }
     }
 
     /**
      * @return contact in XML format
      */
     @GET
     @Consumes({"text/xml", "application/xml"})
     @Produces("application/xml")
     @Path("{name}")
     public ContactsResponse getcontactByXml(
             @PathParam("name") String name) {
         Contact contact = contactDao.get(name);
         String uri = uriInfo != null ? uriInfo.getAbsolutePath().toString() : "/contacts/";
         return new ContactsResponse(uri, "success", contact);
     }
 
     /**
      * @return contact in JSON format
      */
     @GET
     @Consumes("application/json")
     @Produces("application/json")
     @Path("{name}")
     public JSONArray getcontactByJson(@PathParam("name") String name) throws JSONException {
         Contact contact = contactDao.get(name);
         JSONArray arr = new JSONArray();
         arr.put(new ContactResource(contact).toJson());
         return arr;
     }
 
     /**
      * @return all contacts in XML format
      */
     @GET
     @Consumes({"text/xml", "application/xml"})
     @Produces("application/xml")
     public ContactsResponse getAllByXml() {
         Collection contacts = contactDao.getAll();
         String uri = uriInfo != null ? uriInfo.getAbsolutePath().toString() : "/contacts/";
         return new ContactsResponse(uri, "success", contacts);
     }
 
     /**
      * @return contacts in JSON format
      */
     @GET
     @Consumes("application/json")
     @Produces("application/json")
     public JSONArray getAllByJson() throws JSONException {
         Collection contacts = contactDao.getAll();
         JSONArray arr = new JSONArray();
         for (Contact contact : contacts) {
             arr.put(new ContactResource(contact).toJson());
         }
         return arr;
     }
 }
 
 

A few things to note:

  • @Path defines the URI used for accessing the service
  • I am using @PUT to store contacts (as the user is creating new URI as opposed to @POST where the application uses same URI). Also, I don’t have any method for update (which uses PUT) as I already am using PUT and create method simply updates the contact if it already exist.
  • @PathParam is retrieved from the URI itself, e.g. /contacts/myname
  • @FormParam is retrieved from POST form submission
  • I can use the same URI and return different type of data based on Content-Type, e.g. when user sets it to application/xml or text/xml I return XML content and when user sets it to application/json I return JSON format.
  • To return list of contacts I skip the name and simply use /contacts/

Servlet Configuration

I added servlet to handle REST requests to web.xml, e.g.

     <servlet>
         <servlet-name>RestServlet</servlet-name>
         <servlet-class>com.sun.jersey.spi.spring.container.servlet.SpringServlet</servlet-class>
         <init-param>
             <param-name>com.sun.jersey.config.feature.Redirect</param-name>
             <param-value>true</param-value>
         </init-param>
         <init-param>
             <param-name>com.sun.jersey.config.feature.ImplicitViewables</param-name>
             <param-value>true</param-value>
         </init-param>
         <init-param>
             <param-name>com.sun.jersey.config.property.packages</param-name>
             <param-value>rest</param-value>
         </init-param>
         <load-on-startup>1</load-on-startup>
     </servlet>
     <servlet-mapping>
         <servlet-name>RestServlet</servlet-name>
         <url-pattern>/svc/*</url-pattern>
     </servlet-mapping>
 

I used Spring to inject DAOs in real application but if you don’t need it then replace com.sun.jersey.spi.spring.container.servlet.SpringServlet with com.sun.jersey.spi.container.servlet.ServletContainer.

Also note that com.sun.jersey.config.property.packages defines package name of Java classes that defines service classes.

Deploying

I am assuming you already know how to package a war file and deploy it to application server such as Tomcat.

Testing it

Unit Testing it

A big advantage of JSR 311 is ease of test, e.g. here is a sample unit test:

 
 package rest;
 import javax.ws.rs.core.Response;
 
 import junit.framework.TestCase;
 import org.codehaus.jettison.json.JSONArray;
 import org.codehaus.jettison.json.JSONObject;
 
 import com.sun.jersey.api.NotFoundException;
 
 public class ContactServiceTest extends TestCase {
     private ContactService service;
 
     @Override
     protected void setUp() throws Exception {
         service = new ContactService();
     }
 
     public void testCreateContactWithGetContactByXml() {
         final String name = "shahbhat";
         final String email = "shahbhat@myhost";
         final String address = "shahbhat address ";
 
         Response response = service.createContact(name, email, address);
         assertEquals(201, response.getStatus());
         assertNull(response.getEntity());
 
         // recreate the same mapping and it should return no content
         response = service.createContact(name, email, address);
         assertEquals(204, response.getStatus());
         assertNull(response.getEntity());
 
         ContactsResponse contactResponse = service.getContactByXml(name);
         assertEquals(1, contactResponse.getContactResources().size());
         ContactResource contactResource = contactResponse.getContactResources().iterator().next();
         assertEquals(name, contactResource.getName());
         assertEquals(email, contactResource.getEmail());
         assertEquals(address, contactResource.getAddress());
     }
 
     public void testCreateContactWithGetContactByJson() throws Exception {
         final String name = "shahbhat";
         final String email = "shahbhat@myhost";
         final String address = "shahbhat address ";
 
         Response response = service.createContact(name, email, address);
         assertEquals(201, response.getStatus());
         assertNull(response.getEntity());
 
         JSONArray jsonArray = service.getContactByJson(name);
         assertEquals(1, jsonArray.length());
         JSONObject json = jsonArray.getJSONObject(0);
         assertEquals(name, json.getString("name"));
         assertEquals(email, json.getString("email"));
         assertEquals(address, json.getString("address"));
     }
 
     public void testDeleteContact() {
         final String name = "shahbhat";
         final String email = "shahbhat@myhost";
         final String address = "shahbhat address ";
 
         Response response = service.createContact(name, email, address);
         assertEquals(201, response.getStatus());
         assertNull(response.getEntity());
 
         ContactsResponse contactResponse = service.getContactByXml(name);
         assertEquals(1, contactResponse.getContactResources().size());
         ContactResource contactResource = contactResponse.getContactResources().iterator().next();
         assertEquals(name, contactResource.getName());
         assertEquals(email, contactResource.getEmail());
         assertEquals(address, contactResource.getAddress());
 
         response = service.deleteContact(name);
         assertEquals(200, response.getStatus());
 
         contactResponse = service.getContactByXml(name);
         assertEquals(0, contactResponse.getContactResources().size());
 
         //
         response = service.deleteContact(name);
         assertEquals(404, response.getStatus());
 
     }
 
     public void testGetAllContactByXml() {
         service.createContact("shahbhat", "shahbhat email", "shahbhat address");
         service.createContact("bill", "bill email", "bill address");
 
         ContactsResponse contactResponse = service.getAllByXml();
         assertEquals(2, contactResponse.getContactResources().size());
         for (ContactResource contactResource : contactResponse.getContactResources()) {
             if ("shahbhat".equals(contactResource.getName())) {
                 assertEquals("shahbhat email", contactResource.getEmail());
                 assertEquals("shahbhat address", contactResource.getAddress());
             } else if ("bill".equals(contactResource.getName())) {
                 assertEquals("bill email", contactResource.getEmail());
                 assertEquals("bill address", contactResource.getAddress());
             } else {
                 fail("unknown contact " + contactResource);
             }
         }
 
         service.deleteContact("shahbhat");
         service.deleteContact("bill");
 
         contactResponse = service.getAllByXml();
         assertEquals(0, contactResponse.getContactResources().size());
     }
 
 
     public void testGetAllContactByJson() throws Exception {
         service.createContact("shahbhat", "shahbhat email", "shahbhat address");
         service.createContact("bill", "bill email", "bill address");
 
         JSONArray jsonArray = service.getAllByJson();
         assertEquals(2, jsonArray.length());
 
         for (int i=0; i<2; i++) {
             JSONObject json = jsonArray.getJSONObject(i);
             if ("shahbhat".equals(json.getString("name"))) {
                 assertEquals("shahbhat email", json.getString("email"));
                 assertEquals("shahbhat address", json.getString("address"));
             } else if ("bill".equals(json.getString("name"))) {
                 assertEquals("bill email", json.getString("email"));
                 assertEquals("bill address", json.getString("address"));
             } else {
                 fail("unknown contact " + json);
             }
         }
         service.deleteContact("shahbhat");
         service.deleteContact("bill");
 
         jsonArray = service.getAllByJson();
         assertEquals(0, jsonArray.length());
     }
 }
 

Functional Testing it

Once deploy, you can use curl to functionally test it (though there are other automated tools available as well), e.g

Creating Contact

 curl -X PUT -d "email=myemail&address=myaddress"  http://shahbhat.desktop:8080/svc/contacts/bhatti
 

It will create a contact and to retrieve it, use

 curl http://shahbhat.desktop:8080/svc/contacts/bhatti
 

It should return

 <?xml version="1.0" encoding="UTF-8" standalone="yes"?><ContactsResponse><Contacts><Address>myaddress</Address><Email>myemail</Email><Name>bhatti</Name></Contacts><Status>success</Status><Uri>http://shahbhat.desktop:8080/svc/contacts/bhatti</Uri></ContactsResponse>
 

To get JSON format use

 curl --header "Content-Type: application/json" http://shahbhat.desktop:8080/svc/contacts/bhatti
 

and it should return

 [{"name":"bhatti","email":"myemail","address":"myaddress"}]
 

You can create another contact e.g.

 curl -d "email=billemail&address=billaddress"  http://shahbhat.desktop:8080/svc/contacts/bill
 

Now to get all contacts use

 curl http://shahbhat.desktop:8080/svc/contacts/
 

and it will return

 <?xml version="1.0" encoding="UTF-8" standalone="yes"?><ContactsResponse><Contacts><Address>myaddress</Address><Email>myemail</Email><Name>bhatti</Name></Contacts><Contacts><Address>billaddress</Address><Email>billemail</Email><Name>bill</Name></Contacts><Status>success</Status><Uri>http://shahbhat.desktop:8080/svc/contacts/</Uri></ContactsResponse>
 

And to get JSON format use

 curl --header "Content-Type: application/json" http://shahbhat.desktop:8080/svc/contacts/
 

which will return

 [{"name":"bhatti","email":"myemail","address":"myaddress"},{"name":"bill","email":"billemail","address":"billaddress"}]
 

Conclusion

I found this approach is easily testable with standard unit test and easy to develop. Though, there are few glitches, e.g. @QueryParam only works with GET and if you are POSTing to something like /uri?x=yy then you won't see query parameters. But, overall I am happy with the outcome and will continue to use this framework.

October 18, 2008

Concurrency Constructs

Filed under: Computing — admin @ 11:43 am

With the proliferation of multi-cores, concurrency has been a hot topic for past few years. I described a few ways to tackle concurrency in my blog earlier this year. As I mentioned in my blog, this problem is nothing for scientific and research community. Also, many languages have had devised numerous constructs to tackle these abstractions and other languages are adding support for concurency. I will go over some of these abstractions here:

Processes

All modern operating systems support preemptive multitasking and processes have been simple way to add concurrency. For example, this technique is used by Apache web server that launches multiple processes and serve thousands of clients at a time. There are also a number of system primitives for inter-process communication such as shared memory, sockets, pipes and other forms of middlewares and peer to peer technology. This technique is specially suitable for many scripting languages that don’t have concurrency support such as Perl, Ruby, CPython, etc. Despite their scripting nature, these languages are highly productive. For example, much of Amazon’s retail website is built using Perl/Mason and uses multiple processes served by FastCGI for handling concurrent requests. Though, Ruby and Python support green threads, but are not designed for concurrency. For example, Global Interpreter Lock (GIL) in Python only allows one thread to execute at a time. Similarly, Ruby has Global VM lock that also allows one thread to execute at a time. These limitations have also effected kind of applications you can built. For example, one of criticism of Ruby on Rails framework has been its lack of concurrency. Though, there have been a number of solutions that tackle this problem by multi-processing such as clusters of mongrel, FastCGI, etc.

This model is also used by Fork/Join, Map/Reduce, Master-Slave and Leader/Follower style of parallel programming, where the master process creates work and distributes work to workers, who work on it. There are a plenty of libraries and frameworks available for this such as Globus, Hadoop, MPI/MPL, PVM and Fork/Join in Java 7.

Kernel Threads

Many languages support kernel threads that are mapped to lightweight processes (LWP) and scheduled by the operating systems. The operating system context switches between threads using time-slicing that give perception of concurrency. Posix compliant operating systems offer a number of solutions for inter-thread communication and synchronization such as mutex, semaphores, condition variables. Many languages offer support for high level synchronization such as concurrent library in Java.

Green Threads/Fibers

As opposed to native threads, green threads run in user space and scheduled by using yield to allow other thread/fiber to execute. As user threads or fibers rely on collaboration, they are easily subject to starvation. Some languages encapsulate blocking I/O with nonblocking I/O and automatically yield fiber when they make blocking call. Effective use of fibers can help applications scale more than multi-processing or multi-threading, which are generally limited by the system resources. For example, in a performance comparison between Apache and Yaws, where latter was built in Erlang that uses user processes was able to scale upto 80,000 concurrent connections whereas Apache died at around 4000 connections. Ruby 1.9 has just added support for fibers, which can be used to build streams, e.g.

 fib = Fiber.new do  
    x, y = 0, 1 
    loop do  
     Fiber.yield y 
     x,y = y,x+y 
    end 
   end 
   20.times { puts fib.resume } 
 

Streams

Streams are sort of iterators over infinite lists and often used in functional languages, though many imperative programming languages now support them to create infinite lists. You can find a number of examples of Strams in languages like Groovy, Scala, Haskell, etc. Python offers similar capability via generators that contain yield keyword, which causes compiler to suspend generator until next() method of generator is called. For example,

 def generate_ints(N):
     for i in range(N):
         yield i
 
 >>> gen = generate_ints(2)
 >>> gen.next()
 >>> gen.next()
 

Coroutines

Coroutines are similar user threads and unlike subroutines, they have multiple entry and exit points. They are often built using contiuations or generators. In C, setjmp and longjmp can be used to implement coroutine. Ruby 1.9 added support for fibers that are more accurately semi-coroutines [pp 167].

Continuation

Continuation work like saving game at a checkpoint and resuming it. Traditionaly goto were used to implement continuations, though a number of languages have native support for continuations such as Scheme, Smalltalk, Ruby (callcc), Scala, C (setjmp/longjmp), etc. Though, they are not without problems but Seaside web framework based on Smalltalk has shown innovative way to write web applications using continuation passing style.

Actor model

I was introduced to actor model in mid mid 90s when working on graduate and post-graduate work. An actor has its own thread of execution with a mailbox. It communicates to other actors by sending messages that are delivered to its mailbox. Erlang, a language designed for concurrency, is based on actor model and more recently Scala has adopted actor model as well. When I worked for high performance computing (HPC) area in 90s, I used SGI systems that were built using NUMA based shared memory and messaging passing systems/libraries such as MPI, MPL, etc. I found message passing systems were much more scalable than shared memory, so I believe this style of programming will have best chance of succeeding. Though, I admit I found converting algorithms into message passing style is not easy.

Reactive or Event driven style

In reactive or event driven style, systems communicate by sending or listening on events. It’s sort of flip-style of threading, instead of creating new thread for each new concurrent task, it uses a fixed number of threads that cooperate. It is more scalable than multi-threading (kernel) as native threads are limited by available resources and context switching is expensive. For example, event-driven Mongrel server is much more scalable than Mogrel server. A number of J2EE application servers such as Glassfish, Jetty and Tomcat 6 have adopted Grizzly framework that uses asynchronous I/O to implement reactive style servers.

Software Transactional Memory

It’s difficult to write correct program using shared memory and as it requires proper locking of shared data and special attention to thread starvation and deadlock issues. A number of languages like Clojure, Haskell and Fortress have added support for software transactional memory that provide implicit locking through transactions. I have not used this style of programming but it seems to make writing concurrent applications easy. Though, I believe scalability with shared memory may only help with systems upto a few cores or processors.

TupleSpace based model

In tuple space, processes communicate to each other by storing messages in tuple spaces. The tuple space provides simple APIs like get, put, read and eval. Where get/read are blocking operations to implement dataflow based application and eval creates a new process to execute a task and store result in the tuple space. I built a similar system called JavaNOW, though there are a number of open source and commercial frameworks availeble such as JavaSpaces, GigaSpaces, Blitz.

Declarative concurrency

I heard recently Anders Hejlsberg and Guy Steele talked about concurrency and Anders suggested declarative concurrency where the programmer indicates parts of the applications that can be run concurrently. In a way, it is similar to compiler based parallelism used in High performance Fortran or Parallel Fortran that used data flow analysis to create parallel applications.

In nutshell, I find that more and more languages are adding constructs and frameworks to address concurrency and we will have to get used to these constructs to write programs that are future proof.

October 14, 2008

Observe, Orient, Decide and Act

Filed under: Methodologies — admin @ 8:49 pm

I have heard agile evangilists talk about influence of John Boyd on agile methodologies. I have also read about his story in books like Release It and more recently in Simple Architectures for Complex Enterprises, which I am currently reading. You can read more about Boyd’s background from wikipedia link above but he came up with iterative cycle of: observe, orient, decide and act (OODA). One of key finding he made was that shorter feedback or iteration loop of OODA with low quality was better than longer or tiring cycle of OODA with high quality. Despite the fact that everyone calls his/her organization agile, this feedback loop is real essense of agility.

October 12, 2008

Modules, Libraries, Components, Frameworks, Services and Platforms

Filed under: Computing — admin @ 9:21 pm

The software development is complex by nature and over the years, I have seen many abstractions for addressing this complexity such as modules, classes, libraries, components, packages, aspects, frameworks and platform, services, etc. Back when I started programming in 80s, I learned about Structured Programming and Top-down Design. Like modern good software practices it encouraged separation of concernsand was modular design. I learned from structured programming that modules should be loosely coupled and highly cohesive. I learned about various forms of coupling such as common, control, data, stamp, etc and how to avoid it. BASIC was my first programming language and I learned a number of languages afterwards, but C became my primary language. In that language, module meant c file that consisted of closely related functions.

I also understood concept of libraries (static), which were sort of modules or composition of modules that offered some useful functionality that I could link at compile/load time. In early 90s, I started learning object oriented programming and design and started using C++. It also highlighted modularity and introduced classes for combining data structure and behavior. It also emphasised encapsulation or information hiding and interfaces. Despite its impurity, I used C++ for many years. It was similar to C and it also used files for modules by defining closely related files in the same file.

In 1995, I also started using Java, which limited one class per file and offered packages for namespace. I heard many people equated modules to classes but I preferred packages for defining modules. Java also offered jar files for packaging libraries. In early days, Java also promoted JavaBeans as a way to add components and runtime wiring.

Around mid 90s to early 2000, I started learning about components, which were hot topics in those days. I learned Catalysis, Iconix and Component Software. A lot of companies were defining what components meant, such as SOM, Micrsoft’s COM, or their distributed counterparts such as DCOM, DSOM, CORBA, etc. Unlike static libraries, which were packaged as part of the application, those components offered binary and runtime pluggability. I used CORBA for many years, then moved to EJBs in early 2000s which offered more object oriened support for components. Historically, compile time components were difficult to use and led to DLL or Jar hell. Recent standars such as OSGI and Java JSR 291 offer nice dependency management and runtime isolation.

In mid 90s, I enrolled for Ph.D and was looking for interesting topics. One of my professor gave me some papers from Xeros Parc about aspect oriented programming. I ended up choosing different topic but liked the idea of aspects. AOP became popular in early 2000s and I have been using them whenever I get chance. Aspects addressed crosscutting concerns and offered a powerful way to modularize concerns like security, monitoring, transactions, caching, etc.

Web Services and Services oriented Architecture became popular in early 2000 and I started using Sun’s toolkit and Weblogic’s generators to convert Java/EJBs to Web Services. Similar to modules, libraries and components they try to sell on partitioning and reusability, though unlike former, the services didn’t have any compile or load time dependencies.

The component based software development then led to frameworks and I started hearing more about frameworks for UIs, eclipse IDE, etc. I found frameworks were different from libraries in several ways. For example, unlike libraries where client code simply invokes some functionality, the frameworks offered scaffolding to built applications and used inversion of control by offering hooks, callbacks or plugins to implment. I have been using a number of frameworks for many years such as Spring, Eclipse, Rails, etc.

For past a few years, more and more companies now offer platforms to build applications. The platforms are sort of like composition of frameworks with an API to build applications. Though, historically platforms were tied with specific operating systems or languages such as Java, .Net, iPhone, JavaME but Web has become the platform, which offers language and operating system independent services. There are unlimited choices for building new mashups and applications (see http://www.programmableweb.com/. Though, some of the web platforms such as Facebook, MySpace, Google App
Engine and Amazon Web Services offer more complete application hosting facilities and in the case of Facebook, MySpace or Twitter they also offer millions of potential users.

The cost of software development has become significantly cheaper due to open source and easily accessible libraries, frameworks and platforms. Though, what I described in this blog is no news to anyone, but I think it’s imporant to think about these differences and choices when building new applications.

October 3, 2008

Traits of good programmers

Filed under: Java — admin @ 11:00 pm

I love programming and have been doing it for more than 20 years. In that lifetime, I have also worked with a lot of good programmers and have discussed many times “what makes a good programmer” informally or during job interviews. Here are some of important traits that I have found in good programmers:

  • Simplicity and Clarity of thinking – I have found good programmers can think clearly and try to understand essense of underlying problem. Good programmers then develop simple solutions with minimal accidental complexity. I have found systems developed by good programs are easier to understand due to clarity of program design.
  • Passionate about programming – Programmming is a profession and I have seen a lot of programmers who considered programming a 9-5 job, who didn’t care about it outside the job. Many of those programmers did the same job for many years without learning anything new. I have found that best programmers care a great deal about programming and have a great passion about it. One way to distinguish programmers with passion is to ask them about side projects or open source projects that they started or worked on.
  • Continuous Learning – I have found good programmers spend a lot of time learning new skills or finding better ways to do things.
  • Generalist – Due to passion and ability to learn new things, good programmers have breadth of knowledge with varying skills and technolgies. They have modeling, UI, application, systems, database and other skills needed for softrware development.
  • Detail oriented – Programming requires a detail oriented person who can think about a problem and all issues that can effect the solution. I have found good programmers can think of all branch logic and all edge cases when solving a problem.
  • Visualize working of programs – Good programs have ability to visualize inner working of programs and can quickly pinpoint problem areas when things break.
  • Excellence vs Perfectionist – Real world projects are constrained by time and resources and good programmers opt for excellence by doing their best within those constraints rather than being perfectionist.
  • Inquisitive and inquiring mind – One of the hardest part of software development is determining key requirements and differentiating between wants and needs. Often stakeholders don’t know what they want. Good programmers work with users to detrmine what’s really needed for the system. Good programmers can probe users about real underlying problem that they are trying solve. In such systems, agile methodology also help where users can see something working and develop iteratively.
  • Communication – Good programmers communicate in multiple ways, they use design or code to communicate with other programmers. Senior programmers often need to sell or persuade other programmers about architecture or new practices.
  • Integrity and Courage – Though, this is a principle of XP but it’s also one of key trait of good programmers. I have found in most companies, management pulls deadlines and requirements out of the hat and good programmers have courage to speak up to the management when they see death march timelines.
  • Humility – This is also another attribute of XP but I have found in good programmers because programming is hard profession and everyone makes mistake. Though, I have also seen a lot of prima donnas who were smart (some of them) but were difficult to work with. On the other hand, I have found good programmers are humble and easy to work with.
  • Responsibility – Good programmers are profesional and they take full responsibility in developing fully tested and working solutions for the users.
  • Golden hammer – Due to being generalist, good programmers is less likely to use same solution or golden hammer to varying problems. Also, due to breadth of skills he/she is less likely to be swayed by technology or language de jure, instead good programmers try to evaluate real benefit of new technologies with some side projects.
  • Toolbox of solutions – Good programmers learn from varying solutions and often recognize patterns and similar problems that they have previously worked on. They use this toolbox of solutions and patterns when approaching new problem.
  • Productive – Often you hear about good programmers being 10 or 20 times more productive. Though, I admit it’s hard to measure productivity but I have found that good programmers are productive mostly due to above traits. For example, they develop simple code which is faster to develop and has less defect. They focus on essential probems that solve real problems. They are faster mostly because they practice alot with side projects and recognize patterns and previously solved solutions. When debugging and trouble shooting they can visualize software systems and can see problem areas. They don’t like to hackup things that will break in two weeks, instead they have courage to tell stakeholders risks of meeting unrealistic deadlines or releasing untested applications.

September 9, 2008

Setting up JRuby on Rails

Filed under: Ruby — admin @ 11:25 am

In my last blog, I wrote about recent JRuby on Rails project I did at amazon. In this blog, I will show some technical details on setting up JRuby on Rails with Tomcat. I used 2.1.1 of Rails and 1.1.4 of JRuby, here are steps to setup JRuby and Rails:

  • Download Jruby 1.1.4 from http://dist.codehaus.org/jruby/
  • Setup JRUBY_HOME and PATH
     export JRUBY_HOME=installed directory of jruby
     export PATH=$JRUBY_HOME/bin:$PATH
     
  • Install Rails
     gem install rails -y
     
  • Install ActiveRecord JDBC Adapter
     jruby -S gem install activerecord-jdbc-adapter
     
  • Install warbler plugin to create war file, older version of Rails (1.2 or older) used goldspike but the newer version requires warbler
     jruby -S gem install -y  warbler
     
  • Edit config/environment.rb and add
     if RUBY_PLATFORM =~ /java/
       require 'rubygems'
       RAILS_CONNECTION_ADAPTERS = %w(jdbc)
     end
     
  • Download mysql-jdbc driver as I used MySQL. (You may have to download different driver. )
  • Copy mysql-connector-java-5.1.6-bin.jar to lib directory of jruby
  • You can edit config/warble.rb and add other jar files or directories, e.g.
       config.java_libs += FileList["lib/*.jar"]
       config.pathmaps.java_classes << "%{build/classes/,}p"
       config.webxml.jruby.min.runtimes = 2
       config.webxml.jruby.max.runtimes = 24
     
  • Edit config/environment.rb and add define adapter, driver and url as follows:
     development:
       adapter: jdbc
       driver: com.mysql.jdbc.Driver
       url: jdbc:mysql://localhost/rspm_development
       username: rspmdb_user
       password: secret
                                                                                                                                                           
     test:
       adapter: jdbc
       driver: com.mysql.jdbc.Driver
       url: jdbc:mysql://localhost/rspm_test
       username: rspmdb_user
       password: secret
                                                                                                                                                           
     production:
       adapter: jdbc
       driver: com.mysql.jdbc.Driver
       url: jdbc:mysql://localhost/rspm_production
       username: rspmdb_user
       password: secret
     
  • Create a template web.xml.erb in config directory, e.g.
     <!DOCTYPE web-app PUBLIC
       "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN"
     
       "http://java.sun.com/dtd/web-app_2_3.dtd">
     <web-app>
                                                                                                                                                           
         <% webxml.context_params.each do |k,v| %>
         <context-param>
             <param-name><%= k %></param-name>
             <param-value><%= v %></param-value>
     
         </context-param>
         <% end %>
                                                                                                                                                           
        <servlet>
           <servlet-name>Rails</servlet-name>
           <servlet-class>org.jruby.rack.RackServlet</servlet-class>
     
        </servlet>
                                                                                                                                                           
        <servlet-mapping>
            <servlet-name>Rails</servlet-name>
            <url-pattern>/*</url-pattern>
        </servlet-mapping>
     
                                                                                                                                                           
                                                                                                                                                           
       <filter>
         <filter-name>RackFilter</filter-name>
         <filter-class>org.jruby.rack.RackFilter</filter-class>
       </filter>
     
       <filter-mapping>
         <filter-name>RackFilter</filter-name>
         <url-pattern>/*</url-pattern>
       </filter-mapping>
                                                                                                                                                           
        <listener>
     
           <listener-class><%= webxml.servlet_context_listener || 'org.jruby.rack.RackServletContextListener'%></listener-class>
        </listener>
                                                                                                                                                           
     </web-app>
                                                                                                                                                           
     
  • Now create war file
     jruby -S warble
     
  • Finally copy the war file to webapps directory of standard J2EE web container, in my case Tomcat 6.0
  • Now point the browser to your server, e.g. http://localhost:8080/your-controller and voilla.

On final note, I found warble quite slow in generating war file and tomcat takes a while to deploy the war file, so I use WEBrick for development and Tomcat for staging/production.

« Newer PostsOlder Posts »

Powered by WordPress