Modules, Libraries, Components, Frameworks, Services and Platforms
October 12th, 2008
The software development is complex by nature and over the years, I have seen many abstractions for addressing this complexity such as modules, classes, libraries, components, packages, aspects, frameworks and platform, services, etc. Back when I started programming in 80s, I learned about Structured Programming and Top-down Design. Like modern good software practices it encouraged separation of concernsand was modular design. I learned from structured programming that modules should be loosely coupled and highly cohesive. I learned about various forms of coupling such as common, control, data, stamp, etc and how to avoid it. BASIC was my first programming language and I learned a number of languages afterwards, but C became my primary language. In that language, module meant c file that consisted of closely related functions.
I also understood concept of libraries (static), which were sort of modules or composition of modules that offered some useful functionality that I could link at compile/load time. In early 90s, I started learning object oriented programming and design and started using C++. It also highlighted modularity and introduced classes for combining data structure and behavior. It also emphasised encapsulation or information hiding and interfaces. Despite its impurity, I used C++ for many years. It was similar to C and it also used files for modules by defining closely related files in the same file.
In 1995, I also started using Java, which limited one class per file and offered packages for namespace. I heard many people equated modules to classes but I preferred packages for defining modules. Java also offered jar files for packaging libraries. In early days, Java also promoted JavaBeans as a way to add components and runtime wiring.
Around mid 90s to early 2000, I started learning about components, which were hot topics in those days. I learned Catalysis, Iconix and Component Software. A lot of companies were defining what components meant, such as SOM, Micrsoft’s COM, or their distributed counterparts such as DCOM, DSOM, CORBA, etc. Unlike static libraries, which were packaged as part of the application, those components offered binary and runtime pluggability. I used CORBA for many years, then moved to EJBs in early 2000s which offered more object oriened support for components. Historically, compile time components were difficult to use and led to DLL or Jar hell. Recent standars such as OSGI and Java JSR 291 offer nice dependency management and runtime isolation.
Web Services and Services oriented Architecture became popular in early 2000 and I started using Sun’s toolkit and Weblogic’s generators to convert Java/EJBs to Web Services. Similar to modules, libraries and components they try to sell on partitioning and reusability, though unlike former, the services didn’t have any compile or load time dependencies.
The component based software development then led to frameworks and I started hearing more about frameworks for UIs, eclipse IDE, etc. I found frameworks were different from libraries in several ways. For example, unlike libraries where client code simply invokes some functionality, the frameworks offered scaffolding to built applications and used inversion of control by offering hooks, callbacks or plugins to implment. I have been using a number of frameworks for many years such as Spring, Eclipse, Rails, etc.
For past a few years, more and more companies now offer platforms to build applications. The platforms are sort of like composition of frameworks with an API to build applications. Though, historically platforms were tied with specific operating systems or languages such as Java, .Net, iPhone, JavaME but Web has become the platform, which offers language and operating system independent services. There are unlimited choices for building new mashups and applications (see http://www.programmableweb.com/. Though, some of the web platforms such as Facebook, MySpace, Google App Engine and Amazon Web Services offer more complete application hosting facilities and in the case of Facebook, MySpace or Twitter they also offer millions of potential users.
The cost of software development has become significantly cheaper due to open source and easily accessible libraries, frameworks and platforms. Though, what I described in this blog is no news to anyone, but I think it’s imporant to think about these differences and choices when building new applications.
New Photos From Abu Ghraib
October 9th, 2008
Seymour Hersh: Kids sodomized at Abu Ghraib, Pentagon has the videos
Disturbing New Photos From Abu Ghraib
It’s Torture… No Matter What George Bush Calls It
Israeli missiles silence baby’s laughter in Gaza
Slaughtering of children in Gaza
Military Justifies Attack That Killed at Least 33 Afghan Civilians (including 12 children)
http://wiredispatch.com/news/?id=68204
Children in Iraq
Traits of good programmers
October 3rd, 2008
I love programming and have been doing it for more than 20 years. In that lifetime, I have also worked with a lot of good programmers and have discussed many times “what makes a good programmer” informally or during job interviews. Here are some of important traits that I have found in good programmers:
- Simplicity and Clarity of thinking - I have found good programmers can think clearly and try to understand essense of underlying problem. Good programmers then develop simple solutions with minimal accidental complexity. I have found systems developed by good programs are easier to understand due to clarity of program design.
- Passionate about programming - Programmming is a profession and I have seen a lot of programmers who considered programming a 9-5 job, who didn’t care about it outside the job. Many of those programmers did the same job for many years without learning anything new. I have found that best programmers care a great deal about programming and have a great passion about it. One way to distinguish programmers with passion is to ask them about side projects or open source projects that they started or worked on.
- Continuous Learning - I have found good programmers spend a lot of time learning new skills or finding better ways to do things.
- Generalist - Due to passion and ability to learn new things, good programmers have breadth of knowledge with varying skills and technolgies. They have modeling, UI, application, systems, database and other skills needed for softrware development.
- Detail oriented - Programming requires a detail oriented person who can think about a problem and all issues that can effect the solution. I have found good programmers can think of all branch logic and all edge cases when solving a problem.
- Visualize working of programs - Good programs have ability to visualize inner working of programs and can quickly pinpoint problem areas when things break.
- Excellence vs Perfectionist - Real world projects are constrained by time and resources and good programmers opt for excellence by doing their best within those constraints rather than being perfectionist.
- Inquisitive and inquiring mind - One of the hardest part of software development is determining key requirements and differentiating between wants and needs. Often stakeholders don’t know what they want. Good programmers work with users to detrmine what’s really needed for the system. Good programmers can probe users about real underlying problem that they are trying solve. In such systems, agile methodology also help where users can see something working and develop iteratively.
- Communication - Good programmers communicate in multiple ways, they use design or code to communicate with other programmers. Senior programmers often need to sell or persuad other programmers about architecture or new practices.
- Integrity and Courage - Though, this is a principle of XP but it’s also one of key trait of good programmers. I have found in most companies, management pulls deadlines and requirements out of the hate and good programmers have courage to speak up to the management when they see death march timelines.
- Humility - This is also another attribute of XP but I have found in good programmers because programming is hard profession and everyone makes mistake. Though, I have also seen a lot of prima donnas who were smart (some of them) but were difficult to work with. On the other hand, I have found good programmers are humble and easy to work with.
- Responsibility - Good programmers are profesional and they take full responsibility in developing fully tested and working solutions for the users.
- Golden hammer - Due to being generalist, good programmers is less likely to use same solution or golden hammer to varying problems. Also, due to breadth of skills he/she is less likely to be swayed by technology or language de jure, instead good programmers try to evaluate real benefit of new technologies with some side projects.
- Toolbox of solutions - Good programmers learn from varying solutions and often recognize patterns and similar problems that they have previously worked on. They use this toolbox of solutions and patterns when approaching new problem.
- Productive - Often you hear about good programmers being 10 or 20 times more productive. Though, I admit it’s hard to measure productivity but I have found that good programmers are productive mostly due to above traits. For example, they develop simple code which is faster to develop and has less defect. They focus on essential probems that solve real problems. They are faster mostly because they practice alot with side projects and recognize patterns and previously solved solutions. When debugging and trouble shooting they can visualize software systems and can see problem areas. They don’t like to hackup things that will break in two weeks, instead they have courage to tell stakeholders risks of meeting unrealistic deadlines or releasing untested applications.
Muslim Children Gassed at Dayton Mosque After “Obsession” DVD Hits Ohio
September 28th, 2008
Muslim Children Gassed at Dayton Mosque After “Obsession” DVD Hits Ohio
- “She told me that the gas was sprayed into the room where the babies and children were being kept while their mothers prayed together their Ramadan prayers. Panicked mothers ran for their babies, crying for their children so they could flee from the gas that was burning their eyes and throats and lungs. She grabbed her youngest in her arms and grabbed the hand of her other daughter, moving with the others to exit the building and the irritating substance there.
- “The paramedic said the young one was in shock, and gave her oxygen to help her breathe. The child couldn’t stop sobbing.
- “This didn’t happen in some far away place — but right here in Dayton, and to my friends. Many of the Iraqi refugees were praying together at the Mosque Friday evening. People that I know and love.
- “I am hurt and angry. I tell her this is NOT America. She tells me this is not Heaven or Hell — there are good and bad people everywhere.
- “She tells me that her daughters slept with her last night, the little one in her arms and sobbing throughout the night. She tells me she is afraid, and will never return to the mosque, and I wonder what kind of country is this where people have to fear attending their place of worship?
- “The children come into the room, and tell me they want to leave America and return to Syria, where they had fled to from Iraq. They say they like me, … , and other American friends — but they are too afraid and want to leave. Should a 6 and 7 year old even have to contemplate the safety of their living situation?
- “Did the anti-Muslim video circulating in the area have something to do with this incident, or is that just a bizarre coincidence? Who attacks women and children?
- “What am I supposed to say to them? My words can’t keep them safe from what is nothing less than terrorism, American style. Isn’t losing loved ones, their homes, jobs, possessions and homeland enough? Is there no place where they can be safe?
- “She didn’t want me to leave her tonight, but it was after midnight, and I needed to get home and write this to my friends. Tell me — tell me — what am I supposed to say to them?”
Setting up JRuby on Rails
September 9th, 2008
In my last blog, I wrote about recent JRuby on Rails project I did at amazon. In this blog, I will show some technical details on setting up JRuby on Rails with Tomcat. I used 2.1.1 of Rails and 1.1.4 of JRuby, here are steps to setup JRuby and Rails:
- Download Jruby 1.1.4 from http://dist.codehaus.org/jruby/
- Setup JRUBY_HOME and PATH
export JRUBY_HOME=installed directory of jruby export PATH=$JRUBY_HOME/bin:$PATH
- Install Rails
gem install rails -y
- Install ActiveRecord JDBC Adapter
jruby -S gem install activerecord-jdbc-adapter
- Install warbler plugin to create war file, older version of Rails (1.2 or older) used goldspike but the newer version requires warbler
jruby -S gem install -y warbler
- Edit config/environment.rb and add
if RUBY_PLATFORM =~ /java/ require 'rubygems' RAILS_CONNECTION_ADAPTERS = %w(jdbc) end
- Download mysql-jdbc driver as I used MySQL. (You may have to download different driver. )
- Copy mysql-connector-java-5.1.6-bin.jar to lib directory of jruby
- You can edit config/warble.rb and add other jar files or directories, e.g.
config.java_libs += FileList["lib/*.jar"] config.pathmaps.java_classes << "%{build/classes/,}p" config.webxml.jruby.min.runtimes = 2 config.webxml.jruby.max.runtimes = 24 - Edit config/environment.rb and add define adapter, driver and url as follows:
development: adapter: jdbc driver: com.mysql.jdbc.Driver url: jdbc:mysql://localhost/rspm_development username: rspmdb_user password: secret test: adapter: jdbc driver: com.mysql.jdbc.Driver url: jdbc:mysql://localhost/rspm_test username: rspmdb_user password: secret production: adapter: jdbc driver: com.mysql.jdbc.Driver url: jdbc:mysql://localhost/rspm_production username: rspmdb_user password: secret - Create a template web.xml.erb in config directory, e.g.
<!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" "http://java.sun.com/dtd/web-app_2_3.dtd"> <web-app> <% webxml.context_params.each do |k,v| %> <context-param> <param-name><%= k %></param-name> <param-value><%= v %></param-value> </context-param> <% end %> <servlet> <servlet-name>Rails</servlet-name> <servlet-class>org.jruby.rack.RackServlet</servlet-class> </servlet> <servlet-mapping> <servlet-name>Rails</servlet-name> <url-pattern>/*</url-pattern> </servlet-mapping> <filter> <filter-name>RackFilter</filter-name> <filter-class>org.jruby.rack.RackFilter</filter-class> </filter> <filter-mapping> <filter-name>RackFilter</filter-name> <url-pattern>/*</url-pattern> </filter-mapping> <listener> <listener-class><%= webxml.servlet_context_listener || 'org.jruby.rack.RackServletContextListener'%></listener-class> </listener> </web-app>
- Now create war file
jruby -S warble
- Finally copy the war file to webapps directory of standard J2EE web container, in my case Tomcat 6.0
- Now point the browser to your server, e.g. http://localhost:8080/your-controller and voilla.
On final note, I found warble quite slow in generating war file and tomcat takes a while to deploy the war file, so I use WEBrick for development and Tomcat for staging/production.
Rails save the day
September 6th, 2008
In our daily scrum meeting at work last week, we identified one of application that had been crashing often. The application was pretty simple web application written in Java using Spring-MVC, Hibernate, Spring and Apache-commons libraries. However, the developer who released it embedded that application in another much bigger web application in order to save some deployment issues, which can be real pain at amazon. So, I volunteered to address the issue and rewrite the application in JRuby on Rails (2.1). It took me a day to rewrite all application and migrate all data from the old application. Along the way, I also added a couple of some additional features. The original application took a couple of weeks to develop and was consisted of over 6500 lines of code. The ruby version consisted a little less than 800 lines of code. I have been using Ruby on my own since 2004 and started learning Rails in 2005. Though, I started using Rails a bit more on some side projects after I attended Rails conference of 2006. I still use Java for most of the work, and was delighted to show speed of development with Rails at work. Since, the deployment issues are still pain in Rails so I chose JRuby that allows Rails apps to deploy on Tomcat, which we are using for most of the projects. Hopefuly, that will fix the stability issues and save me from pagers as I will be oncall next week.
Implementing HTTP Proxy Service with XSL Transformation
August 27th, 2008
Recently I had to implement a portal that embedded contents from other web applications. I tried to build a framework for easily adding remote applications without any changes to the remote applications. The framework had following design:
- For different functionality and pages we stored feature-link and url in configuration, that configuration basically allowed us to find where to get the HTML/XHTML render page or form.
- However, instead of displaying raw HTML, I first converted it into valid XHTML using Tidy and then transformed such that all form submissions go through our proxy server. As part of the transformation, I added original form URLs, methods and cookies as hidden fields. This allowed me to keep the proxy server without any client state, which was nice for clustering.
- When the form is submitted, the proxy server intercepts it and then makes the request to the remote application and sends back response. The proxy server reads the original target URLs, method types, cookies from the form so that remote application can manage state if it’s using Sessions.
Following are a few simplified classes that I developed for the proxy framework:
Content Transformation
First, I developed HTML to XHTML conversion and transformation to modify forms. Following is the XSLT that I used:1 <xsl:stylesheet 2 xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 3 version="1.0" 4 xmlns:xhtml="http://www.w3.org/1999/xhtml" 5 xmlns="http://www.w3.org/1999/xhtml" 6 exclude-result-prefixes="xhtml"> 7 8 <xsl:param name="callbackHandler"/> 9 <xsl:param name="callbackUser"/> 10 <xsl:param name="callbackState"/> 11 12 <xsl:template match="@* | node()"> 13 <xsl:copy> 14 <xsl:apply-templates select="@* | node()"/> 15 </xsl:copy> 16 </xsl:template> 17 18 <xsl:template match="form"> 19 <xsl:copy> 20 <xsl:apply-templates select="@*"/> 21 <xsl:attribute name="action" xmlns:java="http://xml.apache.org/xslt/java"> 22 <xsl:param name="callbackOriginalActionUrl" select="@action"/> 23 <xsl:text disable-output-escaping="yes">/web/myproxy?myarg=xxx&otherarg=2</xsl:text> 24 <xsl:value-of select="java:com.plexobject.transform.XslContentTransformer.setAction($callbackHandler, string(@action))" /> 25 </xsl:attribute> 26 <xsl:attribute name="method" xmlns:java="http://xml.apache.org/xslt/java"> 27 <xsl:param name="callbackOriginalMethodType" select="@method"/> 28 <xsl:text disable-output-escaping="yes">POST</xsl:text> 29 <xsl:value-of select="java:com.plexobject.transform.XslContentTransformer.setMethod($callbackHandler, string(@method))" /> 30 </xsl:attribute> 31 <xsl:attribute name="id">_Form</xsl:attribute> 32 <xsl:attribute name="name">_Form</xsl:attribute> 33 <input type="hidden" name="_user" value="{$callbackUser}"/> 34 <input type="hidden" name="_originalActionUrl" value="{@action}"/> 35 <input type="hidden" name="_orginalMethodType" value="{@method}"/> 36 <input type="hidden" name="_userState" value="{$callbackState}"/> 37 <xsl:apply-templates select="node()"/> 38 </xsl:copy> 39 </xsl:template> 40 41 42 <xsl:template match="title"/> 43 44 45 </xsl:stylesheet> 46 47 48A few things note
- xsl:param allows passing parameters from the runtime (Java)
- xsl:template is matching for “form” tag and replaces action/method attributes and adds id/name attributes. It then adds a few input hidden fields
- Finally, I am removing title tag
ContentTransformer interface
1 package com.plexobject.transform; 2 3 import java.util.Map; 4 5 public interface ContentTransformer { 6 /** 7 * This method transforms given contents 8 * 9 * @param contents 10 * - input contents 11 * @param properties 12 * - input/output properties for transformation 13 * @return transformed contents 14 * @throws TransformationException 15 * - when error occurs while transforming content. 16 */ 17 public String transform(String contents, Map<String, String> properties) 18 throws TransformationException; 19 } 20 21
ContentTransformer implementation
A few things to note in following implementation:- I use JTidy to convert HTML to XHTML
- I pass some of the parameters to the XSL stylesheet and I also read a few properties back. Though, reading properties back is a bit kludgy but it works.
1 package com.plexobject.transform; 2 3 import java.io.ByteArrayInputStream; 4 import java.io.ByteArrayOutputStream; 5 import java.io.InputStream; 6 import java.util.HashMap; 7 import java.util.Map; 8 9 import javax.xml.transform.Result; 10 import javax.xml.transform.Source; 11 import javax.xml.transform.Transformer; 12 import javax.xml.transform.TransformerException; 13 import javax.xml.transform.TransformerFactory; 14 import javax.xml.transform.stream.StreamResult; 15 import javax.xml.transform.stream.StreamSource; 16 17 import org.w3c.tidy.Tidy; 18 19 public class XslContentTransformer implements ContentTransformer { 20 public static final String ACTION = "form_action"; 21 public static final String METHOD = "form_method"; 22 23 private static final Map<String, Map<String, String>> xslProperties = new HashMap<String, Map<String, String>>(); 24 private volatile Transformer transformer; 25 private final String xslUri; 26 private final boolean useTidy; 27 28 public XslContentTransformer(final String xslUri, final boolean useTidy) { 29 this.xslUri = xslUri; 30 this.useTidy = useTidy; 31 } 32 33 public static final String setAction(final String callbackHandler, 34 final String action) { 35 getPropertiesForCallback(callbackHandler).put(ACTION, action); 36 return ""; 37 } 38 39 public static final String getAction(final String callbackHandler) { 40 return getPropertiesForCallback(callbackHandler).get(ACTION); 41 } 42 43 public static final String setMethod(final String callbackHandler, 44 final String method) { 45 getPropertiesForCallback(callbackHandler).put(METHOD, method); 46 return ""; 47 } 48 49 public static final String getMethod(final String callbackHandler) { 50 return getPropertiesForCallback(callbackHandler).get(METHOD); 51 } 52 53 /** 54 * This method transforms given contents 55 * 56 * @param contents 57 * - input contents 58 * @param properties 59 * - input/output properties for transformation 60 * @return transformed contents 61 * @throws TransformationException 62 * - when error occurs while transforming content. 63 */ 64 public String transform(String contents, Map<String, String> properties) 65 throws TransformationException { 66 initTransformer(); 67 final long started = System.currentTimeMillis(); 68 69 contents = contents.replaceAll("<!–.*?–>", ""); 70 InputStream in = new ByteArrayInputStream(contents.getBytes()); 71 if (useTidy) { 72 in = tidy(in, (int) contents.length()); 73 } 74 75 // 76 final Source xmlSource = new StreamSource(in); 77 final ByteArrayOutputStream out = new ByteArrayOutputStream( 78 (int) contents.length()); 79 80 final Result result = new StreamResult(out); 81 String callbackHandler = properties.get("callbackHandler"); 82 if (callbackHandler == null) { 83 callbackHandler = Thread.currentThread().getName(); 84 } 85 final Map<String, String> props = new HashMap<String, String>(); 86 xslProperties.put(callbackHandler, props); 87 transformer.setParameter("callbackHandler", callbackHandler); 88 for (Map.Entry<String, String> e : properties.entrySet()) { 89 transformer.setParameter(e.getKey(), e.getValue()); 90 } 91 try { 92 transformer.transform(xmlSource, result); 93 } catch (TransformerException e) { 94 throw new TransformationException("Failed to transform " + contents, e); 95 } 96 properties.put(ACTION, getAction(callbackHandler)); 97 properties.put(METHOD, getMethod(callbackHandler)); 98 xslProperties.remove(callbackHandler); 99 return new String(out.toByteArray()); 100 } 101 102 private static final Map<String, String> getPropertiesForCallback( 103 String callbackHandler) { 104 final Map<String, String> props = xslProperties.get(callbackHandler); 105 if (props == null) { 106 throw new NullPointerException( 107 "Failed to find properties for callback " + callbackHandler); 108 } 109 return props; 110 } 111 112 // no synchronization needed, multiple initialization is acceptable 113 private final void initTransformer() { 114 if (transformer == null) { 115 try { 116 TransformerFactory transFact = TransformerFactory.newInstance(); 117 InputStream in = getClass().getResourceAsStream(xslUri); 118 if (in == null) { 119 throw new TransformationException("failed to find xslt " 120 + xslUri); 121 } 122 Source xsltSource = new StreamSource(in); 123 transformer = transFact.newTransformer(xsltSource); 124 } catch (TransformationException e) { 125 throw e; 126 } catch (RuntimeException e) { 127 throw e; 128 } catch (Exception e) { 129 throw new TransformationException( 130 "Failed to initialize XSL transformer", e); 131 } 132 } 133 } 134 135 private final InputStream tidy(InputStream in, int length) { 136 ByteArrayOutputStream out = new ByteArrayOutputStream(length); 137 Tidy converter = new Tidy(); 138 converter.setTidyMark(false); 139 converter.setXmlOut(true); 140 converter.setXmlPi(true); 141 converter.setXmlPIs(true); 142 converter.setNumEntities(true); 143 converter.setDocType("omit"); 144 converter.setUpperCaseTags(false); 145 converter.setUpperCaseAttrs(false); 146 converter.setFixComments(true); 147 converter.parse(in, out); 148 return new ByteArrayInputStream(out.toByteArray()); 149 } 150 } 151 152
Proxy
Following interfaces and classes show how GET/POST requests are proxied:Proxy Interface
1 package com.plexobject.web.proxy; 2 3 import java.io.IOException; 4 import java.util.Map; 5 6 public interface HttpProxy { 7 /** 8 * This method issues a GET or POST request based on method and URI URI specified in the ProxyState 9 * and adds given parameters to the request. 10 * 11 * @param state 12 * - proxy state 13 * @param params 14 * - name/value pairs of parameters that are sent to the get 15 * request 16 */ 17 public ProxyResponse request(ProxyState state, Map<String, String[]> params) 18 throws IOException; 19 } 20 21
Proxy Implementation
Following class implements HttpProxy interface using HTTPClient library:1 package com.plexobject.web.proxy; 2 3 import java.io.IOException; 4 import java.util.ArrayList; 5 import java.util.List; 6 import java.util.Map; 7 8 import org.apache.commons.httpclient.Cookie; 9 import org.apache.commons.httpclient.DefaultHttpMethodRetryHandler; 10 import org.apache.commons.httpclient.HttpClient; 11 import org.apache.commons.httpclient.HttpMethodBase; 12 import org.apache.commons.httpclient.HttpState; 13 import org.apache.commons.httpclient.NameValuePair; 14 import org.apache.commons.httpclient.cookie.CookiePolicy; 15 import org.apache.commons.httpclient.methods.GetMethod; 16 import org.apache.commons.httpclient.methods.PostMethod; 17 import org.apache.commons.httpclient.params.HttpMethodParams; 18 19 import com.plexobject.io.IoUtil; 20 21 public class HttpProxyImpl implements HttpProxy { 22 private static final int CONNECTION_TIMEOUT_MILLIS = 30000; 23 24 /** 25 * This method issues a GET or POST request based on method and URI URI specified in the ProxyState 26 * and adds given parameters to the request. 27 * 28 * @param state 29 * - proxy state 30 * @param params 31 * - name/value pairs of parameters that are sent to the get 32 * request 33 */ 34 public ProxyResponse request(ProxyState state, Map<String, String[]> params) 35 throws IOException { 36 if (state.getMethod() == MethodType.GET) { 37 return get(state, params); 38 } else { 39 return post(state, params); 40 } 41 } 42 43 44 /** 45 * This method issues a GET request on the URI specified in the ProxyState 46 * and adds given parameters to the request. 47 * 48 * @param state 49 * - proxy state 50 * @param params 51 * - name/value pairs of parameters that are sent to the get 52 * request 53 */ 54 private ProxyResponse get(ProxyState state, Map<String, String[]> params) 55 throws IOException { 56 GetMethod method = new GetMethod(state.getUri()); 57 method.setQueryString(toNameValues(params)); 58 return doRequest(state, params, method); 59 } 60 61 /** 62 * This method issues a POST request on the URI specified in the ProxyState 63 * and adds given parameters to the request. 64 * 65 * @param state 66 * - proxy state 67 * @param params 68 * - name/value pairs of parameters that are sent to the POST 69 * request 70 */ 71 private ProxyResponse post(ProxyState state, Map<String, String[]> params) 72 throws IOException { 73 PostMethod method = new PostMethod(state.getUri()); 74 method.setRequestBody(toNameValues(params)); 75 return doRequest(state, params, method); 76 } 77 78 private ProxyResponse doRequest(ProxyState proxyState, 79 Map<String, String[]> params, HttpMethodBase method) 80 throws IOException { 81 long started = System.currentTimeMillis(); 82 HttpClient client = new HttpClient(); 83 client.getHttpConnectionManager().getParams().setConnectionTimeout( 84 CONNECTION_TIMEOUT_MILLIS); 85 client.getParams().setCookiePolicy(CookiePolicy.BROWSER_COMPATIBILITY); 86 method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, 87 new DefaultHttpMethodRetryHandler(3, false)); 88 89 HttpState initialState = new HttpState(); 90 for (Cookie cookie : proxyState.getCookies()) { 91 initialState.addCookie(cookie); 92 } 93 client.setState(initialState); 94 95 try { 96 int statusCode = client.executeMethod(method); 97 String contents = IoUtil.read(method.getResponseBodyAsStream()); 98 // 99 Cookie[] cookies = client.getState().getCookies(); 100 for (Cookie cookie : cookies) { 101 proxyState.addCookie(cookie); 102 } 103 104 return new ProxyResponse(statusCode, contents, proxyState); 105 } catch (RuntimeException e) { 106 throw e; 107 } catch (IOException e) { 108 throw e; 109 } catch (Exception e) { 110 throw new IOException("failed to process request", e); 111 } finally { 112 method.releaseConnection(); 113 } 114 } 115 116 private NameValuePair[] toNameValues(Map<String, String[]> params) { 117 if (params == null || params.size() == 0) { 118 return new NameValuePair[0]; 119 } 120 List<NameValuePair> nvPairs = new ArrayList<NameValuePair>(); 121 for (Map.Entry<String, String[]> e : params.entrySet()) { 122 String[] values = e.getValue(); 123 for (int j = 0; j < values.length; j++) { 124 nvPairs.add(new NameValuePair(e.getKey(), values[j])); 125 } 126 } 127 return (NameValuePair[]) nvPairs.toArray(new NameValuePair[nvPairs 128 .size()]); 129 } 130 } 131 132
ProxyState
Following class maintains URL, cookies, headers, and other information related to web request:1 package com.plexobject.web.proxy; 2 3 import java.io.Serializable; 4 import java.io.UnsupportedEncodingException; 5 import java.net.URLDecoder; 6 import java.net.URLEncoder; 7 import java.util.Collection; 8 import java.util.Date; 9 import java.util.HashMap; 10 import java.util.Map; 11 12 import org.apache.commons.httpclient.Cookie; 13 14 /** 15 * Class: ProxyState 16 * 17 * Description: This class stores state needed to make a proxy request including 18 * method type and cookies. 19 * 20 */ 21 public class ProxyState implements Serializable { 22 private static final long serialVersionUID = 1L; 23 private static final String DATA_DELIMITER = "\n"; 24 private static final String COOKIE_DELIMITER = ";"; 25 private static final String NULL = "null"; 26 27 private String uri; 28 private MethodType method; 29 private Map<String, Cookie> cookies; 30 31 /** 32 * Constructors for ProxyState 33 */ 34 public ProxyState(String uri, String method) { 35 this(uri, MethodType.valueOf(method)); 36 } 37 38 public ProxyState(String uri, MethodType method) { 39 this.uri = uri; 40 this.method = method; 41 this.cookies = new HashMap<String, Cookie>(); 42 } 43 44 /** 45 * @return uri 46 */ 47 public String getUri() { 48 return this.uri; 49 } 50 51 /** 52 * @return method 53 */ 54 public MethodType getMethod() { 55 return this.method; 56 } 57 58 /** 59 * @return cookies 60 */ 61 public Collection<Cookie> getCookies() { 62 return this.cookies.values(); 63 } 64 65 66 /** 67 * @param cookies 68 */ 69 public void addCookies(Collection<Cookie> cookies) { 70 for (Cookie cookie : cookies) { 71 addCookie(cookie); 72 } 73 } 74 75 /** 76 * @param cookie 77 * - to add 78 */ 79 public void addCookie(Cookie cookie) { 80 this.cookies.put(cookie.getName(), cookie); 81 } 82 83 public String getCookieString() { 84 StringBuilder sb = new StringBuilder(512); 85 for (Cookie cookie : cookies.values()) { 86 if (cookie.getDomain() != null) { 87 sb.append(cookie.getDomain()).append(COOKIE_DELIMITER); 88 } else { 89 sb.append(NULL).append(COOKIE_DELIMITER); 90 } 91 sb.append(cookie.getName()).append(COOKIE_DELIMITER).append( 92 cookie.getValue()).append(COOKIE_DELIMITER); 93 94 if (cookie.getPath() != null) { 95 sb.append(cookie.getPath()).append(COOKIE_DELIMITER); 96 } else { 97 sb.append(NULL).append(COOKIE_DELIMITER); 98 } 99 if (cookie.getExpiryDate() != null) { 100 sb.append(String.valueOf(cookie.getExpiryDate().getTime())) 101 .append(COOKIE_DELIMITER); 102 } else { 103 sb.append(NULL).append(COOKIE_DELIMITER); 104 } 105 sb.append(String.valueOf(cookie.getSecure())) 106 .append(DATA_DELIMITER); 107 } 108 return sb.toString(); 109 } 110 111 112 @Override 113 public String toString() { 114 StringBuilder sb = new StringBuilder(512); 115 sb.append(uri.toString()).append(DATA_DELIMITER); 116 sb.append(method.toString()).append(DATA_DELIMITER); 117 sb.append(getCookieString()); 118 return sb.toString(); 119 } 120 121 /** 122 * This method converts proxy state into string based serialized state 123 * 124 * @return string based serialized state 125 */ 126 public String toExternalFormat() { 127 try { 128 return URLEncoder.encode(toString(), "UTF8"); 129 } catch (UnsupportedEncodingException e) { 130 throw new IllegalStateException("failed to encode", e); 131 } 132 } 133 134 /** 135 * This method converts a string based serialized state into the proxy state 136 * 137 * @param ser 138 * - string based serialized state 139 * @return ProxyState 140 * @throws IllegalArgumentException 141 * - if serialized state is null or corrupted. 142 */ 143 public static ProxyState valueOf(String ser) { 144 if (ser == null) 145 throw new IllegalArgumentException("Null serialized object"); 146 String decoded; 147 try { 148 decoded = URLDecoder.decode(ser, "UTF8"); 149 } catch (UnsupportedEncodingException e) { 150 throw new IllegalArgumentException("Unsupported encoding " + ser, e); 151 } 152 String[] lines = decoded.split(DATA_DELIMITER); 153 if (lines.length < 2) 154 throw new IllegalArgumentException( 155 "Insufficient number of tokens in serialized object [" 156 + decoded + "]"); 157 ProxyState state = new ProxyState(lines[0], lines[1]); 158 for (int i = 2; i < lines.length; i++) { 159 String[] cookieFields = lines[i].split(COOKIE_DELIMITER); 160 if (cookieFields.length < 6) 161 throw new IllegalArgumentException( 162 "Insufficient number of tokens 6 in serialized cookies [" 163 + lines[i] + "]/[" + decoded + "]"); 164 String domain = cookieFields[0]; 165 if (NULL.equals(domain)) { 166 domain = null; 167 } 168 String name = cookieFields[1]; 169 String value = cookieFields[2]; 170 String path = cookieFields[3]; 171 if (NULL.equals(path)) { 172 path = null; 173 } 174 Date expires = null; 175 if (!NULL.equals(cookieFields[4])) { 176 expires = new Date(Long.parseLong(cookieFields[4])); 177 } 178 boolean secure = new Boolean(cookieFields[5]).booleanValue(); 179 Cookie cookie = new Cookie(domain, name, value, path, expires, 180 secure); 181 state.addCookie(cookie); 182 } 183 return state; 184 } 185 186 @Override 187 public boolean equals(Object o) { 188 if (this == o) 189 return true; 190 if (!(o instanceof ProxyState)) 191 return false; 192 final ProxyState other = (ProxyState) o; 193 if (uri != null ? !uri.equals(other.uri) : other.uri != null) 194 return false; 195 if (method != null ? !method.equals(other.method) 196 : other.method != null) 197 return false; 198 return true; 199 } 200 201 @Override 202 public int hashCode() { 203 int result; 204 result = (uri != null ? uri.hashCode() : 0); 205 result = 29 * result + (method != null ? method.hashCode() : 0); 206 return result; 207 } 208 } 209 210
ProxyResponse
Following class stores response from the HttpProxy interface:1 package com.plexobject.web.proxy; 2 3 import java.io.Serializable; 4 5 6 /** 7 * Class: ProxyResponse 8 * 9 * Description: This class stores proxy state and response. 10 */ 11 public class ProxyResponse implements Serializable { 12 private static final long serialVersionUID = 1L; 13 private int responseCode; 14 private String contents; 15 private ProxyState state; 16 17 /** 18 * Constructor for ProxyResponse 19 */ 20 public ProxyResponse(int responseCode, String contents, ProxyState state) { 21 this.responseCode = responseCode; 22 this.contents = contents; 23 this.state = state; 24 } 25 26 /** 27 * @return http response code 28 */ 29 public int getResponseCode() { 30 return this.responseCode; 31 } 32 33 /** 34 * @return XHTML contents 35 */ 36 public String getContents() { 37 return this.contents; 38 } 39 40 /** 41 * @return state associated with the proxy web request 42 */ 43 public ProxyState getState() { 44 return this.state; 45 } 46 47 @Override 48 public String toString() { 49 return this.responseCode + "\n" + this.state + "\n" + this.contents; 50 } 51 } 52 53
MethodType
Following class defines enum for http method types:1 package com.plexobject.web.proxy; 2 3 /** 4 * Class: MethodType 5 * 6 * Description: Defines supported method types for proxy request. 7 * 8 */ 9 public enum MethodType { 10 GET, POST; 11 } 12 13
Service Example
Following classes show how above HTTPProxy and ContentTransfomer interfaces can be used with Servlet/Portlet APIs:ProxyService Interface
1 package com.plexobject.web.service; 2 import javax.servlet.http.*; 3 import java.io.*; 4 5 public interface ProxyService { 6 public void render(HttpServletRequest request, HttpServletResponse response) throws IOException ; 7 public void submit(HttpServletRequest request, HttpServletResponse response) throws IOException ; 8 } 9 0
ProxyService Implementation
1 package com.plexobject.web.service; 2 import com.plexobject.web.proxy.*; 3 import com.plexobject.transform.ContentTransformer; 4 import javax.servlet.http.*; 5 import java.io.*; 6 import java.util.*; 7 8 9 public class ProxyServiceImpl implements ProxyService { 10 private HttpProxy httpProxy; 11 private ContentTransformer contentTransformer; 12 public ProxyServiceImpl(HttpProxy httpProxy, ContentTransformer contentTransformer) { 13 this.httpProxy = httpProxy; 14 this.contentTransformer = contentTransformer; 15 } 16 17 public void render(HttpServletRequest request, HttpServletResponse response) throws IOException { 18 String url = "http://plexrails.plexobject.com/guest_book/sign"; 19 ProxyState state = new ProxyState(url, MethodType.GET); 20 String inputXhtml = httpProxy.request(state, null).getContents(); 21 Map<String, String> properties = new HashMap<String, String>(); 22 properties.put("callbackState", state.toExternalFormat()); 23 String transformedXhtml = contentTransformer.transform(inputXhtml, properties); 24 response.getWriter().println(transformedXhtml); 25 } 26 27 public void submit(HttpServletRequest request, HttpServletResponse response) throws IOException { 28 String originalActionUrl = request.getParameter("originalActionUrl"); 29 String orginalMethodType = request.getParameter("orginalMethodType"); 30 ProxyState userState = ProxyState.valueOf(request.getParameter("userState")); 31 Map<String, String[]> params = request.getParameterMap(); 32 ProxyState state = new ProxyState(originalActionUrl, orginalMethodType); 33 state.addCookies(userState.getCookies()); 34 ProxyResponse proxyResponse = httpProxy.request(state, params); 35 response.getWriter().println(proxyResponse.getContents()); 36 } 37 } 38 39
Download Code
You can download above code from here.Acknowledgement
I would like to thank the folks at XSLT forum of Programmer-to-Programmer (http://p2p.wrox.com/forum.asp?FORUM_ID=79) for answering my XSLT questions.U.S. Killed 90, Including 60 Children, in Afghan Village, U.N. Finds
August 26th, 2008
U.S. Killed 90, Including 60 Children, in Afghan Village, U.N. Finds
5 Afghan children killed in raids
U.S. soldiers say they executed Iraqis on riverbank
15 Pakistani Civilians Killed by US-Led Forces
U.S. troops kill 20 in Pakistan attack
U.S. drones kill 13 in missile attack in Pakistan
Film Backs Afghans. Claims of US Killings [of 90+ civilians]
US Attack on School Kills 23 in North Waziristan
Benefits of REST based services
August 14th, 2008
I saw Damien Katz blog on REST, I just don’t get it and I was a bit surprised that he doesn’t get REST especially since he wrote CouchDB based on REST. Though, I admit there are a lot of bad examples of REST services that use REST sort of like RPC over POST, but resource oriented services can be quite simple and powerful. REST principles are building blocks for the web and it has proven to be quite scalable and efficient. I have been developing REST based services for a number of years, in some ways before I learned about Roy Fieldings’ thesis and REST principles. Back in 90s, I worked on building traffic sites and used CORBA to subscribe and publish traffic events. We also published that data on the website, but soon we found a number of people were scraping the website so I wrote a simple XML over HTTP service to download the data that other groups can use. I have found following benefits when using REST based services:
- separating reads from writes. I have worked on large ecommerce and travel website and one of the lesson is to keep your read/query services separate from your transactional services. REST APIs define separate operations for reads and write/updates.
- caching: you can find tons of off the shelf solutions for caching GET requests including hardware solutions. There are tons of features like ETags and cache headers that provide this feature.
- compression: Since REST uses HTTP, you can use compression such as gzip. This can improve the performance of the services.
- idempotency: GET, PUT, DELETE and HEAD are idempotent, which means if designed correctly the request can be retried without any worries about side effects. POST on the other hand is not idempotent and may have side effects.
- bookmarking: GET requests can be easily bookmarked. It is important not to use GET to change state of application.
- security: Though, security has been weakest area of REST as compared to SOAP, but HTTPS and simple authentication surfice. Though, there are better standards like oauth.
- big response size: REST/HTTP is the only service platform that I have seen supports gigabytes of responses. I have done a lot of CORBA based services in 90s, EJBs/SOAP in early 2000s and messaging based services for over ten years. None of those platforms support large size responses.
- simplicity: I find this is the biggest reason for using REST. I can use browser to call GET based requests and write client in any language.
- resources: REST response can include URIs for other APIs and client can change state through these resources. You can use XHTML to embed all these resources that can be easily tested with browsers.
- No need for additional jars: When I used CORBA, EJBs, RMI or JINI, I always had to put client/skeleton jar files. Having worked in large companies where I had to import dozens of these jar files became maintenance problem. With REST, I can simply call the service without importing anything.
- Error codes: HTTP comes with a number of response codes for real life services including thrashing requests such as server busy (503).
- Meta data: As opposed to CORBA, JINI, RMI services, I can pass meta data easily as HTTP supports predefined and user-defined headers. These headers can include information on authentication, quality of service, timeout or other context related data. Occasionally, I add Map<String,String> to APIs when I use Java based services, but it polutes pure APIs.
Love and Hate with Java
July 31st, 2008
For past few years, bashing Java has been really popular though some of the criticism has merits. But in general, due to its popularity, Java has become “the man” who tries to bring everyone down. There are millions of programmers who work for the “Java the man”. I saw recent post from NYU professor, who called Java-savvy college grads to tomorrow’s pizza delivery man. I know Joel Spolsky often mentions teaching C in unversities to help understand pointers and memory management. I agree with notion of teachings multiple languages in universities so that graduates have wide breadth of understanding with differant programming paradigms. I started learning programming back in 80’s on Atari and learned BASIC. I then moved to PC and learned GW-BASIC and then learned C, FORTRAN, Assembler, COBOL, Pascal, C++ in college. I also learned Lisp, Prolog, Perl on my own. In late 80’s and early 90s, I also learned DBase III, RPG and SAS, which was called 4th generation language. Similarly, C, FOTRAN, Pascal, etc. were called third languages, assembly languages were second generation languages. I learned Java in ‘95 when it came out and found it to be much easier to program than C/C++. I also learned Python, Ruby and Erlang for past few years and have been learning Haskell and Scala these days.
For most part, Java has been my primary language with some use of C++, Perl, Ruby, Python/Jython (and Erlang on my own). Though I wish I could use more Erlang but I don’t have same experience with Erlang as I have with Java. Over time, Java managed to take a lot of C/C++ share of the market. Also, Java has managed to buid large ecosystem with open source and commercial suites of libraries and frameworks. I often hear that Java is so enterprisy and popular in large companies, but truth is that Java has proven itself to be reliable language. Steve Yeggie also mentioned in his blog how Google primarily uses Java, C++, Python and Javascript.
I like the polyglot environment, where I write performance critical c