Shahzad Bhatti Welcome to my ramblings and rants!

I started at Amazon last year, but didn’t actually got chance to work with them until recently when we had to integrate with Amazon Ecommerce Service (ECS).

Amazon Web Services come in two flavors: REST and SOAP. According to inside sources about 70% use REST. I also found that REST interface was more reliable and simple. Though, I will describe both techniques here:

Getting Access ID

First, visit http://www.amazon.com/gp/browse.html?node=3435361 to get your own access key.

RTFM

I will describe ECS here and it comes with 450 pages of documentation, though most of it just describes URLs and input/output fields. You can find documentation and sample code at http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=59. I also found Eric Giguerre’s tutorial on AWS very useful.

Other interesting links include: blog site for updates on AWS, a Forum #1, Forum #2 and FAQ.

Services

Inside ECS, you will find following services:

• ItemSearch
• BrowseNodeLookup
• CustomerContentLookup
• ItemLookup
• ListLookup
• SellerLookup
• SellerListingLookup
• SimilarityLookup
• TransactionLookup

REST Approach

The rest approach is pretty simple, in fact you can simply type in following
URL to your browser (with your access key) and will see the results (in XML)
right away:

Finding images for Harry Potter Video:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=[your-key] &Operation=ItemSearch&SearchIndex=Video&Keywords=potter%20harry&ResponseGroup=Images

Finding images for Harry Potter Video:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=[your-key] &Operation=ItemSearch&SearchIndex=Books&Keywords=rails&ResponseGroup=Request,Small

Finding ASINS by keywords:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=your-key &Operation=ItemSearch&SearchIndex=Books&Keywords=rails&ResponseGroup=ItemIds

Find DVD cover art:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=[ID]&Operation=ItemSearch &SearchIndex=DVD &Keywords=potter%20harry &ResponseGroup=Images

Find CDs that contain music by Beethoven:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Music &ResponseGroup=Small,Tracks &Composer=Beethoven

Find by Vendor:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Apparel &ResponseGroup=Large,Variations &MerchantId=[ID] &ItemPage=1
http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Apparel &ResponseGroup=Large,Variations &MerchantId=[ID] &ItemPage=2

Find all new products on Amazon that cost less than $1:00

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Blended &ResponseGroup=Small,Offers &MerchantId=All &MaximumPrice=99

Find all new/old products on Amazon that cost less than $1:00

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Blended &ResponseGroup=Small,Offers &MerchantId=All &MaximumPrice=99 &Condition=All

Find used Barbie dolls

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Toys &Title=Barbie &Manufacturer=Mattel &Condition=All &ItemPage=1
or
http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Toys &Title=Barbie &Manufacturer=Mattel &Condition=All &ItemPage=2
Scenario #6:

Search for Godiva dark

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=GourmetFood &Keywords=dark%20chocolate
&Manufacturer=Godiva

Search for purple products

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Blended &Keywords=purple

Find competitive pricing

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=ABC1,ABC2,P12345 &ResponseGroup=Request,Small,Offers &Condition=All &MerchantId=All

Find a toy by UPC

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup Amazon E-Commerce Service Developer Guide 46 &IdType=UPC &ItemId=[UPC] &SearchIndex=Toys &ResponseGroup=Request,Small,Offers &Condition=Collectible &MerchantId=All

Find a particular gas gril

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &IdType=UPC &ItemId=[UPC] &SearchIndex=OutdoorLiving &DeliveryMethod=ISUP &ISPUPostalCode=12345 &ResponseGroup=Request,Small,Offers &Condition=All &MerchantId=All

Compare pricing for different size/color

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &IdType=SKU &ItemId=[SKU1,SKU2,SKU3] &SearchIndex=Apparel &ResponseGroup=Request,Small,Offers,Variations &MerchantId=[ID]

Find a book

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=[ASIN] &SearchIndex=Books &ResponseGroup=Request,ItemAttributes,Offers

Find by ASIN

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=[ASIN]

Find reviews for bestsellers

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] Amazon E-Commerce Service Developer Guide 47 &Operation=ItemLookup &ItemId=[ASIN] &SearchIndex=Books &ResponseGroup=Request,EditorialReview,Reviews,SalesRank

See additional customer reviews

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=[ASIN] &SearchIndex=Books &ResponseGroup=Request,Reviews &ReviewPage=2 http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=[ASIN] &SearchIndex=Books &ResponseGroup=Request,Reviews &ReviewPage=3

Lookup samples and notes

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=SimilarityLookup &ItemId=ABC1 &ResponseGroup=Request,Small,Offers &Condition=All &MerchantId=All

Lookup similar group of products

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=SimilarityLookup &ItemId=ABC1,ABC2,ABC3 &ResponseGroup=Request,Small,Offers &Condition=All &MerchantId=All

The REST base URLs are:

• United States (US): http://webservices.amazon.com/onca/xml?Service=AWSECommerceService
• United Kingdom (UK): http://webservices.amazon.co.uk/onca/xml?Service=AWSECommerceService
• Germany (DE): http://webservices.amazon.de/onca/xml?Service=AWSECommerceService
• Japan (JP): http://webservices.amazon.co.jp/onca/xml?Service=AWSECommerceService
• Canada (CA): http://webservices.amazon.ca/onca/xml?Service=AWSECommerceService
• France (FR): http://webservices.amazon.fr/onca/xml?Service=AWSECommerceService

REST request is pretty simple, in Java all you need is to create URL and add all service arguments as form arguments. For example,

 1
 2     String createUrl(Map<String,String> map) {
 3         StringBuilder b = new StringBuilder(urlTarget);
 4         b.append("&AWSAccessKeyId=");
 5         b.append(subscriptionId);
 6         if (associateTag != null){
 7             b.append("&AssociateTag=");
 8             b.append(associateTag);
 9         }
10
11         for (Map.Entry<string,> entry : map.entrySet()) {
12             b.append('&');
13             b.append(entry.getKey());
14             b.append('=');
15             try {
16                 b.append(URLEncoder.encode(entry.getValue(), "UTF8"));
17             } catch (UnsupportedEncodingException e) {
18                 throw new RuntimeException("Failed to encode '" + entry.getValue() + "'", e);
19             }
20         }
21         return b.toString();
22     }
23     Map map ... setup
24     URL u = new URL(createurl(map));
25     URLConnection connection = u.openConnection();
26     InputStream in = connection.getInputStream();
27
28 }
29
30

// create DOM and parse XML here

SOAP interface:

The SOAP based interaction is more complicated. First thing you need is to download Java client and you can also find
javadocs.

The SOAP endpoints:

• US: http://webservices.amazon.com/onca/soap?Service=AWSECommerceService
• UK: http://webservices.amazon.co.uk/onca/soap?Service=AWSECommerceService
• DE: http://webservices.amazon.de/onca/soap?Service=AWSECommerceService
• JP: http://webservices.amazon.co.jp/onca/soap?Service=AWSECommerceService
• CA: http://webservices.amazon.ca/onca/soap?Service=AWSECommerceService
• FR: http://webservices.amazon.fr/onca/soap?Service=AWSECommerceService

The WSDL locations for each SOAP endpoint are:

• US: http://webservices.amazon.com/AWSECommerceService/AWSECommerceService.wsdl
• UK: http://webservices.amazon.com/AWSECommerceService/UK/AWSECommerceService.wsdl
• DE: http://webservices.amazon.com/AWSECommerceService/DE/AWSECommerceService.wsdl
• JP: http://webservices.amazon.com/AWSECommerceService/JP/AWSECommerceService.wsdl
• CA: http://webservices.amazon.com/AWSECommerceService/CA/AWSECommerceService.wsdl
• FR: http://webservices.amazon.com/AWSECommerceService/FR/AWSECommerceService.wsdl

Finally, the XML schemas are available from these locations:

• US: http://webservices.amazon.com/AWSECommerceService/AWSECommerceService.xsd
• UK: http://webservices.amazon.com/AWSECommerceService/UK/AWSECommerceService.xsd
• DE: http://webservices.amazon.com/AWSECommerceService/DE/AWSECommerceService.xsd
• JP: http://webservices.amazon.com/AWSECommerceService/JP/AWSECommerceService.xsd
• CA: http://webservices.amazon.com/AWSECommerceService/CA/AWSECommerceService.xsd
• FR: http://webservices.amazon.com/AWSECommerceService/FR/AWSECommerceService.xsd

Show me the code

Here is the complete code for both REST and SOAP, though you will need to add following jars to the CLASSPATH:
junit.jar

• AWS4JavaSample.jar
• axis.jar
• jaxrpc.jar
• commons-logging.jar
• commons-discovery.jar
• saaj.jar
• jdom.jar

  1 import junit.framework.TestCase;
  2 import java.util.Iterator;
  3 import java.util.List;
  4 import java.util.ArrayList;
  5 import java.util.Map;
  6 import java.util.HashMap;
  7 import java.net.URL;
  8 import java.net.URLConnection;
  9 import java.net.MalformedURLException;
 10 import java.net.URLEncoder;
 11 import java.io.UnsupportedEncodingException;
 12 import java.io.InputStream;
 13 import java.io.IOException;
 14
 15 import java.lang.reflect.Constructor;
 16 import java.lang.reflect.Method;
 17 import java.lang.reflect.InvocationTargetException;
 18
 19 import java.rmi.RemoteException;
 20 import javax.xml.rpc.ServiceException;
 21
 22 import com.amazon.xml.AWSECommerceService.ItemSearchRequest;
 23 import com.amazon.xml.AWSECommerceService._ItemSearchResponse;
 24 import com.amazon.xml.AWSECommerceService._ItemSearch;
 25 import com.amazon.xml.AWSECommerceService._Items;
 26 import com.amazon.xml.AWSECommerceService._Item;
 27 import com.amazon.xml.AWSECommerceService.AWSECommerceServicePortType;
 28 import com.amazon.xml.AWSECommerceService.AWSECommerceService;
 29 import com.amazon.xml.AWSECommerceService.AWSECommerceServiceLocator;
 30 import org.jdom.*;
 31 import org.jdom.input.*;
 32 import org.jdom.output.*;
 33
 34 public class EcsTest extends TestCase {
 35     protected void setUp() throws Exception {
 36     }
 37
 38
 39     private SoapRequest setupSoapRequest(String operation) throws MalformedURLException, ServiceException {
 40         SoapRequest request = new SoapRequest("US", "xml", "YourKey", null);
 41         request.put("ItemPage", "1");
 42         request.put("SearchIndex", "Books");
 43         request.put("Keywords", "rails");
 44         request.put("ResponseGroup", "ItemIds");
 45         request.put("Sort", "salesrank");
 46         request.put("ResponseGroup", "SalesRank,Small" );
 47         return request;
 48     }
 49
 50
 51     private RestRequest setupRestRequest(String operation) {
 52         RestRequest request = new RestRequest("US", "soap", "YourKey", null);
 53         //request.put("AWSAccessKeyId", "YourKey");
 54         //request.put("Author", author);
 55         request.put("ItemPage", "1");
 56         request.put("Operation", operation);
 57         request.put("SearchIndex", "Books");
 58         request.put("Keywords", "rails");
 59         request.put("ResponseGroup", "ItemIds");
 60         request.put("Sort", "salesrank");
 61         //request.put("ResponseGroup", "SalesRank,Small" );
 62         request.put("ResponseGroup", "ItemIds" );
 63         return request;
 64     }
 65     //
 66     public void testGetAsinsBySoap() throws Exception {
 67         SoapRequest request = setupSoapRequest("ItemSearch");
 68         _Items[] items = request.invoke();
 69         for (int i=0; i<items.length; i++) {
 70             _Item[] item = items[i].getItem();
 71             for (int j=0; i<item.length; j++) {
 72                 assertTrue(item[j].getASIN() != null);
 73             }
 74         }
 75     }
 76
 77
 78     public void testGetAsinsByRest() throws Exception {
 79         RestRequest request = setupRestRequest("ItemSearch");
 80         URL u = new URL(request.toString());
 81         URLConnection connection = u.openConnection();
 82         InputStream in = connection.getInputStream();
 83         List<String> asins = parseAsins(in);
 84         for (String asin : asins) {
 85             System.out.println("asin: " + asin);
 86         }
 87     }
 88     private List<String> parseAsins(InputStream in) throws IOException, JDOMException {
 89         SAXBuilder builder = new SAXBuilder();
 90         Document doc = builder.build(in);
 91
 92         Format format = Format.getPrettyFormat();
 93         XMLOutputter out = new XMLOutputter(format);
 94         out.output(doc, System.out);
 95
 96         Element root = doc.getRootElement();
 97         Namespace ns = root.getNamespace();
 98         Element items = root.getChild( "Items", ns );
 99         Element request = items.getChild( "Request", ns );
100
101         // First make sure the response is valid
102
103         String isValid = request.getChild("IsValid", ns).getTextTrim();
104         if (!isValid.equals("True")){
105             throw new RuntimeException("Invalid response " + isValid);
106         }
107
108         // Now make sure there are no errors -- would be a good
109         // idea to collect and print them
110
111         Element errors = request.getChild("Errors", ns);
112
113         if( errors != null ){
114             throw new RuntimeException("One or more errors in the response " + errors);
115         }
116         int max = Integer.parseInt( items.getChild( "TotalResults", ns ).getTextTrim() );
117         List itemList = items.getChildren("Item", ns);
118         List<String> asins = new ArrayList<String>();
119         Iterator it = itemList.iterator();
120         while( it.hasNext()) {
121             Element item = (Element) it.next();
122             Element asinElement = item.getChild("ASIN", ns);
123             asins.add(asinElement.getTextTrim());
124             Element rankElement = item.getChild("SalesRank", ns);
125             if ( rankElement != null ) {
126                 int rank = extractRank( rankElement );
127                 String title = item.getChild( "ItemAttributes", ns ).getChild( "Title", ns ).getTextTrim();
128
129                 System.out.println( "Rank=" + rank + " Title=" + title );
130
131             }
132         }
133         return asins;
134     }
135
136     private int extractRank( Element e ) throws NumberFormatException {
137         String       rank = e.getTextTrim();
138         int          len = rank.length();
139         StringBuffer b = new StringBuffer( len );
140
141         for( int i = 0; i < len; ++i ){
142             char ch = rank.charAt( i );
143             if( ch == '.' || ch == ',' || ch == ' ' ) continue;
144             b.append( ch );
145         }
146
147         return Integer.parseInt( b.toString() );
148     }
149
150
151
152     public static void main(String[] args) {
153         junit.textui.TestRunner.run(EcsTest.class);
154     }
155 }
156
157 class SoapRequest extends HashMap<String, String> {
158     final String locale;
159     final String protocol;
160     final String subscriptionId;
161     final String associateTag;
162     final ItemSearchRequest itemSearchRequest;
163     final AWSECommerceServicePortType port;
164     SoapRequest(String locale, String protocol, String subscriptionId, String associateTag) throws MalformedURLException, ServiceException {
165         this.locale = locale;
166         this.protocol = protocol;
167         this.subscriptionId = subscriptionId;
168         this.associateTag = associateTag;
169         String urlTarget = new EcsUtils().getUrl("US", "soap");
170         this.itemSearchRequest = new ItemSearchRequest();
171         AWSECommerceService apd = new AWSECommerceServiceLocator();
172         this.port = apd.getAWSECommerceServicePort(new URL(urlTarget));
173     }
174     private void set(String key, Object value) throws NoSuchMethodException, IllegalAccessException, InvocationTargetException, InstantiationException {
175         if (value == null) return;
176         String name = "set" + key;
177         Method[] methods = itemSearchRequest.getClass().getMethods();
178         for (int i=0; i<methods.length; i++) {
179             if (methods[i].getName().equals(name) && methods[i].getParameterTypes().length == 1) {
180                 Class arg = methods[i].getParameterTypes()[0];
181                 if (arg == String[].class) {
182                     value = value.toString().split(",");
183                 } else if (!arg.isAssignableFrom(value.getClass())) {
184                     Constructor ctor = null;
185                     try {
186                         ctor = arg.getConstructor(value.getClass());
187                     } catch (NoSuchMethodException e) {
188                     }
189                     if (ctor != null) {
190                         value = ctor.newInstance(value);
191                     } else {
192                         throw new IllegalArgumentException("Failed to invoke setter for '" + name + "' with value '" + value + "' with signature " + methods[i] + " in " + itemSearchRequest.getClass().getName());
193                     }
194                 }
195                 methods[i].invoke(itemSearchRequest, value);
196                 return;
197             }
198         }
199         throw new IllegalArgumentException("Failed to find setter for '" + name + "' with value '" + value + "' in " + itemSearchRequest.getClass().getName());
200     }
201     //
202     public _Items[] invoke() throws NoSuchMethodException, IllegalAccessException, InvocationTargetException, InstantiationException, RemoteException {
203         for (Map.Entry<String, String> entry : this.entrySet()) {
204             if (entry.getValue() != null) {
205                 set(entry.getKey(), entry.getValue());
206             }
207         }
208         _ItemSearch itemSearchBody = new _ItemSearch();
209         itemSearchBody.setSubscriptionId(subscriptionId);
210         if (associateTag != null){
211             itemSearchBody.setAssociateTag(associateTag);
212         }
213         //itemSearchBody.setValidate(...);
214         itemSearchBody.setRequest(new ItemSearchRequest[] {itemSearchRequest});
215         return port.itemSearch(itemSearchBody).getItems();
216     }
217 }
218
219
220 class RestRequest extends HashMap<String, String> {
221     final String locale;
222     final String protocol;
223     final String subscriptionId;
224     final String associateTag;
225     final String urlTarget;
226     RestRequest(String locale, String protocol, String subscriptionId, String associateTag) {
227         this.locale = locale;
228         this.protocol = protocol;
229         this.subscriptionId = subscriptionId;
230         this.associateTag = associateTag;
231         this.urlTarget = new EcsUtils().getUrl("US", "xml");
232     }
233     public String toString() {
234         StringBuilder b = new StringBuilder(urlTarget);
235         //b.append("&SubscriptionId=");
236         b.append("&AWSAccessKeyId=");
237         b.append(subscriptionId);
238         if (associateTag != null){
239             b.append("&AssociateTag=");
240             b.append(associateTag);
241         }
242
243         for (Map.Entry<String, String> entry : this.entrySet()) {
244             b.append('&');
245             b.append(entry.getKey());
246             b.append('=');
247             try {
248                 b.append(URLEncoder.encode(entry.getValue(), "UTF8"));
249             } catch (UnsupportedEncodingException e) {
250                 throw new RuntimeException("Failed to encode '" + entry.getValue() + "'", e);
251             }
252         }
253         return b.toString();
254     }
255 }
256
257
258 class UrlUtils {
259     private static final String BASE_URL = "http://webservices.amazon.";
260     private static final String[] LOCALES = new String[] {"US", "UK", "DE", "JP", "CA", "FR"};
261     private static final String[] DOMAIN_SUFFIX = new String[] {"com", "co.uk", "de", "co.jp", "ca", "fr"};
262     private final Map<String, String> restUrls;
263     private final Map<String, String> soapUrls;
264
265     UrlUtils() {
266         soapUrls =createUrls("soap");
267         restUrls = createUrls("xml");
268     }
269     public String getUrl(String locale, String protocol) {
270         if ("xml".equalsIgnoreCase(protocol)) {
271             return restUrls.get(locale);
272         } else {
273             return soapUrls.get(locale);
274         }
275     }
276     private static Map<String, String> createUrls(String protocol) {
277         Map<String, String> map = new HashMap<String, String>();
278         for (int i=0; i<LOCALES.length; i++) {
279             map.put(LOCALES[i], BASE_URL + DOMAIN_SUFFIX[i] + "/onca/" +
280                     protocol + "?Service=AWSECommerceService");
281         }
282         return map;
283     }
284 }
285
286
287
288
289
290
291

I have been doing Java programming for over ten years and when I work with Rails, I miss some of the conventions Java has. One of those convention is to use resource bundles for all user messages. Java provides ResourceBundle for that, but other MVC frameworks such as Struts and Spring MVC provides nice support for that. One of the thing that I like in Ruby is expressive and succint syntax, so it took me about an hour to whip up the functionality I needed. I wanted to use the resources file using method invocation or hash syntax, e.g.

MessageResource.my_message

Optionally I could pass in arguments, e.g. if I had message like:

errors_range: "{0} is not in the range {1} through {2}."

which takes in three arguments then I can invoke:

MessageResource.errors_range('credit-card', 10, 20, 30)

In addition, I could use hash access syntax, e.g.

MessageResource[:errors_range, 'credit-card', 10, 20, 30]


The other thing I needed was that if I specify locale then it
should find the message from that locale, e.g.

MessageResource.es.errors_range('credit-card', 10, 20, 30)


Finally, I wanted this to be robust so that if the key-code for the
message or locale is not found then it returns key-code instead of
throwing exception or passing nil back (this can waste a lot of time
when developing web applications.)

So without further delay, here is the code:

 1 require 'yaml'
 2 require 'resource_bundle.rb'
 3 
 4 class MessageResource
 5   @@bundles = {}
 6   def self.[](code, *args)
 7     self.populate unless @@bundles.size > 0
 8     key = code.to_s
 9     if key.length == 2 and @@bundles.has_key? key
10        bundle = @@bundles[key] || "??#{code}-#{args}??"
11     else
12        bundle = @@bundles['en']
13        if key.length == 2
14           bundle
15        else
16           bundle[key, *args] || "??#{code}??"
17        end
18      end
19   end
20 
21   ###################
22   # Populates all resource bundles and stores them by locale.
23   ###################
24   def self.populate
25     unless @@bundles.size > 0
26     files = Dir.glob(File.dirname(__FILE__) + "/../../../config/messages*")
27     files.each do |f|
28       locale = 'en'
29       locale = $&.slice(1,2) if f =~ /_...yml/
30       begin
31         messages = YAML.load_file(f)
32       rescue ArgumentError => e
33         raise ArgumentError, "Invalid resource file #{f} -- #{e}", caller
34       end
35       raise ArgumentError, "Invalid resource file #{f}", caller if !messages
36       @@bundles[locale] = ResourceBundle.new(locale, messages)
37     end
38   end
39   end
40 
41   ###################
42   # Defines method_missing for class that simply invokes array operator.
43   ###################
44   def self.method_missing(sym, *args)
45     self[sym, *args]
46   end
47 
48   protected
49   def initialize;end
50 end
51 
52 #
53 #################################################################
54 class ResourceBundle
55   def initialize(locale, messages)
56     @locale = locale
57     @messages = messages
58   end
59   #
60   ###################
61   # Overloads brackets to access messages by keys.
62   ###################
63   def [](code, *args)
64     message = @messages[code.to_s]
65     if message && args
66     args.each_with_index {|arg, i|
67       message.gsub!(/{#{i}}/, arg.to_s)
68     }
69   end
70   message || "???#{code}-#{args.join(',')}???"
71   end
72   ###################
73   # Defines method_missing for this instance that simply invokes array operator.
74   ###################
75   def method_missing(key, *args)
76     self[key, *args]
77   end
78 end

Note, I am using YAML syntax to store messages, here is a sample file:
—
notice_password_changed_wrong: “Your old password did not match, please try again.”
notice_password_mismatch: “Your new password did not match confirmed password, please try again.”
notice_password_changed: “Password was changed successfully.”
notice_password_reset_failed: “Password could not be reset, please try again.”
notice_password_reset_success: “Password was reset successfully.”

Moving my Blog from Blojsom to WordPress

I recently changed my ISP from my old friend Aligen (courtesy of Farrad Ali for providing free access since ’98) to HostMonster. So, instead of running my own Tomcat now I am relying on PHP and Rails. The next thing was how do I move my blogs and preserve the dates. Blojsom was simple file based software, but WordPress uses mysql. So I wrote a simple ruby script to convert it.

The first thing I did was to download dbi from

http://rubyforge.org/projects/ruby-dbi/

I then uncompressed it

		tar xzf dbi-0.1.1.tar.gz

Then

cd ~/ruby-dbi

Since, I don’t have root access on my host, I could not install it to the /usr/bin directory. So I created my own ruby directory

mkdir ~/ruby

and then ran config with my own bin directory as

ruby setup.rb config –bin-dir=~/bin –with=dbi,dbd_mysql –rb-dir=~/ruby –so-dir=~/ruby

Next I ran
ruby setup.rb setup

and then

ruby setup.rb install

Now then fun part, following is a ruby script that reads my flat files and inserts them into wordpress database:

 1 #!/usr/bin/ruby
 2 require 'dbi'
 3 
 4 #
 5 ### import blogs from old blog directory to wordpress
 6 #
 7 class ImportBlogs
 8   def initialize(webapp_dir)
 9     @webapp_dir = webapp_dir
10   end
11 
12   def delete_all
13     DBI.connect('DBI:Mysql:weblog', 'weblog', '*****') do | dbh |
14       dbh.do('delete from wp_posts where id > 2')
15     end
16   end
17 
18   def add_all
19     files = Dir.glob("#{@webapp_dir}/*").delete_if { |f| File.directory?(f) }
20     DBI.connect('DBI:Mysql:weblog', 'weblog', '*****') do | dbh |
21       id = 3
22       post_author = 1
23       sql = "insert into wp_posts(post_author, post_date, post_date_gmt, post_content, post_title, post_category, post_excerpt, post_name, post_modified, post_modified_gmt, guid) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
24       dbh.prepare(sql) do | sth |
25         files.each do |f|
26           lines = nil
27           File.open(f, "r") do |file|
28             lines = file.readlines
29           end
30           post_content = lines.join(' ')
31           post_title = lines[0]
32           post_excerpt = post_content.slice(0,255)
33           post_date = post_date_gmt = post_modified = post_modified_gmt = File.new(f).mtime
34           post_category = 3
35           post_name = File.basename(f)
36           guid = "http://weblog.plexobject.com/?p=#{id}"
37           puts "Adding #{f} mtime #{post_date}"
38           sth.execute(post_author, post_date, post_date_gmt, post_content, post_title, post_category, post_excerpt, post_name, post_modified, post_modified_gmt, guid)
39           id += 1
40         end
41       end
42     end
43   end
44 end
45 
46 ib = ImportBlogs.new('~/webapps/blojsom/computing')
47 ib.delete_all
48 ib.add_all
49 
50 

Finally, I ran it as follows:
ruby -I /usr/lib/ruby/gems/1.8/gems/mysql-2.7 -I /usr/lib/ruby/gems/1.8/gems/mysql-2.7/lib -I ~/ruby -I ~/ruby/DBD -I ~/ruby/dbi -I ~/ruby/DBD/Mysql import_blogs.rb
Voilla, I got everything as expected.

Popular Blog Entries

• Agile Methodologies Under the Hood
• Development in large IT Shops
• Enums and Lookups in Ruby and Java
• Integrating Ruby with Java using ActiveMQ and Stomp
• RailsConf 2006 Day 3
• RailsConf 2006 Day 2
• RailsConf 2006 Day 1
• Integrating ActiveMQ with JBoss
• Agile Purists
• Implementing Const in Java
• Brainstorm Chicago April 19-20
• Message-Driven-Pojos with ActiveMQ
• Sudoku Solver in Ruby
• Nexus for next generation languages
• Scripting Flame War
• Log locally and query globally
  
• Responsibility vs Accountability
• Web 2.0
• Bohrbug Vs Heisenbug
• Keeping the truck number low
• How overwork leads to dumb workers
• Best Practices 2004 (East)
• What is right methodology?
• Offshoring
• Design by Committee
• Software Development Profession is Doomed?
• What’s in the Title
• Software Rewrites
• OO Summary

Courage
One of core practices of agile methodologies such as XP is courage. In
almost every company I have worked with, you have to deal with endless
pressure from managers or customers to meet ridiculous deadlines or
expectations. Courage is standing up for such nonsense. I heard similar
message from Ken Schwaber. However, in most places people tell their managers
or customers what they want to listen. And if you try to speak up against
impossible timelines, then your loyality or commitment is questioned. Also,
in order to meet these deadlines, you end up sacrificing quality and
while code becomes unmanageable. I have seen countless examples, where
a software product starts small, probably written by average joe programmer
and the software becomes a hit and more features are piled up. Soon, the
whole systems becomes unmaintanable and adding new features becomes very
costly. Finally, manager starts rewrite project. The new project starts
with great expectations and adds a lot more wishlist, which adds more work
than time. In the end, programmers try to finish the project with same
hacks and they go back to first situation. Instead of costly rewrites,
agile methodologies encourage high quality and refactoring to keep the code
fluid. This takes courage both from development, management and customers.

Unfortunately, my real life experience has been like most people where you
are given a date and features without any estimate and negotation. I have
worked for a number of fortune 100 or fortune 500 companies and development
in all these places is pretty much same, i.e., you feel powerless when you
are dealing with unrealistic timelines. And despite the fact that you want
to do good job by writing good quality code with decent coverge (even with
overtime), often you are forced to sacrifice all that.

I have been also reading Mary Poppendieck‘s Implementing Lean Software Development. She also talks about companies who last longer than
other companies and key difference is that long lived companies focus on
long term benefits and quality cost more in short term, but gives more long
term benefits.

See Agile Methodologies Under the Hood

Agile Methodologies Under the Hood
Based on research and Scott Ambler’s article, agile methodologies have crossed
chasm and moved to mainstream. But I find most companies seems to follow
practices without understanding or adopting underlying foundations for these methodologies.

Agile manifesto describes four principles such as:
Individuals and interactions over processes and tools; Working software over
comprehensive documentation; Customer collaboration over contract negotiation;
Responding to change over following a plan. But how do we adopt these
principles. Most people focus on practices, for example they may follow
practices of XP such as
Pair Programming, TDD, Refactoring, Small Releases, Continuous
Integration, Collective Ownership, Sustainable Work Pace, etc. without
accepting all of its values. Crystal Clear has similar properties, though some
of they are like practices such as Frequent Delivery, Reflective Improvement,
and Automated Testing, whereas others are like principles such as
Osmotic Communication, Collaboration, Personal Safety.
Scrum is PM based process that
practices such as Frequent delivery, Daily Scrums and principles such as
Customer focus, Transparency, Energized work hours.
Lean Software Development is
largely based on practices such as Delivery Fast and TDD (Quality) and
principles such as Eliminate Waste, Knowledge, Defer Commitment and Respect.
ASD don’t have practices and is
largely based on principles such as Speculate, Collaborate and Learning.
I don’t include RUP,
though there are a number of agile variations.
Similarly, FDD is also iterative
based process, but it’s a bit more BDUF similar to RUP.
Though, I have not looked closely at DSDM and PRINCE2, but they seem to be
general purpose PM based methodologies.

As you can see there are common themes across them such as:

• Short Iterative Development
• Up front Quality using Testing, Pairing
• Communication/Reflective Environment
• Transparency/Big Visible Charts
• Respect/Personal Safety
• Collaboration

However, at deeper level agile methodologies are about two things: egalitarian
culture and giving up control. In agile organizations, everyone works
for a common goal without any selfish motives. Principles such as collaboration,
honest communication, self-organizing teams, responsibility, accountability
requires people will do the right thing without any kind of control. This is
probably biggest reason, why agile methodologies are adopted only at skin
level because most people are driven by personal success, ambition and power.
At many companies, people use information as a tool of control and power. Some
of the practices such as respect, collaboration and honest communication
are hard.
Agile processes assume developers, customers and core stake holders have
same commitment. However, in practice it often falls short. For example,
XP recommends collective ownership of code that can be really difficult in
teams where a few developers are not responsible and don’t believe in
excellence.

Second, Agile methodologies are all about giving control: the management
gives control of planning, deliverables to the development team.
Most companies are run based on command-control and don’t give up such control.
The
development in turn gives control to the customers for dictating what needs
to be delivered. The development team work on features defined by customers,
they can’t just spent months on big architectures or frameworks without
providing real software. Obviously, it works only if everyone is collaborating,
communicating and have a reward system that favors these principles (instead of usual hero based culture or firefighting culture).

In most places, clients don’t trust software companies, that is why they
work on big requirement specification because otherwise they will be
cornered into expensive change-request cycles. These companies create RFPs
and software/consulting companies respond by providing services for entire
project. On the other hand agile projects cannot be run based on BRUF,
they need continuous collaboration and the best form of delivery is piece
meal. It means the software developers may risk being fired if they
can’t deliver in first few iterations.