Shahzad Bhatti Welcome to my ramblings and rants!

Problem Description

I recently had to make some architecture changes to our application and before making those changes, I decided to add a suite of functional tests to cover essential areas of the application. Like most real world dev. shops, testing is not integral part of the development process at Amazon and though, we had some unit tests but didn’t have any functional tests. I needed to get something quickly. Another thing was that our application uses a lot of AJAX, especially search functionality retrieves results using AJAX instead of showing everything when the form is submitted. So, I needed something that could test AJAX with real browser. I had heard of Selenium and a bit experience with it so I chose it for functional testing. Next, I needed to setup a load test environment and I chose Grinder due to some prior experience as well. The rest of the blog shows how to actually setup these two together as there were some glitches.

Selenium IDE

First, install Firefox plugin for Selenium IDE from http://www.openqa.org/selenium-ide/. I used it to record functional tests, which is pretty easy. Once the plugin is installed, select it from Tools->Selenium IDE and it will automatically start recording it. Point your firefox to your application and start capturing the use case that you are interested in. I suggest recording one use case at a time and when you are finished with the use case, switch over to the IDE and click “Stop Recording” red button. You can then choose to export the captured use case in a number of languages. I chose to export it to Java. For example, my simple test looked like:

 1 package com.amazon.biw2.webapp.tests; 
 2 
 3 import com.thoughtworks.selenium.*; 
 4 import java.util.regex.Pattern; 
 5 import java.util.Arrays; 
 6 import junit.framework.TestSuite; 
 7 
 8 
 9 public class BiwSearchTest extends SeleneseTestCase { 
10     private static final String url = System.getProperty("url", "http://shahbhat.desktop:8080"); 
11     static final long SLEEP_TIME = 60000; 
12 
13 
14     public void setUp() { 
15         selenium = new DefaultSelenium("localhost", 4444, "*firefox", url); 
16         selenium.start(); 
17     } 
18 
19 
20     public void tearDown() { 
21         selenium.stop(); 
22     } 
23 
24     public void testProductSearch() throws Exception { 
25         selenium.open("/searchProductForm.html?TestMode=true"); 
26         selenium.waitForPageToLoad(String.valueOf(SLEEP_TIME)); 
27         selenium.click("link=Search by Products"); 
28         selenium.select("reportType", "label=Unfilled Demand"); 
29         selenium.select("GLProductGroup", "label=Apparel"); 
30         selenium.click("//a[@onclick='if(validateSelection()){submitSearch();} return false;']"); 
31         selenium.waitForPageToLoad(String.valueOf(SLEEP_TIME)); 
32         verifyFalse(selenium.isTextPresent("No results")); 
33         verifyTrue(selenium.isTextPresent("Loading: ")); 
34         selenium.getEval("this.browserbot.getCurrentWindow().initBody();"); 
35         String done = selenium.getValue("allDataFieldsComplete"); 
36         while ("true".equals(done) == false) { 
37             System.out.println("Waiting to load " + done); 
38             Thread.currentThread().sleep(5000); 
39             done = selenium.isElementPresent("allDataFieldsComplete") ? selenium.getValue("allDataFieldsComplete") : "not-present"; 
40         } 
41     } 
42 
43     public static void main(String[] args) throws Exception { 
44         junit.textui.TestRunner.run(BiwSearchTest.class); 
45     } 
46 
47     public static TestSuite suite() { 
48         TestSuite suite = new TestSuite(BiwSearchTest.class); 
49         return suite; 
50     } 
51 } 
52

---

Selenium Remote Control

Next, I downloaded Selenium Remote Control from http://www.openqa.org/selenium-rc/. However, I found that that version does not work with Firefox 2.0 so I downloaded snapshot version from http://release.openqa.org/selenium-remote-control/nightly/. After unzipping it, I started remote control server with:

java -jar selenium-server.jar

I then compiled and ran my program
javac -d classes -classpath classes:selenium-remote-control-0.9.1-SNAPSHOT/java/selenium-java-client-driver.jar:selenium-remote-control-0.9.1-SNAPSHOT/server/selenium-server.jar: BiwSearchTest.java


java -classpath classes:selenium-remote-control-0.9.1-SNAPSHOT/java/selenium-java-client-driver.jar:selenium-remote-control-0.9.1-SNAPSHOT/server/selenium-server.jar: com.amazon.biw2.webapp.tests.BiwSearchTest


As expected, it launched a firefox browser window (I closed all my firefox before running it), ran my test and then closed the browser.

Selenium Gotchas

I ran into a couple of Selenium gotaches. First, Selenium does not run your onLoad javascript and I had to explicitly call eval method to start the javascript. Second, some of the operations on Selenium don’t work. For example, one of the form showed results in another window and to get the handle of new window, I tried iterating through windows using Selenium’s getAllWindowsByIds/Names/Titles, but could not do it. I even tried getAttributeFromAllWindows, but that didn’t work either. Also, SeleniumTest class does not define constructor to take in name of a test, so you can’t just run a single test.

Setting up Grinder

Next I downloaded grinder 3.0 from http://grinder.sourceforge.net/download.html. I picked latest beta 33 release. I then created a properties file as follows:

grinder.processes=2 
grinder.threads=5 
grinder.runs=5 
grinder.script=biw_search.py 
grinder.logDirectory=logs 
grinder.numberOfOldLogs=0 
grinder.statistics.delayReports = 1 
grinder.consoleHost=localhost 
grinder.consolePort=6372 
grinder.processIncrementInterval=60000ms 
grinder.initialSleepTime=60000ms 
grinder.reportToConsole.interval=500ms

I then created a python script to kick off the Selenium test.

 1 from net.grinder.script.Grinder import grinder 
 2 from net.grinder.script import Test 
 3 from net.grinder.plugin.http import HTTPRequest 
 4 from java.lang import System 
 5 from java.lang import String 
 6 
 7 from junit.framework import TestSuite 
 8 from junit.framework import TestResult 
 9 
10 from com.amazon.biw2.webapp.tests import BiwAdpTest 
11 from com.amazon.biw2.webapp.tests import BiwAsinSearchTest 
12 from com.amazon.biw2.webapp.tests import BiwKeywordSearchTest 
13 from com.amazon.biw2.webapp.tests import BiwMediaSearchTest 
14 from com.amazon.biw2.webapp.tests import BiwProductSearchTest 
15 from com.amazon.biw2.webapp.tests import BiwSearchTest 
16 
17 
18 
19 def createTestRunner(script): 
20     exec("x = %s.TestRunner()" % script) 
21     return x 
22 
23 class TestRunner: 
24     def __init__(self): 
25         tid = grinder.threadID 
26         self.initialisationTime = System.currentTimeMillis() 
27         #if tid % 5 == 4: 
28         #    self.testRunner = createTestRunner(scripts[1]) 
29 
30     def __call__(self): 
31         # Turn off automatic reporting for the current worker thread. 
32         # Having done this, the script can modify or set the statistics 
33         # before they are sent to the log and the console. 
34         grinder.statistics.delayReports = 1 
35 
36         tid = grinder.threadID 
37 
38         # Creates a Test Suite. 
39         if tid % 5 == 4: 
40             suite = TestSuite(BiwAdpTest().getClass()) 
41         elif tid % 5 == 3: 
42             suite = TestSuite(BiwAsinSearchTest().getClass()) 
43         elif tid % 5 == 2: 
44             suite = TestSuite(BiwKeywordSearchTest().getClass()) 
45         elif tid % 5 == 1: 
46             suite = TestSuite(BiwMediaSearchTest().getClass()) 
47         else: 
48             suite = TestSuite(BiwProductSearchTest().getClass()) 
49 
50         # Returns the tests as an enumeration. 
51         tests = suite.tests(); 
52 
53         # Iterate over the tests. 
54         testNumber = 0 
55         for test in tests: 
56             testNumber += 1 
57             testCase = Test(testNumber, test.getName()).wrap(suite) 
58 
59             testResult = TestResult() 
60             testCase.runTest(test, testResult) 
61 
62             if testResult.errorCount() > 0: 
63                 grinder.statistics.success = 0 
64             elif testResult.failureCount() > 0: 
65                 grinder.statistics.success = 0 
66 
67 
68     def __del__(self): 
69         elapsed = System.currentTimeMillis() - self.initialisationTime 
70

Grinder Gotchas

First I tried to call my test directly, but noticed that I wasn’t getting any data, so I had to wrap my test with Test to get the statistics.

Setting up Grinder Console/Agents

When setting up grinder, you launch test agents on multiple machine and then start a console on your local laptop or workstation. You kick of tests from console, once you start the tests all agents will start running your tests and will send back statistics to the console and you can view them on your console. For my test, I ran both agent and console on my laptop, but running agents on multiple servers will be similar.
I started console using
java -classpath grinder-3.0-beta33/lib/grinder-j2se5.jar;grinder-3.0-beta33/lib/grinder-xmlbeans.jar;grinder-3.0-beta33/lib/grinder.jar;grinder-3.0-beta33/lib/jsr173_1.0_api.jar;grinder-3.0-beta33/lib/jython.jar;grinder-3.0-beta33/lib/picocontainer-1.2-RC-1.jar;grinder-3.0-beta33/lib/xbean.jar net.grinder.Console


---

I then kicked of test agent using
java -classpath -classpath classes:selenium-remote-control-0.9.1-SNAPSHOT/java/selenium-java-client-driver.jar:selenium-remote-control-0.9.1-SNAPSHOT/server/selenium-server.jar:grinder-3.0-beta33/lib/grinder-j2se5.jar;grinder-3.0-beta33/lib/grinder-xmlbeans.jar;grinder-3.0-beta33/lib/grinder.jar;grinder-3.0-beta33/lib/jsr173_1.0_api.jar;grinder-3.0-beta33/lib/jython.jar;grinder-3.0-beta33/lib/picocontainer-1.2-RC-1.jar;grinder-3.0-beta33/lib/xbean.jar net.grinder.Grinder grinder.properties

By default the agent waits until you kick off the test from console so I switched over to console and click start and the console started collecting data. Now onto real work which is writing my own algorithm for load balancer because default load balancer is just too dumb. What I would like is an algorithm that takes user's network into account to find closest server (we have three data servers all over world), and actual health of the server.

1. A service should use convention over configuration so there should be minimal configuratios required.
2. A configuration name should be descriptive and should include service name if it is required.
3. A configuration should support aggregation of properties from different sources. For example, you may be combining some configuration properties from multiple files.
4. A configuration should support property overrides, either using embedded property files. external property files or runtime arguments.
5. A configuration should support hierarchical properties.
6. A configuration should be reloadable either using file touch or database refresh.
7. The service should provide a way to dump the configurations it is using.
8. The configuration files should support symbols that can be provided by the runtime environment. For example, in certain cases, you may need to define similar configurations for different environments such as database name for US is US_myapp and for Canada, it is CA_myapp.
9. A service container should independently load each service and all dependent configurations so that they are managed indpendently.
10. The configuration system should support annotations based properties if supported by underlying language.

1. A service should be language and platform independent and should be easily locatable via registries such as LDAP, UDDI, ebXML. Though, lower level services or closely cohesive services can use native languages or protocols. A service should have a unique name and a version number, so that different versions can be running at the same time.
2. A service should require a minimal installation. In other words, clients should not be required to install any software locally, though a simple jar file is acceptable that can be easily embedded on the client. However, a service should ideally use protocols that are already available on most computers such as TCP/UDP or HTTP, which can be easily done for REST based services.
3. A service should fail fast. During service initialization or invocation, it should verify all inputs and dependencies and throw informative errors right away.
4. A service should provide meaningful and specific error messages or error codes. There should not be messages like InternalError or unknown-error. Additionally, if stack trace is available then it should pass a copy of it as well.
5. Though, some high level services can be stateful, but in general services should be stateless.
6. A service should provide a way to pass batch of input data, i.e, they should be easily composable. However, in order to minimize memory consumptions, it could provide scrolling APIs instead of return list of items.
7. A service should provide a way to invoke asynchronously. This allows service to reduce dependencies. Also, services should be easily recoverable in case of crashes, i.e., no messages are lost.
8. A service implementation should use queues for incoming requests, which can be done either by using messaging middleware or via some connection queue. The queues can also be persistent for reliable services so that when a service is down, the requests are not lost. Also, messages should be ordered so that execution of those messages are also performed in order. For example, when insert request and update requests are not in order, then data can be corrupted.
9. A service implementation should use pull instead of push for invocation. This helps scalability because when server is busy, service can throttle requests.
10. A service should provide logging, monitoring, life-cycle management and for financial or secured services authentication and auditing capabilities. A good lifecycle management can also be used to hot deploy service, where a new version of the service can listen for new requests and old service can serve existing requests and then dies off once it’s done. A service should be easily configurable and should support an API to dump the configuration. One of the parameter I like to have for a service is timeout.

Ever since Spring/Hibernate came about in early 2000, there has been increasingly bashing on J2EE. I saw another post on similar topic from Dan CresWell. He questions using J2EE to mean many things and using J2EE in resumes. I have been doing distributed programming for over ten years from BSD sockets, to Java, CORBA, EJB, JINI, Messaging, etc. I found J2EE containers were a great improvements over their predecessors. Though, I agree that J2EE umbrella means a lot of things to different people, and we should be using real words to describe enterprise level concerns such as deployment, monitoring, networking, concurrency, etc. For any large enterprise systems, the need for monitoring, scaling, security and availability are very critical. J2EE created a abstraction for developers for writing enterprise and distributed programming by implementing internal threading, transactions, security, remoting, and monitoring (JMX). Needless to say, it was mistake to create that abstraction layer that forced developers to forget about the realities of underlying system. However, things like monitoring, hot deployment, security and high availability should only be managed in a container or centralized way. These problems are really hard and I have not seen any reliable solutions outside commercial space. These problems are best handled at architecture level rather than at implementation level such as using J2EE. I still see large organizations use J2EE in one way or the other. Though, they may be using some in-house frameworks for monitoring, deployment, etc. For example, many large companies use many stateless services, which are managed independently. In a lot of cases, a database or memory server is used for state. Such architecture can lend itself easily for high availability. For example, at Amazon or Ebay, the site is backed by many many services and at any time, one or more service may be restarting, but it does not effect most of the users.

Dan also raises question about putting J2EE in resumes to get more responses. I find most companies use keyword search for finding matching candidate, but it’s mistake. You can never rely on some computer program to present you the matching candidates. You will have to read the resumes and talk to the people to learn if they understand these concepts and have relevant experience.

I started at Amazon last year, but didn’t actually got chance to work with them until recently when we had to integrate with Amazon Ecommerce Service (ECS).

Amazon Web Services come in two flavors: REST and SOAP. According to inside sources about 70% use REST. I also found that REST interface was more reliable and simple. Though, I will describe both techniques here:

Getting Access ID

First, visit http://www.amazon.com/gp/browse.html?node=3435361 to get your own access key.

RTFM

I will describe ECS here and it comes with 450 pages of documentation, though most of it just describes URLs and input/output fields. You can find documentation and sample code at http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=59. I also found Eric Giguerre’s tutorial on AWS very useful.

Other interesting links include: blog site for updates on AWS, a Forum #1, Forum #2 and FAQ.

Services

Inside ECS, you will find following services:

• ItemSearch
• BrowseNodeLookup
• CustomerContentLookup
• ItemLookup
• ListLookup
• SellerLookup
• SellerListingLookup
• SimilarityLookup
• TransactionLookup

REST Approach

The rest approach is pretty simple, in fact you can simply type in following
URL to your browser (with your access key) and will see the results (in XML)
right away:

Finding images for Harry Potter Video:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=[your-key] &Operation=ItemSearch&SearchIndex=Video&Keywords=potter%20harry&ResponseGroup=Images

Finding images for Harry Potter Video:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=[your-key] &Operation=ItemSearch&SearchIndex=Books&Keywords=rails&ResponseGroup=Request,Small

Finding ASINS by keywords:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=your-key &Operation=ItemSearch&SearchIndex=Books&Keywords=rails&ResponseGroup=ItemIds

Find DVD cover art:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=[ID]&Operation=ItemSearch &SearchIndex=DVD &Keywords=potter%20harry &ResponseGroup=Images

Find CDs that contain music by Beethoven:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Music &ResponseGroup=Small,Tracks &Composer=Beethoven

Find by Vendor:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Apparel &ResponseGroup=Large,Variations &MerchantId=[ID] &ItemPage=1
http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Apparel &ResponseGroup=Large,Variations &MerchantId=[ID] &ItemPage=2

Find all new products on Amazon that cost less than $1:00

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Blended &ResponseGroup=Small,Offers &MerchantId=All &MaximumPrice=99

Find all new/old products on Amazon that cost less than $1:00

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Blended &ResponseGroup=Small,Offers &MerchantId=All &MaximumPrice=99 &Condition=All

Find used Barbie dolls

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Toys &Title=Barbie &Manufacturer=Mattel &Condition=All &ItemPage=1
or
http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Toys &Title=Barbie &Manufacturer=Mattel &Condition=All &ItemPage=2
Scenario #6:

Search for Godiva dark

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=GourmetFood &Keywords=dark%20chocolate
&Manufacturer=Godiva

Search for purple products

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Blended &Keywords=purple

Find competitive pricing

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=ABC1,ABC2,P12345 &ResponseGroup=Request,Small,Offers &Condition=All &MerchantId=All

Find a toy by UPC

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup Amazon E-Commerce Service Developer Guide 46 &IdType=UPC &ItemId=[UPC] &SearchIndex=Toys &ResponseGroup=Request,Small,Offers &Condition=Collectible &MerchantId=All

Find a particular gas gril

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &IdType=UPC &ItemId=[UPC] &SearchIndex=OutdoorLiving &DeliveryMethod=ISUP &ISPUPostalCode=12345 &ResponseGroup=Request,Small,Offers &Condition=All &MerchantId=All

Compare pricing for different size/color

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &IdType=SKU &ItemId=[SKU1,SKU2,SKU3] &SearchIndex=Apparel &ResponseGroup=Request,Small,Offers,Variations &MerchantId=[ID]

Find a book

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=[ASIN] &SearchIndex=Books &ResponseGroup=Request,ItemAttributes,Offers

Find by ASIN

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=[ASIN]

Find reviews for bestsellers

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] Amazon E-Commerce Service Developer Guide 47 &Operation=ItemLookup &ItemId=[ASIN] &SearchIndex=Books &ResponseGroup=Request,EditorialReview,Reviews,SalesRank

See additional customer reviews

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=[ASIN] &SearchIndex=Books &ResponseGroup=Request,Reviews &ReviewPage=2 http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=[ASIN] &SearchIndex=Books &ResponseGroup=Request,Reviews &ReviewPage=3

Lookup samples and notes

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=SimilarityLookup &ItemId=ABC1 &ResponseGroup=Request,Small,Offers &Condition=All &MerchantId=All

Lookup similar group of products

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=SimilarityLookup &ItemId=ABC1,ABC2,ABC3 &ResponseGroup=Request,Small,Offers &Condition=All &MerchantId=All

The REST base URLs are:

• United States (US): http://webservices.amazon.com/onca/xml?Service=AWSECommerceService
• United Kingdom (UK): http://webservices.amazon.co.uk/onca/xml?Service=AWSECommerceService
• Germany (DE): http://webservices.amazon.de/onca/xml?Service=AWSECommerceService
• Japan (JP): http://webservices.amazon.co.jp/onca/xml?Service=AWSECommerceService
• Canada (CA): http://webservices.amazon.ca/onca/xml?Service=AWSECommerceService
• France (FR): http://webservices.amazon.fr/onca/xml?Service=AWSECommerceService

REST request is pretty simple, in Java all you need is to create URL and add all service arguments as form arguments. For example,

 1
 2     String createUrl(Map<String,String> map) {
 3         StringBuilder b = new StringBuilder(urlTarget);
 4         b.append("&AWSAccessKeyId=");
 5         b.append(subscriptionId);
 6         if (associateTag != null){
 7             b.append("&AssociateTag=");
 8             b.append(associateTag);
 9         }
10
11         for (Map.Entry<string,> entry : map.entrySet()) {
12             b.append('&');
13             b.append(entry.getKey());
14             b.append('=');
15             try {
16                 b.append(URLEncoder.encode(entry.getValue(), "UTF8"));
17             } catch (UnsupportedEncodingException e) {
18                 throw new RuntimeException("Failed to encode '" + entry.getValue() + "'", e);
19             }
20         }
21         return b.toString();
22     }
23     Map map ... setup
24     URL u = new URL(createurl(map));
25     URLConnection connection = u.openConnection();
26     InputStream in = connection.getInputStream();
27
28 }
29
30

// create DOM and parse XML here

SOAP interface:

The SOAP based interaction is more complicated. First thing you need is to download Java client and you can also find
javadocs.

The SOAP endpoints:

• US: http://webservices.amazon.com/onca/soap?Service=AWSECommerceService
• UK: http://webservices.amazon.co.uk/onca/soap?Service=AWSECommerceService
• DE: http://webservices.amazon.de/onca/soap?Service=AWSECommerceService
• JP: http://webservices.amazon.co.jp/onca/soap?Service=AWSECommerceService
• CA: http://webservices.amazon.ca/onca/soap?Service=AWSECommerceService
• FR: http://webservices.amazon.fr/onca/soap?Service=AWSECommerceService

The WSDL locations for each SOAP endpoint are:

• US: http://webservices.amazon.com/AWSECommerceService/AWSECommerceService.wsdl
• UK: http://webservices.amazon.com/AWSECommerceService/UK/AWSECommerceService.wsdl
• DE: http://webservices.amazon.com/AWSECommerceService/DE/AWSECommerceService.wsdl
• JP: http://webservices.amazon.com/AWSECommerceService/JP/AWSECommerceService.wsdl
• CA: http://webservices.amazon.com/AWSECommerceService/CA/AWSECommerceService.wsdl
• FR: http://webservices.amazon.com/AWSECommerceService/FR/AWSECommerceService.wsdl

Finally, the XML schemas are available from these locations:

• US: http://webservices.amazon.com/AWSECommerceService/AWSECommerceService.xsd
• UK: http://webservices.amazon.com/AWSECommerceService/UK/AWSECommerceService.xsd
• DE: http://webservices.amazon.com/AWSECommerceService/DE/AWSECommerceService.xsd
• JP: http://webservices.amazon.com/AWSECommerceService/JP/AWSECommerceService.xsd
• CA: http://webservices.amazon.com/AWSECommerceService/CA/AWSECommerceService.xsd
• FR: http://webservices.amazon.com/AWSECommerceService/FR/AWSECommerceService.xsd

Show me the code

Here is the complete code for both REST and SOAP, though you will need to add following jars to the CLASSPATH:
junit.jar

• AWS4JavaSample.jar
• axis.jar
• jaxrpc.jar
• commons-logging.jar
• commons-discovery.jar
• saaj.jar
• jdom.jar

  1 import junit.framework.TestCase;
  2 import java.util.Iterator;
  3 import java.util.List;
  4 import java.util.ArrayList;
  5 import java.util.Map;
  6 import java.util.HashMap;
  7 import java.net.URL;
  8 import java.net.URLConnection;
  9 import java.net.MalformedURLException;
 10 import java.net.URLEncoder;
 11 import java.io.UnsupportedEncodingException;
 12 import java.io.InputStream;
 13 import java.io.IOException;
 14
 15 import java.lang.reflect.Constructor;
 16 import java.lang.reflect.Method;
 17 import java.lang.reflect.InvocationTargetException;
 18
 19 import java.rmi.RemoteException;
 20 import javax.xml.rpc.ServiceException;
 21
 22 import com.amazon.xml.AWSECommerceService.ItemSearchRequest;
 23 import com.amazon.xml.AWSECommerceService._ItemSearchResponse;
 24 import com.amazon.xml.AWSECommerceService._ItemSearch;
 25 import com.amazon.xml.AWSECommerceService._Items;
 26 import com.amazon.xml.AWSECommerceService._Item;
 27 import com.amazon.xml.AWSECommerceService.AWSECommerceServicePortType;
 28 import com.amazon.xml.AWSECommerceService.AWSECommerceService;
 29 import com.amazon.xml.AWSECommerceService.AWSECommerceServiceLocator;
 30 import org.jdom.*;
 31 import org.jdom.input.*;
 32 import org.jdom.output.*;
 33
 34 public class EcsTest extends TestCase {
 35     protected void setUp() throws Exception {
 36     }
 37
 38
 39     private SoapRequest setupSoapRequest(String operation) throws MalformedURLException, ServiceException {
 40         SoapRequest request = new SoapRequest("US", "xml", "YourKey", null);
 41         request.put("ItemPage", "1");
 42         request.put("SearchIndex", "Books");
 43         request.put("Keywords", "rails");
 44         request.put("ResponseGroup", "ItemIds");
 45         request.put("Sort", "salesrank");
 46         request.put("ResponseGroup", "SalesRank,Small" );
 47         return request;
 48     }
 49
 50
 51     private RestRequest setupRestRequest(String operation) {
 52         RestRequest request = new RestRequest("US", "soap", "YourKey", null);
 53         //request.put("AWSAccessKeyId", "YourKey");
 54         //request.put("Author", author);
 55         request.put("ItemPage", "1");
 56         request.put("Operation", operation);
 57         request.put("SearchIndex", "Books");
 58         request.put("Keywords", "rails");
 59         request.put("ResponseGroup", "ItemIds");
 60         request.put("Sort", "salesrank");
 61         //request.put("ResponseGroup", "SalesRank,Small" );
 62         request.put("ResponseGroup", "ItemIds" );
 63         return request;
 64     }
 65     //
 66     public void testGetAsinsBySoap() throws Exception {
 67         SoapRequest request = setupSoapRequest("ItemSearch");
 68         _Items[] items = request.invoke();
 69         for (int i=0; i<items.length; i++) {
 70             _Item[] item = items[i].getItem();
 71             for (int j=0; i<item.length; j++) {
 72                 assertTrue(item[j].getASIN() != null);
 73             }
 74         }
 75     }
 76
 77
 78     public void testGetAsinsByRest() throws Exception {
 79         RestRequest request = setupRestRequest("ItemSearch");
 80         URL u = new URL(request.toString());
 81         URLConnection connection = u.openConnection();
 82         InputStream in = connection.getInputStream();
 83         List<String> asins = parseAsins(in);
 84         for (String asin : asins) {
 85             System.out.println("asin: " + asin);
 86         }
 87     }
 88     private List<String> parseAsins(InputStream in) throws IOException, JDOMException {
 89         SAXBuilder builder = new SAXBuilder();
 90         Document doc = builder.build(in);
 91
 92         Format format = Format.getPrettyFormat();
 93         XMLOutputter out = new XMLOutputter(format);
 94         out.output(doc, System.out);
 95
 96         Element root = doc.getRootElement();
 97         Namespace ns = root.getNamespace();
 98         Element items = root.getChild( "Items", ns );
 99         Element request = items.getChild( "Request", ns );
100
101         // First make sure the response is valid
102
103         String isValid = request.getChild("IsValid", ns).getTextTrim();
104         if (!isValid.equals("True")){
105             throw new RuntimeException("Invalid response " + isValid);
106         }
107
108         // Now make sure there are no errors -- would be a good
109         // idea to collect and print them
110
111         Element errors = request.getChild("Errors", ns);
112
113         if( errors != null ){
114             throw new RuntimeException("One or more errors in the response " + errors);
115         }
116         int max = Integer.parseInt( items.getChild( "TotalResults", ns ).getTextTrim() );
117         List itemList = items.getChildren("Item", ns);
118         List<String> asins = new ArrayList<String>();
119         Iterator it = itemList.iterator();
120         while( it.hasNext()) {
121             Element item = (Element) it.next();
122             Element asinElement = item.getChild("ASIN", ns);
123             asins.add(asinElement.getTextTrim());
124             Element rankElement = item.getChild("SalesRank", ns);
125             if ( rankElement != null ) {
126                 int rank = extractRank( rankElement );
127                 String title = item.getChild( "ItemAttributes", ns ).getChild( "Title", ns ).getTextTrim();
128
129                 System.out.println( "Rank=" + rank + " Title=" + title );
130
131             }
132         }
133         return asins;
134     }
135
136     private int extractRank( Element e ) throws NumberFormatException {
137         String       rank = e.getTextTrim();
138         int          len = rank.length();
139         StringBuffer b = new StringBuffer( len );
140
141         for( int i = 0; i < len; ++i ){
142             char ch = rank.charAt( i );
143             if( ch == '.' || ch == ',' || ch == ' ' ) continue;
144             b.append( ch );
145         }
146
147         return Integer.parseInt( b.toString() );
148     }
149
150
151
152     public static void main(String[] args) {
153         junit.textui.TestRunner.run(EcsTest.class);
154     }
155 }
156
157 class SoapRequest extends HashMap<String, String> {
158     final String locale;
159     final String protocol;
160     final String subscriptionId;
161     final String associateTag;
162     final ItemSearchRequest itemSearchRequest;
163     final AWSECommerceServicePortType port;
164     SoapRequest(String locale, String protocol, String subscriptionId, String associateTag) throws MalformedURLException, ServiceException {
165         this.locale = locale;
166         this.protocol = protocol;
167         this.subscriptionId = subscriptionId;
168         this.associateTag = associateTag;
169         String urlTarget = new EcsUtils().getUrl("US", "soap");
170         this.itemSearchRequest = new ItemSearchRequest();
171         AWSECommerceService apd = new AWSECommerceServiceLocator();
172         this.port = apd.getAWSECommerceServicePort(new URL(urlTarget));
173     }
174     private void set(String key, Object value) throws NoSuchMethodException, IllegalAccessException, InvocationTargetException, InstantiationException {
175         if (value == null) return;
176         String name = "set" + key;
177         Method[] methods = itemSearchRequest.getClass().getMethods();
178         for (int i=0; i<methods.length; i++) {
179             if (methods[i].getName().equals(name) && methods[i].getParameterTypes().length == 1) {
180                 Class arg = methods[i].getParameterTypes()[0];
181                 if (arg == String[].class) {
182                     value = value.toString().split(",");
183                 } else if (!arg.isAssignableFrom(value.getClass())) {
184                     Constructor ctor = null;
185                     try {
186                         ctor = arg.getConstructor(value.getClass());
187                     } catch (NoSuchMethodException e) {
188                     }
189                     if (ctor != null) {
190                         value = ctor.newInstance(value);
191                     } else {
192                         throw new IllegalArgumentException("Failed to invoke setter for '" + name + "' with value '" + value + "' with signature " + methods[i] + " in " + itemSearchRequest.getClass().getName());
193                     }
194                 }
195                 methods[i].invoke(itemSearchRequest, value);
196                 return;
197             }
198         }
199         throw new IllegalArgumentException("Failed to find setter for '" + name + "' with value '" + value + "' in " + itemSearchRequest.getClass().getName());
200     }
201     //
202     public _Items[] invoke() throws NoSuchMethodException, IllegalAccessException, InvocationTargetException, InstantiationException, RemoteException {
203         for (Map.Entry<String, String> entry : this.entrySet()) {
204             if (entry.getValue() != null) {
205                 set(entry.getKey(), entry.getValue());
206             }
207         }
208         _ItemSearch itemSearchBody = new _ItemSearch();
209         itemSearchBody.setSubscriptionId(subscriptionId);
210         if (associateTag != null){
211             itemSearchBody.setAssociateTag(associateTag);
212         }
213         //itemSearchBody.setValidate(...);
214         itemSearchBody.setRequest(new ItemSearchRequest[] {itemSearchRequest});
215         return port.itemSearch(itemSearchBody).getItems();
216     }
217 }
218
219
220 class RestRequest extends HashMap<String, String> {
221     final String locale;
222     final String protocol;
223     final String subscriptionId;
224     final String associateTag;
225     final String urlTarget;
226     RestRequest(String locale, String protocol, String subscriptionId, String associateTag) {
227         this.locale = locale;
228         this.protocol = protocol;
229         this.subscriptionId = subscriptionId;
230         this.associateTag = associateTag;
231         this.urlTarget = new EcsUtils().getUrl("US", "xml");
232     }
233     public String toString() {
234         StringBuilder b = new StringBuilder(urlTarget);
235         //b.append("&SubscriptionId=");
236         b.append("&AWSAccessKeyId=");
237         b.append(subscriptionId);
238         if (associateTag != null){
239             b.append("&AssociateTag=");
240             b.append(associateTag);
241         }
242
243         for (Map.Entry<String, String> entry : this.entrySet()) {
244             b.append('&');
245             b.append(entry.getKey());
246             b.append('=');
247             try {
248                 b.append(URLEncoder.encode(entry.getValue(), "UTF8"));
249             } catch (UnsupportedEncodingException e) {
250                 throw new RuntimeException("Failed to encode '" + entry.getValue() + "'", e);
251             }
252         }
253         return b.toString();
254     }
255 }
256
257
258 class UrlUtils {
259     private static final String BASE_URL = "http://webservices.amazon.";
260     private static final String[] LOCALES = new String[] {"US", "UK", "DE", "JP", "CA", "FR"};
261     private static final String[] DOMAIN_SUFFIX = new String[] {"com", "co.uk", "de", "co.jp", "ca", "fr"};
262     private final Map<String, String> restUrls;
263     private final Map<String, String> soapUrls;
264
265     UrlUtils() {
266         soapUrls =createUrls("soap");
267         restUrls = createUrls("xml");
268     }
269     public String getUrl(String locale, String protocol) {
270         if ("xml".equalsIgnoreCase(protocol)) {
271             return restUrls.get(locale);
272         } else {
273             return soapUrls.get(locale);
274         }
275     }
276     private static Map<String, String> createUrls(String protocol) {
277         Map<String, String> map = new HashMap<String, String>();
278         for (int i=0; i<LOCALES.length; i++) {
279             map.put(LOCALES[i], BASE_URL + DOMAIN_SUFFIX[i] + "/onca/" +
280                     protocol + "?Service=AWSECommerceService");
281         }
282         return map;
283     }
284 }
285
286
287
288
289
290
291

I have been doing Java programming for over ten years and when I work with Rails, I miss some of the conventions Java has. One of those convention is to use resource bundles for all user messages. Java provides ResourceBundle for that, but other MVC frameworks such as Struts and Spring MVC provides nice support for that. One of the thing that I like in Ruby is expressive and succint syntax, so it took me about an hour to whip up the functionality I needed. I wanted to use the resources file using method invocation or hash syntax, e.g.

MessageResource.my_message

Optionally I could pass in arguments, e.g. if I had message like:

errors_range: "{0} is not in the range {1} through {2}."

which takes in three arguments then I can invoke:

MessageResource.errors_range('credit-card', 10, 20, 30)

In addition, I could use hash access syntax, e.g.

MessageResource[:errors_range, 'credit-card', 10, 20, 30]


The other thing I needed was that if I specify locale then it
should find the message from that locale, e.g.

MessageResource.es.errors_range('credit-card', 10, 20, 30)


Finally, I wanted this to be robust so that if the key-code for the
message or locale is not found then it returns key-code instead of
throwing exception or passing nil back (this can waste a lot of time
when developing web applications.)

So without further delay, here is the code:

 1 require 'yaml'
 2 require 'resource_bundle.rb'
 3 
 4 class MessageResource
 5   @@bundles = {}
 6   def self.[](code, *args)
 7     self.populate unless @@bundles.size > 0
 8     key = code.to_s
 9     if key.length == 2 and @@bundles.has_key? key
10        bundle = @@bundles[key] || "??#{code}-#{args}??"
11     else
12        bundle = @@bundles['en']
13        if key.length == 2
14           bundle
15        else
16           bundle[key, *args] || "??#{code}??"
17        end
18      end
19   end
20 
21   ###################
22   # Populates all resource bundles and stores them by locale.
23   ###################
24   def self.populate
25     unless @@bundles.size > 0
26     files = Dir.glob(File.dirname(__FILE__) + "/../../../config/messages*")
27     files.each do |f|
28       locale = 'en'
29       locale = $&.slice(1,2) if f =~ /_...yml/
30       begin
31         messages = YAML.load_file(f)
32       rescue ArgumentError => e
33         raise ArgumentError, "Invalid resource file #{f} -- #{e}", caller
34       end
35       raise ArgumentError, "Invalid resource file #{f}", caller if !messages
36       @@bundles[locale] = ResourceBundle.new(locale, messages)
37     end
38   end
39   end
40 
41   ###################
42   # Defines method_missing for class that simply invokes array operator.
43   ###################
44   def self.method_missing(sym, *args)
45     self[sym, *args]
46   end
47 
48   protected
49   def initialize;end
50 end
51 
52 #
53 #################################################################
54 class ResourceBundle
55   def initialize(locale, messages)
56     @locale = locale
57     @messages = messages
58   end
59   #
60   ###################
61   # Overloads brackets to access messages by keys.
62   ###################
63   def [](code, *args)
64     message = @messages[code.to_s]
65     if message && args
66     args.each_with_index {|arg, i|
67       message.gsub!(/{#{i}}/, arg.to_s)
68     }
69   end
70   message || "???#{code}-#{args.join(',')}???"
71   end
72   ###################
73   # Defines method_missing for this instance that simply invokes array operator.
74   ###################
75   def method_missing(key, *args)
76     self[key, *args]
77   end
78 end

Note, I am using YAML syntax to store messages, here is a sample file:
—
notice_password_changed_wrong: “Your old password did not match, please try again.”
notice_password_mismatch: “Your new password did not match confirmed password, please try again.”
notice_password_changed: “Password was changed successfully.”
notice_password_reset_failed: “Password could not be reset, please try again.”
notice_password_reset_success: “Password was reset successfully.”

Moving my Blog from Blojsom to WordPress

I recently changed my ISP from my old friend Aligen (courtesy of Farrad Ali for providing free access since ’98) to HostMonster. So, instead of running my own Tomcat now I am relying on PHP and Rails. The next thing was how do I move my blogs and preserve the dates. Blojsom was simple file based software, but WordPress uses mysql. So I wrote a simple ruby script to convert it.

The first thing I did was to download dbi from

http://rubyforge.org/projects/ruby-dbi/

I then uncompressed it

		tar xzf dbi-0.1.1.tar.gz

Then

cd ~/ruby-dbi

Since, I don’t have root access on my host, I could not install it to the /usr/bin directory. So I created my own ruby directory

mkdir ~/ruby

and then ran config with my own bin directory as

ruby setup.rb config –bin-dir=~/bin –with=dbi,dbd_mysql –rb-dir=~/ruby –so-dir=~/ruby

Next I ran
ruby setup.rb setup

and then

ruby setup.rb install

Now then fun part, following is a ruby script that reads my flat files and inserts them into wordpress database:

 1 #!/usr/bin/ruby
 2 require 'dbi'
 3 
 4 #
 5 ### import blogs from old blog directory to wordpress
 6 #
 7 class ImportBlogs
 8   def initialize(webapp_dir)
 9     @webapp_dir = webapp_dir
10   end
11 
12   def delete_all
13     DBI.connect('DBI:Mysql:weblog', 'weblog', '*****') do | dbh |
14       dbh.do('delete from wp_posts where id > 2')
15     end
16   end
17 
18   def add_all
19     files = Dir.glob("#{@webapp_dir}/*").delete_if { |f| File.directory?(f) }
20     DBI.connect('DBI:Mysql:weblog', 'weblog', '*****') do | dbh |
21       id = 3
22       post_author = 1
23       sql = "insert into wp_posts(post_author, post_date, post_date_gmt, post_content, post_title, post_category, post_excerpt, post_name, post_modified, post_modified_gmt, guid) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
24       dbh.prepare(sql) do | sth |
25         files.each do |f|
26           lines = nil
27           File.open(f, "r") do |file|
28             lines = file.readlines
29           end
30           post_content = lines.join(' ')
31           post_title = lines[0]
32           post_excerpt = post_content.slice(0,255)
33           post_date = post_date_gmt = post_modified = post_modified_gmt = File.new(f).mtime
34           post_category = 3
35           post_name = File.basename(f)
36           guid = "http://weblog.plexobject.com/?p=#{id}"
37           puts "Adding #{f} mtime #{post_date}"
38           sth.execute(post_author, post_date, post_date_gmt, post_content, post_title, post_category, post_excerpt, post_name, post_modified, post_modified_gmt, guid)
39           id += 1
40         end
41       end
42     end
43   end
44 end
45 
46 ib = ImportBlogs.new('~/webapps/blojsom/computing')
47 ib.delete_all
48 ib.add_all
49 
50 

Finally, I ran it as follows:
ruby -I /usr/lib/ruby/gems/1.8/gems/mysql-2.7 -I /usr/lib/ruby/gems/1.8/gems/mysql-2.7/lib -I ~/ruby -I ~/ruby/DBD -I ~/ruby/dbi -I ~/ruby/DBD/Mysql import_blogs.rb
Voilla, I got everything as expected.

Popular Blog Entries

• Agile Methodologies Under the Hood
• Development in large IT Shops
• Enums and Lookups in Ruby and Java
• Integrating Ruby with Java using ActiveMQ and Stomp
• RailsConf 2006 Day 3
• RailsConf 2006 Day 2
• RailsConf 2006 Day 1
• Integrating ActiveMQ with JBoss
• Agile Purists
• Implementing Const in Java
• Brainstorm Chicago April 19-20
• Message-Driven-Pojos with ActiveMQ
• Sudoku Solver in Ruby
• Nexus for next generation languages
• Scripting Flame War
• Log locally and query globally
  
• Responsibility vs Accountability
• Web 2.0
• Bohrbug Vs Heisenbug
• Keeping the truck number low
• How overwork leads to dumb workers
• Best Practices 2004 (East)
• What is right methodology?
• Offshoring
• Design by Committee
• Software Development Profession is Doomed?
• What’s in the Title
• Software Rewrites
• OO Summary

Courage
One of core practices of agile methodologies such as XP is courage. In
almost every company I have worked with, you have to deal with endless
pressure from managers or customers to meet ridiculous deadlines or
expectations. Courage is standing up for such nonsense. I heard similar
message from Ken Schwaber. However, in most places people tell their managers
or customers what they want to listen. And if you try to speak up against
impossible timelines, then your loyality or commitment is questioned. Also,
in order to meet these deadlines, you end up sacrificing quality and
while code becomes unmanageable. I have seen countless examples, where
a software product starts small, probably written by average joe programmer
and the software becomes a hit and more features are piled up. Soon, the
whole systems becomes unmaintanable and adding new features becomes very
costly. Finally, manager starts rewrite project. The new project starts
with great expectations and adds a lot more wishlist, which adds more work
than time. In the end, programmers try to finish the project with same
hacks and they go back to first situation. Instead of costly rewrites,
agile methodologies encourage high quality and refactoring to keep the code
fluid. This takes courage both from development, management and customers.

Unfortunately, my real life experience has been like most people where you
are given a date and features without any estimate and negotation. I have
worked for a number of fortune 100 or fortune 500 companies and development
in all these places is pretty much same, i.e., you feel powerless when you
are dealing with unrealistic timelines. And despite the fact that you want
to do good job by writing good quality code with decent coverge (even with
overtime), often you are forced to sacrifice all that.

I have been also reading Mary Poppendieck‘s Implementing Lean Software Development. She also talks about companies who last longer than
other companies and key difference is that long lived companies focus on
long term benefits and quality cost more in short term, but gives more long
term benefits.

See Agile Methodologies Under the Hood