Shahzad Bhatti Welcome to my ramblings and rants!

April 12, 2007

Working with Amazon Web Services

Filed under: Computing — admin @ 10:06 pm

I started at Amazon last year, but didn’t actually got chance to work with them until recently when we had to integrate with Amazon Ecommerce Service (ECS).

Amazon Web Services come in two flavors: REST and SOAP. According to inside sources about 70% use REST. I also found that REST interface was more reliable and simple. Though, I will describe both techniques here:

Getting Access ID

First, visit http://www.amazon.com/gp/browse.html?node=3435361 to get your own access key.

RTFM

I will describe ECS here and it comes with 450 pages of documentation, though most of it just describes URLs and input/output fields. You can find documentation and sample code at http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=59. I also found Eric Giguerre’s tutorial on AWS very useful.

Other interesting links include: blog site for updates on AWS, a Forum #1, Forum #2 and FAQ.

Services

Inside ECS, you will find following services:

  • ItemSearch
  • BrowseNodeLookup
  • CustomerContentLookup
  • ItemLookup
  • ListLookup
  • SellerLookup
  • SellerListingLookup
  • SimilarityLookup
  • TransactionLookup

REST Approach

The rest approach is pretty simple, in fact you can simply type in following
URL to your browser (with your access key) and will see the results (in XML)
right away:

Finding images for Harry Potter Video:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=[your-key] &Operation=ItemSearch&SearchIndex=Video&Keywords=potter%20harry&ResponseGroup=Images

Finding images for Harry Potter Video:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=[your-key] &Operation=ItemSearch&SearchIndex=Books&Keywords=rails&ResponseGroup=Request,Small

Finding ASINS by keywords:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=your-key &Operation=ItemSearch&SearchIndex=Books&Keywords=rails&ResponseGroup=ItemIds

Find DVD cover art:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=[ID]&Operation=ItemSearch &SearchIndex=DVD &Keywords=potter%20harry &ResponseGroup=Images

Find CDs that contain music by Beethoven:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Music &ResponseGroup=Small,Tracks &Composer=Beethoven

Find by Vendor:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Apparel &ResponseGroup=Large,Variations &MerchantId=[ID] &ItemPage=1
http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Apparel &ResponseGroup=Large,Variations &MerchantId=[ID] &ItemPage=2

Find all new products on Amazon that cost less than $1:00

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Blended &ResponseGroup=Small,Offers &MerchantId=All &MaximumPrice=99

Find all new/old products on Amazon that cost less than $1:00

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Blended &ResponseGroup=Small,Offers &MerchantId=All &MaximumPrice=99 &Condition=All

Find used Barbie dolls

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Toys &Title=Barbie &Manufacturer=Mattel &Condition=All &ItemPage=1
or
http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Toys &Title=Barbie &Manufacturer=Mattel &Condition=All &ItemPage=2
Scenario #6:

Search for Godiva dark

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=GourmetFood &Keywords=dark%20chocolate
&Manufacturer=Godiva

Search for purple products

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemSearch &SearchIndex=Blended &Keywords=purple

Find competitive pricing

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=ABC1,ABC2,P12345 &ResponseGroup=Request,Small,Offers &Condition=All &MerchantId=All

Find a toy by UPC

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup Amazon E-Commerce Service Developer Guide 46 &IdType=UPC &ItemId=[UPC] &SearchIndex=Toys &ResponseGroup=Request,Small,Offers &Condition=Collectible &MerchantId=All

Find a particular gas gril

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &IdType=UPC &ItemId=[UPC] &SearchIndex=OutdoorLiving &DeliveryMethod=ISUP &ISPUPostalCode=12345 &ResponseGroup=Request,Small,Offers &Condition=All &MerchantId=All

Compare pricing for different size/color

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &IdType=SKU &ItemId=[SKU1,SKU2,SKU3] &SearchIndex=Apparel &ResponseGroup=Request,Small,Offers,Variations &MerchantId=[ID]

Find a book

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=[ASIN] &SearchIndex=Books &ResponseGroup=Request,ItemAttributes,Offers

Find by ASIN

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=[ASIN]

Find reviews for bestsellers

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] Amazon E-Commerce Service Developer Guide 47 &Operation=ItemLookup &ItemId=[ASIN] &SearchIndex=Books &ResponseGroup=Request,EditorialReview,Reviews,SalesRank

See additional customer reviews

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=[ASIN] &SearchIndex=Books &ResponseGroup=Request,Reviews &ReviewPage=2 http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=ItemLookup &ItemId=[ASIN] &SearchIndex=Books &ResponseGroup=Request,Reviews &ReviewPage=3

Lookup samples and notes

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=SimilarityLookup &ItemId=ABC1 &ResponseGroup=Request,Small,Offers &Condition=All &MerchantId=All

Lookup similar group of products

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService &AWSAccessKeyId=[ID] &Operation=SimilarityLookup &ItemId=ABC1,ABC2,ABC3 &ResponseGroup=Request,Small,Offers &Condition=All &MerchantId=All

The REST base URLs are:

  • United States (US): http://webservices.amazon.com/onca/xml?Service=AWSECommerceService
  • United Kingdom (UK): http://webservices.amazon.co.uk/onca/xml?Service=AWSECommerceService
  • Germany (DE): http://webservices.amazon.de/onca/xml?Service=AWSECommerceService
  • Japan (JP): http://webservices.amazon.co.jp/onca/xml?Service=AWSECommerceService
  • Canada (CA): http://webservices.amazon.ca/onca/xml?Service=AWSECommerceService
  • France (FR): http://webservices.amazon.fr/onca/xml?Service=AWSECommerceService

REST request is pretty simple, in Java all you need is to create URL and add all service arguments as form arguments. For example,

 1
 2     String createUrl(Map<String,String> map) {
 3         StringBuilder b = new StringBuilder(urlTarget);
 4         b.append("&AWSAccessKeyId=");
 5         b.append(subscriptionId);
 6         if (associateTag != null){
 7             b.append("&AssociateTag=");
 8             b.append(associateTag);
 9         }
10
11         for (Map.Entry<string,> entry : map.entrySet()) {
12             b.append('&');
13             b.append(entry.getKey());
14             b.append('=');
15             try {
16                 b.append(URLEncoder.encode(entry.getValue(), "UTF8"));
17             } catch (UnsupportedEncodingException e) {
18                 throw new RuntimeException("Failed to encode '" + entry.getValue() + "'", e);
19             }
20         }
21         return b.toString();
22     }
23     Map map ... setup
24     URL u = new URL(createurl(map));
25     URLConnection connection = u.openConnection();
26     InputStream in = connection.getInputStream();
27
28 }
29
30

// create DOM and parse XML here

SOAP interface:

The SOAP based interaction is more complicated. First thing you need is to download Java client and you can also find
javadocs
.

The SOAP endpoints:

  • US: http://webservices.amazon.com/onca/soap?Service=AWSECommerceService
  • UK: http://webservices.amazon.co.uk/onca/soap?Service=AWSECommerceService
  • DE: http://webservices.amazon.de/onca/soap?Service=AWSECommerceService
  • JP: http://webservices.amazon.co.jp/onca/soap?Service=AWSECommerceService
  • CA: http://webservices.amazon.ca/onca/soap?Service=AWSECommerceService
  • FR: http://webservices.amazon.fr/onca/soap?Service=AWSECommerceService

The WSDL locations for each SOAP endpoint are:

  • US: http://webservices.amazon.com/AWSECommerceService/AWSECommerceService.wsdl
  • UK: http://webservices.amazon.com/AWSECommerceService/UK/AWSECommerceService.wsdl
  • DE: http://webservices.amazon.com/AWSECommerceService/DE/AWSECommerceService.wsdl
  • JP: http://webservices.amazon.com/AWSECommerceService/JP/AWSECommerceService.wsdl
  • CA: http://webservices.amazon.com/AWSECommerceService/CA/AWSECommerceService.wsdl
  • FR: http://webservices.amazon.com/AWSECommerceService/FR/AWSECommerceService.wsdl

Finally, the XML schemas are available from these locations:

  • US: http://webservices.amazon.com/AWSECommerceService/AWSECommerceService.xsd
  • UK: http://webservices.amazon.com/AWSECommerceService/UK/AWSECommerceService.xsd
  • DE: http://webservices.amazon.com/AWSECommerceService/DE/AWSECommerceService.xsd
  • JP: http://webservices.amazon.com/AWSECommerceService/JP/AWSECommerceService.xsd
  • CA: http://webservices.amazon.com/AWSECommerceService/CA/AWSECommerceService.xsd
  • FR: http://webservices.amazon.com/AWSECommerceService/FR/AWSECommerceService.xsd

Show me the code

Here is the complete code for both REST and SOAP, though you will need to add following jars to the CLASSPATH:
junit.jar

  • AWS4JavaSample.jar
  • axis.jar
  • jaxrpc.jar
  • commons-logging.jar
  • commons-discovery.jar
  • saaj.jar
  • jdom.jar
  1 import junit.framework.TestCase;
  2 import java.util.Iterator;
  3 import java.util.List;
  4 import java.util.ArrayList;
  5 import java.util.Map;
  6 import java.util.HashMap;
  7 import java.net.URL;
  8 import java.net.URLConnection;
  9 import java.net.MalformedURLException;
 10 import java.net.URLEncoder;
 11 import java.io.UnsupportedEncodingException;
 12 import java.io.InputStream;
 13 import java.io.IOException;
 14
 15 import java.lang.reflect.Constructor;
 16 import java.lang.reflect.Method;
 17 import java.lang.reflect.InvocationTargetException;
 18
 19 import java.rmi.RemoteException;
 20 import javax.xml.rpc.ServiceException;
 21
 22 import com.amazon.xml.AWSECommerceService.ItemSearchRequest;
 23 import com.amazon.xml.AWSECommerceService._ItemSearchResponse;
 24 import com.amazon.xml.AWSECommerceService._ItemSearch;
 25 import com.amazon.xml.AWSECommerceService._Items;
 26 import com.amazon.xml.AWSECommerceService._Item;
 27 import com.amazon.xml.AWSECommerceService.AWSECommerceServicePortType;
 28 import com.amazon.xml.AWSECommerceService.AWSECommerceService;
 29 import com.amazon.xml.AWSECommerceService.AWSECommerceServiceLocator;
 30 import org.jdom.*;
 31 import org.jdom.input.*;
 32 import org.jdom.output.*;
 33
 34 public class EcsTest extends TestCase {
 35     protected void setUp() throws Exception {
 36     }
 37
 38
 39     private SoapRequest setupSoapRequest(String operation) throws MalformedURLException, ServiceException {
 40         SoapRequest request = new SoapRequest("US", "xml", "YourKey", null);
 41         request.put("ItemPage", "1");
 42         request.put("SearchIndex", "Books");
 43         request.put("Keywords", "rails");
 44         request.put("ResponseGroup", "ItemIds");
 45         request.put("Sort", "salesrank");
 46         request.put("ResponseGroup", "SalesRank,Small" );
 47         return request;
 48     }
 49
 50
 51     private RestRequest setupRestRequest(String operation) {
 52         RestRequest request = new RestRequest("US", "soap", "YourKey", null);
 53         //request.put("AWSAccessKeyId", "YourKey");
 54         //request.put("Author", author);
 55         request.put("ItemPage", "1");
 56         request.put("Operation", operation);
 57         request.put("SearchIndex", "Books");
 58         request.put("Keywords", "rails");
 59         request.put("ResponseGroup", "ItemIds");
 60         request.put("Sort", "salesrank");
 61         //request.put("ResponseGroup", "SalesRank,Small" );
 62         request.put("ResponseGroup", "ItemIds" );
 63         return request;
 64     }
 65     //
 66     public void testGetAsinsBySoap() throws Exception {
 67         SoapRequest request = setupSoapRequest("ItemSearch");
 68         _Items[] items = request.invoke();
 69         for (int i=0; i<items.length; i++) {
 70             _Item[] item = items[i].getItem();
 71             for (int j=0; i<item.length; j++) {
 72                 assertTrue(item[j].getASIN() != null);
 73             }
 74         }
 75     }
 76
 77
 78     public void testGetAsinsByRest() throws Exception {
 79         RestRequest request = setupRestRequest("ItemSearch");
 80         URL u = new URL(request.toString());
 81         URLConnection connection = u.openConnection();
 82         InputStream in = connection.getInputStream();
 83         List<String> asins = parseAsins(in);
 84         for (String asin : asins) {
 85             System.out.println("asin: " + asin);
 86         }
 87     }
 88     private List<String> parseAsins(InputStream in) throws IOException, JDOMException {
 89         SAXBuilder builder = new SAXBuilder();
 90         Document doc = builder.build(in);
 91
 92         Format format = Format.getPrettyFormat();
 93         XMLOutputter out = new XMLOutputter(format);
 94         out.output(doc, System.out);
 95
 96         Element root = doc.getRootElement();
 97         Namespace ns = root.getNamespace();
 98         Element items = root.getChild( "Items", ns );
 99         Element request = items.getChild( "Request", ns );
100
101         // First make sure the response is valid
102
103         String isValid = request.getChild("IsValid", ns).getTextTrim();
104         if (!isValid.equals("True")){
105             throw new RuntimeException("Invalid response " + isValid);
106         }
107
108         // Now make sure there are no errors -- would be a good
109         // idea to collect and print them
110
111         Element errors = request.getChild("Errors", ns);
112
113         if( errors != null ){
114             throw new RuntimeException("One or more errors in the response " + errors);
115         }
116         int max = Integer.parseInt( items.getChild( "TotalResults", ns ).getTextTrim() );
117         List itemList = items.getChildren("Item", ns);
118         List<String> asins = new ArrayList<String>();
119         Iterator it = itemList.iterator();
120         while( it.hasNext()) {
121             Element item = (Element) it.next();
122             Element asinElement = item.getChild("ASIN", ns);
123             asins.add(asinElement.getTextTrim());
124             Element rankElement = item.getChild("SalesRank", ns);
125             if ( rankElement != null ) {
126                 int rank = extractRank( rankElement );
127                 String title = item.getChild( "ItemAttributes", ns ).getChild( "Title", ns ).getTextTrim();
128
129                 System.out.println( "Rank=" + rank + " Title=" + title );
130
131             }
132         }
133         return asins;
134     }
135
136     private int extractRank( Element e ) throws NumberFormatException {
137         String       rank = e.getTextTrim();
138         int          len = rank.length();
139         StringBuffer b = new StringBuffer( len );
140
141         for( int i = 0; i < len; ++i ){
142             char ch = rank.charAt( i );
143             if( ch == '.' || ch == ',' || ch == ' ' ) continue;
144             b.append( ch );
145         }
146
147         return Integer.parseInt( b.toString() );
148     }
149
150
151
152     public static void main(String[] args) {
153         junit.textui.TestRunner.run(EcsTest.class);
154     }
155 }
156
157 class SoapRequest extends HashMap<String, String> {
158     final String locale;
159     final String protocol;
160     final String subscriptionId;
161     final String associateTag;
162     final ItemSearchRequest itemSearchRequest;
163     final AWSECommerceServicePortType port;
164     SoapRequest(String locale, String protocol, String subscriptionId, String associateTag) throws MalformedURLException, ServiceException {
165         this.locale = locale;
166         this.protocol = protocol;
167         this.subscriptionId = subscriptionId;
168         this.associateTag = associateTag;
169         String urlTarget = new EcsUtils().getUrl("US", "soap");
170         this.itemSearchRequest = new ItemSearchRequest();
171         AWSECommerceService apd = new AWSECommerceServiceLocator();
172         this.port = apd.getAWSECommerceServicePort(new URL(urlTarget));
173     }
174     private void set(String key, Object value) throws NoSuchMethodException, IllegalAccessException, InvocationTargetException, InstantiationException {
175         if (value == null) return;
176         String name = "set" + key;
177         Method[] methods = itemSearchRequest.getClass().getMethods();
178         for (int i=0; i<methods.length; i++) {
179             if (methods[i].getName().equals(name) && methods[i].getParameterTypes().length == 1) {
180                 Class arg = methods[i].getParameterTypes()[0];
181                 if (arg == String[].class) {
182                     value = value.toString().split(",");
183                 } else if (!arg.isAssignableFrom(value.getClass())) {
184                     Constructor ctor = null;
185                     try {
186                         ctor = arg.getConstructor(value.getClass());
187                     } catch (NoSuchMethodException e) {
188                     }
189                     if (ctor != null) {
190                         value = ctor.newInstance(value);
191                     } else {
192                         throw new IllegalArgumentException("Failed to invoke setter for '" + name + "' with value '" + value + "' with signature " + methods[i] + " in " + itemSearchRequest.getClass().getName());
193                     }
194                 }
195                 methods[i].invoke(itemSearchRequest, value);
196                 return;
197             }
198         }
199         throw new IllegalArgumentException("Failed to find setter for '" + name + "' with value '" + value + "' in " + itemSearchRequest.getClass().getName());
200     }
201     //
202     public _Items[] invoke() throws NoSuchMethodException, IllegalAccessException, InvocationTargetException, InstantiationException, RemoteException {
203         for (Map.Entry<String, String> entry : this.entrySet()) {
204             if (entry.getValue() != null) {
205                 set(entry.getKey(), entry.getValue());
206             }
207         }
208         _ItemSearch itemSearchBody = new _ItemSearch();
209         itemSearchBody.setSubscriptionId(subscriptionId);
210         if (associateTag != null){
211             itemSearchBody.setAssociateTag(associateTag);
212         }
213         //itemSearchBody.setValidate(...);
214         itemSearchBody.setRequest(new ItemSearchRequest[] {itemSearchRequest});
215         return port.itemSearch(itemSearchBody).getItems();
216     }
217 }
218
219
220 class RestRequest extends HashMap<String, String> {
221     final String locale;
222     final String protocol;
223     final String subscriptionId;
224     final String associateTag;
225     final String urlTarget;
226     RestRequest(String locale, String protocol, String subscriptionId, String associateTag) {
227         this.locale = locale;
228         this.protocol = protocol;
229         this.subscriptionId = subscriptionId;
230         this.associateTag = associateTag;
231         this.urlTarget = new EcsUtils().getUrl("US", "xml");
232     }
233     public String toString() {
234         StringBuilder b = new StringBuilder(urlTarget);
235         //b.append("&SubscriptionId=");
236         b.append("&AWSAccessKeyId=");
237         b.append(subscriptionId);
238         if (associateTag != null){
239             b.append("&AssociateTag=");
240             b.append(associateTag);
241         }
242
243         for (Map.Entry<String, String> entry : this.entrySet()) {
244             b.append('&');
245             b.append(entry.getKey());
246             b.append('=');
247             try {
248                 b.append(URLEncoder.encode(entry.getValue(), "UTF8"));
249             } catch (UnsupportedEncodingException e) {
250                 throw new RuntimeException("Failed to encode '" + entry.getValue() + "'", e);
251             }
252         }
253         return b.toString();
254     }
255 }
256
257
258 class UrlUtils {
259     private static final String BASE_URL = "http://webservices.amazon.";
260     private static final String[] LOCALES = new String[] {"US", "UK", "DE", "JP", "CA", "FR"};
261     private static final String[] DOMAIN_SUFFIX = new String[] {"com", "co.uk", "de", "co.jp", "ca", "fr"};
262     private final Map<String, String> restUrls;
263     private final Map<String, String> soapUrls;
264
265     UrlUtils() {
266         soapUrls =createUrls("soap");
267         restUrls = createUrls("xml");
268     }
269     public String getUrl(String locale, String protocol) {
270         if ("xml".equalsIgnoreCase(protocol)) {
271             return restUrls.get(locale);
272         } else {
273             return soapUrls.get(locale);
274         }
275     }
276     private static Map<String, String> createUrls(String protocol) {
277         Map<String, String> map = new HashMap<String, String>();
278         for (int i=0; i<LOCALES.length; i++) {
279             map.put(LOCALES[i], BASE_URL + DOMAIN_SUFFIX[i] + "/onca/" +
280                     protocol + "?Service=AWSECommerceService");
281         }
282         return map;
283     }
284 }
285
286
287
288
289
290
291

No Comments

No comments yet.

RSS feed for comments on this post. TrackBack URL

Sorry, the comment form is closed at this time.

Powered by WordPress