Shahzad Bhatti Welcome to my ramblings and rants!

Amazon has recently announced limited beta availability of a new web service SimpleDB, which is somewhat similar to Amazon’s Dynamo service can store key-value based data. However, Amazon’s Dynamo service is not publicly available. The SimpleDB web service provides both REST and SOAP based APIs, though the REST APIs are really XML over HTTP and use simple query and RPC style parameters to create or retrieve data. You can use either GET or POST to submit HTTP requests. A key concept in SimpleDB are Domains which are comparable to the buckets in S3 and provides naming scope. Similar to S3, you are limited to the number of domains you can create. Inside domain, you can create rows of key/value pairs. The collection of key/value pair for a single row is called Item. When you save an item, you give it a name. Unlike a row in relational database, you can save items with varying number of of key/value pairs to the same domain. It is suggested that you use that name to retrieve the item. Though, SimpleDB also supports simple query language, but it only works if the query takes less than five seconds. SimpleDB is not meant to be used in a relational style, rather you are encouraged to store all related data you need with the item. However, there is limit on number of key/values you can save in an item (~255) , size of key/value (~1024) or total size of data in a single domain (~10 GB). All keys and values must be String. This also means in order for queries to work, the data must be stored in lexicographical format. Another big difference with the relational database is that SimpleDB is designed for high availability and it guarantees eventual consistency. Also, it provides transactions only at a single key/value pair. Thus, it is not suitable for applications that require higher level of consistency or transactions (See CAP principle or BASE transactions). On the other hand due to high availability, you can built highly scalable applications. Since the data is replicated across multiple servers, you may also get performance benefit by using multiple threads when reading a number of items concurrently.

Following are the core APIs that provide basic CRUD operations:

• Create/Delete/List domains
• Save attributes
• Update attributes that replace previously stored attributes
• Read attributes using Item name with option to query using attribute key/value pairs
• Delete attributes

I have been using SimpleDB for a few months for some internal applications at Amazon. Here I am showing a Java and Erlang client code to access SimpleDB. Since, the service is actually not available for public yet so you may not be able to try it.

Java Client

  1 import java.io.BufferedReader;
  2 import java.io.IOException;
  3 import java.io.InputStreamReader;
  4 import java.io.UnsupportedEncodingException;
  5 import java.net.HttpURLConnection;
  6 import java.net.MalformedURLException;
  7 import java.net.URL;
  8 import java.net.URLEncoder;
  9 import java.security.InvalidKeyException;
 10 import java.security.NoSuchAlgorithmException;
 11 import java.text.Collator;
 12 import java.text.FieldPosition;
 13 import java.text.NumberFormat;
 14 import java.text.SimpleDateFormat;
 15 import java.util.ArrayList;
 16 import java.util.Date;
 17 import java.util.HashMap;
 18 import java.util.List;
 19 import java.util.Map;
 20 import java.util.TreeMap;
 21
 22 import javax.crypto.Mac;
 23 import javax.crypto.spec.SecretKeySpec;
 24 import javax.xml.parsers.DocumentBuilder;
 25 import javax.xml.parsers.DocumentBuilderFactory;
 26 import javax.xml.parsers.ParserConfigurationException;
 27
 28 import org.w3c.dom.Document;
 29 import org.w3c.dom.Element;
 30 import org.w3c.dom.Node;
 31 import org.w3c.dom.NodeList;
 32 import org.xml.sax.InputSource;
 33 import org.xml.sax.SAXException;
 34
 35 /**
 36  * This class provides business delegate to access SDB web service using REST approach.
 37  *
 38  */
 39 public class SdbDelegate {
 40     private static final String DEFAULT_ENCODING = "UTF-8";
 41     private static final String HMAC_SHA1_ALGORITHM = "HmacSHA1";
 42     private static final String MORETOKEN_STRING = "MoreToken";
 43
 44     private final String accessId;
 45     private final String secretKey;
 46     private final String uri;
 47     private final String version;
 48
 49     public static class Response {
 50         public final Document rootDocument;
 51         public final Element rootElement;
 52         public final String requestId;
 53         public final String boxUsage;
 54         public final String moreToken;
 55         public final Object result;
 56         public Response(Document rootDocument, Element rootElement, String requestId, String boxUsage, String moreToken, Object result) {
 57             this.rootDocument = rootDocument;
 58             this.rootElement = rootElement;
 59             this.requestId = requestId;
 60             this.boxUsage = boxUsage;
 61             this.moreToken = moreToken;
 62             this.result = result;
 63         }
 64     }
 65
 66     public SdbDelegate() {
 67         this("yourkey", "yoursecret", "http://sdb.amazonaws.com", "2007-11-07");
 68     }
 69
 70
 71     public SdbDelegate(String accessId, String secretKey, String uri, String version) {
 72         this.accessId = accessId;
 73         this.secretKey = secretKey;
 74         this.uri = uri;
 75         this.version = version;
 76     }
 77
 78
 79     public void delete(String bucket, String name) {
 80         deleteAttributes(bucket, name);
 81     }
 82
 83     public void createDomain(String domainName) {
 84         Map<String,String> parameters = getBaseParameters("CreateDomain");
 85         parameters.put("DomainName",encode(domainName));
 86         invoke(parameters);
 87     }
 88
 89     public void deleteDomain(String domainName) {
 90         Map<String,String> parameters = getBaseParameters("DeleteDomain");
 91         parameters.put("DomainName",encode(domainName));
 92         invoke(parameters);
 93     }
 94
 95     public List<String> listDomains() {
 96         return listDomains(null);
 97     }
 98
 99     public List<String> listDomains(StringBuilder moreToken) {
100         return (List<String>) listDomains(moreToken, null);
101     }
102
103     public List<String> listDomains(StringBuilder moreToken, Integer maxResults) {
104         Map<String,String> parameters = getBaseParameters("ListDomains");
105         if (moreToken != null && moreToken.length() > 0) {
106             parameters.put(MORETOKEN_STRING, moreToken.toString());
107         }
108         if (maxResults != null) {
109             parameters.put("MaxResults",maxResults.toString());
110         }
111         Response response = invoke(parameters);
112         if (moreToken != null) {
113             moreToken.setLength(0);
114             moreToken.append(response.moreToken);
115         }
116
117         Element rootElement = (Element) response.rootElement;
118         NodeList domainNodes = rootElement.getElementsByTagName("DomainName");
119         List<String> domains = new ArrayList<String>();
120         for (int i = 0; i < domainNodes.getLength(); ++i) {
121             Node domainNode = domainNodes.item(i);
122             Node domainNameNode = domainNode.getFirstChild();
123             String domainName = domainNameNode.getNodeValue();
124             domains.add(domainName);
125         }
126
127         //
128         NodeList moreTokenNodes = rootElement.getElementsByTagName(MORETOKEN_STRING);
129         if (moreTokenNodes.getLength() > 0) {
130             Element moreTokenElement = (Element) moreTokenNodes.item(0);
131             moreTokenElement.normalize();
132             Node tokenNode = moreTokenElement.getFirstChild();
133             String newMoreToken = tokenNode.getNodeValue();
134             if (moreToken != null && (newMoreToken.compareToIgnoreCase("null") == 0 || newMoreToken.length() == 0)) {
135                 moreToken.setLength(0);
136             }
137         }
138         return domains;
139     }
140
141
142     public void deleteAttributes(String domain, String identifier) {
143         deleteAttributes(domain, identifier, new HashMap<String, String>());
144     }
145
146     /**
147      * Delete specified atrributes and/or values from item
148      *
149      * @param   attributes      set of attributes
150      * @return                  result of the operation
151      */
152     public void deleteAttributes(String domain, String identifier, Map<String, String> attributes) {
153         Map<String,String> parameters = getBaseParameters("DeleteAttributes");
154         parameters.put("ItemName",identifier);
155         parameters.put("DomainName",domain);
156         if (attributes != null) {
157             int i = 0;
158             for (Map.Entry<String, String> attr : attributes.entrySet()) {
159                 String name = "Attribute."+ i + ".Name";
160                 parameters.put(name,attr.getKey());
161                 if (attr.getValue() != null) {
162                     String value = "Attribute."+ i + ".Value";
163                     parameters.put(value, attr.getValue());
164                 }
165                 ++i;
166             }
167         }
168         invoke(parameters);
169     }
170
171
172     public void putAttributes(String domain, String identifier, Map<String, String> attributes) {
173         putAttributes(domain, identifier, attributes, true);
174     }
175
176
177     public void putAttributes(String domain, String identifier, Map<String, String> attributes, boolean replace) {
178         Map<String,String> parameters = getBaseParameters("PutAttributes");
179         parameters.put("ItemName", identifier);
180         parameters.put("DomainName", domain);
181         int i = 0;
182         if (attributes.size() > 255) throw new IllegalArgumentException("more than 255 attributes not supported");
183         System.out.println("Writing " + attributes.size() + " attributes for " + identifier);
184         for (Map.Entry<String, String> attr : attributes.entrySet()) {
185             if (attr.getKey().length() > 700) throw new IllegalArgumentException("name '" + attr.getKey() + "' more than 700 bytes");
186             if (attr.getValue().length() > 700) throw new IllegalArgumentException("value '" + attr.getValue() + "' of name '" + attr.getKey() + "' more than 700 bytes");
187             String name = "Attribute."+ i + ".Name";
188             parameters.put(name,attr.getKey());
189             String value = "Attribute."+ i + ".Value";
190             parameters.put(value, attr.getValue());
191             i++;
192         }
193         //
194         if (replace) {
195             parameters.put("Replace","true");
196         }
197         invoke(parameters);
198    }
199
200     public void replaceAttributes(String domain, String identifier, Map<String, String> attributes) {
201          putAttributes(domain, identifier, attributes,true);
202     }
203
204
205     public Map<String, String> getAttributes(String domain, String identifier){
206         return getAttributes(domain, identifier, new HashMap<String, String>());
207     }
208
209     public Map<String, String> getAttributes(String domain, String identifier, Map<String, String> attributes) {
210         String NAME_STRING = "Name";
211         String VALUE_STRING = "Value";
212         String ATTRIBUTE_STRING = "Attribute";
213
214         Map<String,String> parameters =  getBaseParameters("GetAttributes");
215         parameters.put("ItemName", identifier);
216         parameters.put("DomainName",domain);
217
218         int k = 0;
219         for (Map.Entry<String, String> attr : attributes.entrySet()) {
220             String name = "AttributeName."+k;
221             parameters.put(name,attr.getKey());
222             ++k;
223         }
224
225         //
226         Response response =  invoke(parameters);
227         Element rootElement = (Element) response.rootElement;
228         NodeList attributeNodes = rootElement.getElementsByTagName(ATTRIBUTE_STRING);
229         Map<String, String> returnAttributes = new TreeMap<String, String>(String.CASE_INSENSITIVE_ORDER);
230         //
231         for (int i = 0; i < attributeNodes.getLength(); ++i) {
232             Node attributeNode = attributeNodes.item(i);
233
234             Element attributeElement = (Element)attributeNode;
235
236             NodeList nameNodes = attributeElement.getElementsByTagName(NAME_STRING);
237             Node nameNode = nameNodes.item(0);
238             nameNode.normalize();
239             Node attributeNameNode = nameNode.getFirstChild();
240             String attributeName = attributeNameNode.getNodeValue();
241             NodeList valueNodes = attributeElement.getElementsByTagName(VALUE_STRING);
242             Node valueNode = valueNodes.item(0);
243             valueNode.normalize();
244             Node attributeValueNode = valueNode.getFirstChild();
245             String attributeValue = attributeValueNode.getNodeValue();
246             returnAttributes.put(attributeName,attributeValue);
247         }
248         return returnAttributes;
249     }
250
251
252     protected Response invoke(Map<String,String> parameters) {
253         try {
254             Document document = getResponse(parameters);
255             Element rootElement = document.getDocumentElement();
256             Element responseStatus = (Element) rootElement.getElementsByTagName("ResponseStatus").item(0);
257             Node requestIdNode = responseStatus.getElementsByTagName("RequestId").item(0);
258             String requestId = null;
259             if (requestIdNode != null && requestIdNode.getFirstChild() != null) {
260                 requestId = requestIdNode.getFirstChild().getNodeValue();
261             }
262             String boxUsage = null;
263             Node boxUsagesNode = responseStatus.getElementsByTagName("BoxUsage").item(0);
264             if (boxUsagesNode != null && boxUsagesNode.getFirstChild() != null) {
265                 boxUsage = boxUsagesNode.getFirstChild().getNodeValue();
266             }
267             return new Response(document, rootElement, requestId, boxUsage, null, null);
268         } catch (RuntimeException e) {
269             throw e;
270         } catch (Exception e) {
271             throw new RuntimeException(e);
272         }
273     }
274
275
276     private static String getSignature(String data, String key) throws NoSuchAlgorithmException, InvalidKeyException {
277         SecretKeySpec signingKey = new SecretKeySpec(key.getBytes(),HMAC_SHA1_ALGORITHM);
278         Mac mac = Mac.getInstance(HMAC_SHA1_ALGORITHM);
279         mac.init(signingKey);
280         byte[] rawHmac = mac.doFinal(data.getBytes());
281         return new sun.misc.BASE64Encoder().encode(rawHmac);
282     }
283
284     private Map<String,String> getBaseParameters(String action) {
285         Collator myCollator = Collator.getInstance();
286         // Note that these parameters must be sent in order
287         Map<String,String> baseParameters = new TreeMap<String,String>(myCollator);
288         baseParameters.put("Action", action);
289         baseParameters.put("Version", version);
290         baseParameters.put("AWSAccessKeyId", accessId);
291         baseParameters.put("SignatureVersion", "1");
292
293         Date curTime = new Date();
294         SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ");
295         StringBuffer timeBuffer = new StringBuffer();
296         FieldPosition fp = new FieldPosition(NumberFormat.INTEGER_FIELD);
297         dateFormat.format(curTime, timeBuffer, fp);
298         timeBuffer.insert(timeBuffer.length()-2, ":");
299
300         baseParameters.put("Timestamp",timeBuffer.toString());
301         return baseParameters;
302     }
303
304     private Document getResponse(Map<String,String> parameters) throws UnsupportedEncodingException, MalformedURLException, IOException, NoSuchAlgorithmException, SAXException, ParserConfigurationException, InvalidKeyException {
305         String path = uri + getUrlPath(parameters);
306         URL url = new URL(path);
307         HttpURLConnection connection = (HttpURLConnection) url.openConnection();
308         connection.setDoOutput(true);
309         connection.setRequestMethod("GET");
310         connection.connect();
311
312         System.out.println("Connecting to " + url);
313         int respCode = connection.getResponseCode();
314
315         if (respCode == HttpURLConnection.HTTP_OK) {
316             InputSource inputSource = new InputSource(new BufferedReader(new InputStreamReader(connection.getInputStream())));
317             DocumentBuilderFactory documentBuilderFactory = DocumentBuilderFactory.newInstance();
318             DocumentBuilder documentBuilder = documentBuilderFactory.newDocumentBuilder();
319             return documentBuilder.parse(inputSource);
320         } else {
321             InputSource inputSource = new InputSource(new BufferedReader(new InputStreamReader(connection.getErrorStream())));
322             DocumentBuilderFactory documentBuilderFactory = DocumentBuilderFactory.newInstance();
323             DocumentBuilder documentBuilder = documentBuilderFactory.newDocumentBuilder();
324             Document document = documentBuilder.parse(inputSource);
325             Element rootElement = document.getDocumentElement();
326             Node requestIdNode = rootElement.getElementsByTagName("RequestID").item(0);
327             StringBuilder errorMessage = new StringBuilder("Failed to invoke input request '" + path + "'");
328
329             //
330             if (requestIdNode != null && requestIdNode.getFirstChild() != null) {
331                 errorMessage.append(" with requestId " + requestIdNode.getFirstChild().getNodeValue());
332                 Element errors = (Element) rootElement.getElementsByTagName("Errors").item(0);
333                 if (errors != null) {
334                     Element error = (Element) errors.getElementsByTagName("Error").item(0);
335                     if (error != null) {
336                         Node messageNode = error.getElementsByTagName("Message").item(0);
337                         if (messageNode != null) errorMessage.append(" due to " + messageNode.getFirstChild().getNodeValue());
338                     }
339                 }
340             }
341             throw new RuntimeException(errorMessage.toString());
342         }
343     }
344
345
346    private String encode(String name){
347         try {
348              return URLEncoder.encode(name, DEFAULT_ENCODING);
349         } catch (UnsupportedEncodingException ex) {
350             throw new RuntimeException("Failed to encode '" + name + "'");
351         }
352     }
353
354
355     private String getUrlPath(Map<String,String> parameters) throws UnsupportedEncodingException, NoSuchAlgorithmException, InvalidKeyException {
356         StringBuilder queryString = new StringBuilder("?");
357         StringBuilder signatureData = new StringBuilder();
358         boolean first = true;
359         for (Map.Entry<String, String> param : parameters.entrySet()) {
360             if (first) {
361                 first = false;
362             } else {
363                 queryString.append("&");
364             }
365
366             signatureData.append(param.getKey() + param.getValue());
367             queryString.append(param.getKey() + "=" + URLEncoder.encode((String)param.getValue(), DEFAULT_ENCODING));
368         }
369
370         queryString.append("&Signature=" + URLEncoder.encode(getSignature(signatureData.toString(),secretKey), DEFAULT_ENCODING));
371         return queryString.toString();
372     }
373 }
374

Java Client Test

 1 import junit.framework.TestCase;
 2 import junit.framework.Test;
 3 import junit.framework.TestSuite;
 4
 5 import java.util.List;
 6 import java.util.Map;
 7 import java.util.HashMap;
 8
 9 public class SdbDelegateTest extends TestCase {
10     private SdbDelegate fixture;
11     private String domainName;
12
13     public void setUp() {
14         this.fixture = new SdbDelegate();
15         this.domainName = "TestDomain";
16     }
17
18     public void testCreateDomain() {
19         fixture.createDomain(domainName);
20         assertTrue(fixture.listDomains().contains(domainName));
21     }
22
23     public void testDeleteDomain() {
24         fixture.deleteDomain(domainName);
25         assertFalse(fixture.listDomains().contains(domainName));
26     }
27
28
29     public void testGetPutAttributes() {
30         fixture.createDomain(domainName);
31         Map<String, String> attributes = new HashMap<String, String>();
32         attributes.put("StreetAddress", "705 5th Ave");
33         attributes.put("City", "Seattle");
34         attributes.put("State", "WA");
35         attributes.put("Zip", "98101");
36         fixture.putAttributes(domainName, "TCC", attributes, true);
37         Map<String, String> attributes2 = fixture.getAttributes(domainName, "TCC");
38         System.out.println("Retrieved " + attributes2);
39         assertEquals(attributes, attributes2);
40     }
41
42     public static Test suite() {
43         TestSuite suite = new TestSuite();
44         suite.addTestSuite(SdbDelegateTest.class);
45         return suite;
46     }
47
48     public static void main(String[] args) {
49         junit.textui.TestRunner.run(suite());
50     }
51 }
52

Erlang Client

  1 %%%-------------------------------------------------------------------
  2 %%% @author Shahzad Bhatti <bhatti@plexobject.com>
  3 %%% @doc
  4 %%%   Business delegate to access SimpleDB
  5 %%% @end
  6 %%%-------------------------------------------------------------------
  7 -module(sdb_delegate).
  8 -include_lib("xmerl/include/xmerl.hrl").
  9
 10
 11 %%%-------------------------------------------------------------------
 12 %%% Public APIs
 13 %%%-------------------------------------------------------------------
 14 -export([list_domains/0, list_domains/1, list_domains/2, delete_domain/1, get_attributes/2, get_attributes/3, create_domain/1, put_attributes/3, put_attributes/4, delete_item/2, delete_attributes/2, delete_attributes/3, replace_attributes/3]).
 15
 16
 17 %%%-------------------------------------------------------------------
 18 %%% Test Methods
 19 %%%-------------------------------------------------------------------
 20 -export([test/0]).
 21
 22
 23 %%%-------------------------------------------------------------------
 24 %%% Configuration Functions %%%
 25 %%%-------------------------------------------------------------------
 26 uri() ->
 27    "http://sdb.amazonaws.com?".
 28
 29
 30 version() ->
 31    "2007-11-07".
 32
 33 access_key() ->
 34    "yourkey".
 35
 36 secret_key() ->
 37     "secretkey".
 38
 39
 40
 41 %%%-------------------------------------------------------------------
 42 %%% Public APIs to access SimpleDB 
 43 %%%-------------------------------------------------------------------
 44 start() ->
 45     crypto:start(),
 46     inets:start().
 47
 48 stop() ->
 49     init:stop().
 50
 51
 52 list_domains() ->
 53     list_domains(nil, nil).
 54 list_domains(MoreToken) ->
 55     list_domains(MoreToken, nil).
 56
 57 list_domains(MoreToken, MaxResults) ->
 58     Base = base_parameters("ListDomains"),
 59     Base1 = if MoreToken == nil -> Base; true -> Base ++ [["MoreToken", MoreToken]] end,
 60     Base2 = if MaxResults == nil -> Base1; true -> Base1 ++ [["MaxResults", MaxResults]] end,
 61     Xml = rest_request(Base2),
 62     xml_values(xmerl_xpath:string("//DomainName/text()", Xml)).
 63
 64
 65 create_domain(Domain) ->
 66     Base = [["DomainName", url_encode(Domain)]|
 67                 base_parameters("CreateDomain")],
 68     rest_request(Base).
 69
 70 delete_domain(Domain) ->
 71     Base = [["DomainName", url_encode(Domain)]|
 72                 base_parameters("DeleteDomain")],
 73     rest_request(Base).
 74
 75 replace_attributes(Domain, ItemName, Attributes) ->
 76     put_attributes(Domain, ItemName, Attributes, true).
 77 put_attributes(Domain, ItemName, Attributes) ->
 78     put_attributes(Domain, ItemName, Attributes, false).
 79 put_attributes(Domain, ItemName, Attributes, Replace) ->
 80     Base = [["DomainName", url_encode(Domain)],
 81             ["ItemName", url_encode(ItemName)]|
 82                 base_parameters("PutAttributes")] ++
 83                 encode_attributes(Attributes),
 84     Base1 = if Replace == false -> Base; true -> Base ++ [["Replace", "true"]] end,
 85     rest_request(Base1).
 86
 87
 88 delete_item(Domain, ItemName) ->
 89     delete_attributes(Domain, ItemName).
 90
 91
 92 delete_attributes(Domain, ItemName) ->
 93     delete_attributes(Domain, ItemName, nil).
 94 delete_attributes(Domain, ItemName, AttributeNames) ->
 95     Base = [["DomainName", url_encode(Domain)],
 96             ["ItemName", url_encode(ItemName)]|
 97                 base_parameters("DeleteAttributes")] ++
 98                 encode_attribute_names(AttributeNames),
 99     rest_request(Base).
100
101 get_attributes(Domain, ItemName) ->
102     get_attributes(Domain, ItemName, nil).
103 get_attributes(Domain, ItemName, AttributeNames) ->
104     Base = [["DomainName", url_encode(Domain)],
105             ["ItemName", url_encode(ItemName)]|
106                 base_parameters("GetAttributes")] ++
107                 encode_attribute_names(AttributeNames),
108     Xml = rest_request(Base),
109     xml_names_values(xmerl_xpath:string("//Attribute", Xml)).
110
111
112 %%%-------------------------------------------------------------------
113 %%% Private Functions
114 %%%-------------------------------------------------------------------
115
116 rest_request(Params) ->
117     Url = uri() ++ query_string(Params),
118     Response = http:request(Url),
119     case Response of
120         {ok, {{_HttpVersion, StatusCode, _ErrorMessage}, _Headers, Body }} ->
121             error_logger:info_msg("URL ~p Status ~p~n", [Url, StatusCode]),
122             {Xml, _Rest} = xmerl_scan:string(Body),
123             %%%error_logger:info_msg("Xml ~p~n", [Xml]),
124             case StatusCode of
125                 200 ->
126                     Xml;
127                 _ ->
128                    Error = xml_values(xmerl_xpath:string("//Message/text()", Xml)),
129                    throw({Error})
130             end;
131         {error, Message} ->
132             case Message of
133                 timeout ->
134                     io:format("URL ~p Timedout, retrying~n", [Url]),
135                     sleep(1000),
136                     rest_request(Params);
137                 true ->
138                     throw({Message})
139             end
140     end.
141
142
143 query_string(Params) ->
144     Params1 = lists:sort(
145         fun([Elem1, _], [Elem2, _]) ->
146             string:to_lower(Elem1) > string:to_lower(Elem2) end,
147         Params),
148     {QueryStr, SignatureData} =
149         lists:foldr(fun query_string/2, {"", ""}, Params1),
150     QueryStr ++ "Signature=" ++ url_encode(signature(SignatureData)).
151
152
153 query_string([Key, Value], {QueryStr, SignatureData}) ->
154     QueryStr1 = QueryStr ++ Key ++ "=" ++ url_encode(Value) ++ "&",
155     SignatureData1 = SignatureData ++ Key ++ Value,
156     {QueryStr1, SignatureData1}.
157
158
159 encode_attributes(Attributes) when Attributes == nil ->
160     [];
161 encode_attributes(Attributes) ->
162     {Encoded, _} = lists:foldr(fun encode_attributes/2, {[], 0}, Attributes),
163     Encoded.
164
165 encode_attributes([Key, Value], {Encoded, I}) ->
166     KeyName = "Attribute." ++ integer_to_list(I) ++ ".Name",
167     KeyValue = "Attribute." ++ integer_to_list(I) ++ ".Value",
168     {[[KeyName, Key], [KeyValue, Value]|Encoded], I+1}.
169
170
171 encode_attribute_names(Attributes) when Attributes == nil ->
172     [];
173 encode_attribute_names(Attributes) ->
174     {Encoded, _} = lists:foldr(fun encode_attribute_names/2, {[], 0}, Attributes),
175     Encoded.
176
177 encode_attribute_names(Key, {Encoded, I}) ->
178     KeyName = "Attribute." ++ integer_to_list(I) ++ ".Name",
179     {[[KeyName, Key]|Encoded], I+1}.
180
181
182
183
184 %%%
185 % Converts a number into 2-digit 
186 %%%
187 two_digit(X) when is_integer(X), X >= 10 ->
188     integer_to_list(X);
189 two_digit(X) when is_integer(X), X < 10 ->
190     "0" ++ integer_to_list(X).
191
192 abs_two_digit(X) when X < 0 ->
193     two_digit(0-X);
194 abs_two_digit(X) when X >= 0 ->
195     two_digit(X).
196
197 %%%
198 % Returns Coordinated Universal Time (Greenwich Mean Time) time zone,
199 %%%
200 timestamp() ->
201     {{_, _, _}, {_LocalHour, _LocalMin, _}} = LocalDateTime = calendar:local_time(),
202     [{{Year, Month, Day}, {Hour, Min, Sec}}] =
203         calendar:local_time_to_universal_time_dst(LocalDateTime),
204     Z = gmt_difference(),
205     integer_to_list(Year) ++ "-" ++ two_digit(Month) ++ "-" ++ two_digit(Day)
206         ++ "T" ++ two_digit(Hour) ++ ":" ++ two_digit(Min) ++ ":" ++
207         two_digit(Sec) ++ Z.
208
209
210 gmt_difference() ->
211     "-08:00".
212
213
214
215 %%%
216 % Returns HMAC encoded access key
217 %%%
218 signature(Data) ->
219     hmac(secret_key(), Data).
220
221 hmac(SecretKey, Data) ->
222     http_base_64:encode(
223           binary_to_list(crypto:sha_mac(SecretKey, Data))).
224
225
226 base_parameters(Action) ->
227     [["Action", Action],
228      ["AWSAccessKeyId", access_key()],
229      ["Version", version()],
230      ["SignatureVersion", "1"],
231      ["Timestamp", timestamp()]].
232
233
234
235 %%%
236 % This method retrieves node value from the XML records that are returned
237 % after scanning tags.
238 %%%
239 xml_values(List) ->
240     lists:foldr(fun xml_values/2, [], List).
241
242 xml_values(#xmlText{value=Value}, List) ->
243     [Value|List].
244
245
246 xml_names_values(List) ->
247     lists:foldr(fun xml_names_values/2, [], List).
248
249 xml_names_values(Xml, List) ->
250     [ #xmlText{value=Name} ]  = xmerl_xpath:string("//Name/text()", Xml),
251     [ #xmlText{value=Value} ]  = xmerl_xpath:string("//Value/text()", Xml),
252     [[Name, Value]|List].
253
254
255 sleep(T) ->
256     receive
257     after T ->
258        true
259     end.
260
261
262 %%%
263 % URL encode - borrowed from CouchDB
264 %%%
265 url_encode([H|T]) ->
266     if
267         H >= $a, $z >= H ->
268             [H|url_encode(T)];
269         H >= $A, $Z >= H ->
270             [H|url_encode(T)];
271         H >= $0, $9 >= H ->
272             [H|url_encode(T)];
273         H == $_; H == $.; H == $-; H == $: ->
274             [H|url_encode(T)];
275         true ->
276             case lists:flatten(io_lib:format("~.16.0B", [H])) of
277                 [X, Y] ->
278                     [$%, X, Y | url_encode(T)];
279                 [X] ->
280                     [$%, $0, X | url_encode(T)]
281             end
282     end;
283 url_encode([]) ->
284     [].
285
286
287 %%%-------------------------------------------------------------------
288 %%% Test Functions
289 %%%-------------------------------------------------------------------
290
291 test_list_domains() ->
292     list_domains().
293
294 test_create_domain() ->
295     create_domain("TestDomain"),
296     ["TestDomain"] = lists:filter(
297                         fun(Elem) -> Elem == "TestDomain" end,
298                         list_domains()).
299
300
301 test_delete_domain() ->
302     delete_domain("TestDomain"),
303     [] = lists:filter(
304                         fun(Elem) -> Elem == "TestDomain" end,
305                         list_domains()).
306
307 test_put_get_attributes() ->
308     create_domain("TestDomain"),
309     Attributes = lists:sort([
310         ["StreetAddress", "705 5th Ave"],
311         ["City", "Seattle"],
312         ["State", "WA"],
313         ["Zip", "98101"]
314         ]),
315     put_attributes("TestDomain", "TCC", Attributes),
316     Attributes = lists:sort(get_attributes("TestDomain", "TCC")).
317
318 test_put_delete_attributes() ->
319     create_domain("TestDomain"),
320     Attributes = lists:sort([
321         ["StreetAddress", "705 5th Ave"],
322         ["City", "Seattle"],
323         ["State", "WA"],
324         ["Zip", "98101"]
325         ]),
326     put_attributes("TestDomain", "TCC", Attributes),
327     delete_attributes("TestDomain", "TCC"),
328     sleep(1000),        %% Let it sync
329     [] = get_attributes("TestDomain", "TCC").
330
331 test() ->
332     start(),
333     test_create_domain(),
334     test_delete_domain(),
335     test_put_get_attributes(),
336     test_put_delete_attributes(),
337     stop().
338

Note that in Erlang, I am hard coding time-zone difference, I wish Erlang had better support for time-zones, and URL encoding for UTF8.

Another word of caution, in order to use this code for real applications, you will need better handle on error processing especially because web services may timeout and you will need to retry.

You can download the code from here:

http://weblog.plexobject.com/SdbDelegate.java

http://weblog.plexobject.com/SdbDelegateTest.java

http://weblog.plexobject.com/sdb_delegate.erl

I am also adding the Erlang code to http://code.google.com/p/erlsdb/.

Further References:

http://awsdocs.s3.amazonaws.com/SDB/2007-11-07/sdb-dg-2007-11-07.pdf?

http://docs.amazonwebservices.com/AmazonSimpleDB/2007-11-07/DeveloperGuide/?

http://aws.amazon.com/simpledb

http://www.informationweek.com/news/showArticle.jhtml?articleID=204803008

You can send me comments or suggestions to improve this code at bhatti AT plexobject DOT com.

Update (Dec 17, 2007): I saw some comments on how unRESTful SimpleDB’s REST APIs are and I agree. Here is my attempt to RESTify them:

Create Domain
PUT http://sdb.amazonaws.com/DomainName

I am using PUT because I am defining the URL and per REST specs I should use PUT instead of POST. Also, I am relying on HTTP headers for authentication information.

Delete Domain
DELETE http://sdb.amazonaws.com/DomainName

List Domains
GET http://sdb.amazonaws.com?MaxDomains=XX&NextToken=YY
I am specifying parameters explicitly rather than using changing URI

Put Attributes
POST http://sdb.amazonaws.com/DomainName/ItemName
—POST BODY-BEGIN—
attributeName=attributeValue&…
—POST BODY-END—

I am specifying attributes as part of the POST body

Delete Attributes
DELETE http://sdb.amazonaws.com/DomainName/ItemName?attributeName1&…

Get Attributes
GET http://sdb.amazonaws.com/DomainName/ItemName?attributeName1&…

Query Items
GET http://sdb.amazonaws.com/DomainName/ItemName?queryExpression=…

Note that these URLs are based on Sam Ruby’s book on REST APIs. Also, Rails uses similar URLs for RESTful controllers.