Amazon has recently announced limited beta availability of a new web service SimpleDB, which is somewhat similar to Amazon’s Dynamo service can store key-value based data. However, Amazon’s Dynamo service is not publicly available. The SimpleDB web service provides both REST and SOAP based APIs, though the REST APIs are really XML over HTTP and use simple query and RPC style parameters to create or retrieve data. You can use either GET or POST to submit HTTP requests. A key concept in SimpleDB are Domains which are comparable to the buckets in S3 and provides naming scope. Similar to S3, you are limited to the number of domains you can create. Inside domain, you can create rows of key/value pairs. The collection of key/value pair for a single row is called Item. When you save an item, you give it a name. Unlike a row in relational database, you can save items with varying number of of key/value pairs to the same domain. It is suggested that you use that name to retrieve the item. Though, SimpleDB also supports simple query language, but it only works if the query takes less than five seconds. SimpleDB is not meant to be used in a relational style, rather you are encouraged to store all related data you need with the item. However, there is limit on number of key/values you can save in an item (~255) , size of key/value (~1024) or total size of data in a single domain (~10 GB). All keys and values must be String. This also means in order for queries to work, the data must be stored in lexicographical format. Another big difference with the relational database is that SimpleDB is designed for high availability and it guarantees eventual consistency. Also, it provides transactions only at a single key/value pair. Thus, it is not suitable for applications that require higher level of consistency or transactions (See CAP principle or BASE transactions). On the other hand due to high availability, you can built highly scalable applications. Since the data is replicated across multiple servers, you may also get performance benefit by using multiple threads when reading a number of items concurrently.
Following are the core APIs that provide basic CRUD operations:
- Create/Delete/List domains
- Save attributes
- Update attributes that replace previously stored attributes
- Read attributes using Item name with option to query using attribute key/value pairs
- Delete attributes
I have been using SimpleDB for a few months for some internal applications at Amazon. Here I am showing a Java and Erlang client code to access SimpleDB. Since, the service is actually not available for public yet so you may not be able to try it.
Java Client
1 import java.io.BufferedReader;
2 import java.io.IOException;
3 import java.io.InputStreamReader;
4 import java.io.UnsupportedEncodingException;
5 import java.net.HttpURLConnection;
6 import java.net.MalformedURLException;
7 import java.net.URL;
8 import java.net.URLEncoder;
9 import java.security.InvalidKeyException;
10 import java.security.NoSuchAlgorithmException;
11 import java.text.Collator;
12 import java.text.FieldPosition;
13 import java.text.NumberFormat;
14 import java.text.SimpleDateFormat;
15 import java.util.ArrayList;
16 import java.util.Date;
17 import java.util.HashMap;
18 import java.util.List;
19 import java.util.Map;
20 import java.util.TreeMap;
21
22 import javax.crypto.Mac;
23 import javax.crypto.spec.SecretKeySpec;
24 import javax.xml.parsers.DocumentBuilder;
25 import javax.xml.parsers.DocumentBuilderFactory;
26 import javax.xml.parsers.ParserConfigurationException;
27
28 import org.w3c.dom.Document;
29 import org.w3c.dom.Element;
30 import org.w3c.dom.Node;
31 import org.w3c.dom.NodeList;
32 import org.xml.sax.InputSource;
33 import org.xml.sax.SAXException;
34
35 /**
36 * This class provides business delegate to access SDB web service using REST approach.
37 *
38 */
39 public class SdbDelegate {
40 private static final String DEFAULT_ENCODING = "UTF-8";
41 private static final String HMAC_SHA1_ALGORITHM = "HmacSHA1";
42 private static final String MORETOKEN_STRING = "MoreToken";
43
44 private final String accessId;
45 private final String secretKey;
46 private final String uri;
47 private final String version;
48
49 public static class Response {
50 public final Document rootDocument;
51 public final Element rootElement;
52 public final String requestId;
53 public final String boxUsage;
54 public final String moreToken;
55 public final Object result;
56 public Response(Document rootDocument, Element rootElement, String requestId, String boxUsage, String moreToken, Object result) {
57 this.rootDocument = rootDocument;
58 this.rootElement = rootElement;
59 this.requestId = requestId;
60 this.boxUsage = boxUsage;
61 this.moreToken = moreToken;
62 this.result = result;
63 }
64 }
65
66 public SdbDelegate() {
67 this("yourkey", "yoursecret", "http://sdb.amazonaws.com", "2007-11-07");
68 }
69
70
71 public SdbDelegate(String accessId, String secretKey, String uri, String version) {
72 this.accessId = accessId;
73 this.secretKey = secretKey;
74 this.uri = uri;
75 this.version = version;
76 }
77
78
79 public void delete(String bucket, String name) {
80 deleteAttributes(bucket, name);
81 }
82
83 public void createDomain(String domainName) {
84 Map<String,String> parameters = getBaseParameters("CreateDomain");
85 parameters.put("DomainName",encode(domainName));
86 invoke(parameters);
87 }
88
89 public void deleteDomain(String domainName) {
90 Map<String,String> parameters = getBaseParameters("DeleteDomain");
91 parameters.put("DomainName",encode(domainName));
92 invoke(parameters);
93 }
94
95 public List<String> listDomains() {
96 return listDomains(null);
97 }
98
99 public List<String> listDomains(StringBuilder moreToken) {
100 return (List<String>) listDomains(moreToken, null);
101 }
102
103 public List<String> listDomains(StringBuilder moreToken, Integer maxResults) {
104 Map<String,String> parameters = getBaseParameters("ListDomains");
105 if (moreToken != null && moreToken.length() > 0) {
106 parameters.put(MORETOKEN_STRING, moreToken.toString());
107 }
108 if (maxResults != null) {
109 parameters.put("MaxResults",maxResults.toString());
110 }
111 Response response = invoke(parameters);
112 if (moreToken != null) {
113 moreToken.setLength(0);
114 moreToken.append(response.moreToken);
115 }
116
117 Element rootElement = (Element) response.rootElement;
118 NodeList domainNodes = rootElement.getElementsByTagName("DomainName");
119 List<String> domains = new ArrayList<String>();
120 for (int i = 0; i < domainNodes.getLength(); ++i) {
121 Node domainNode = domainNodes.item(i);
122 Node domainNameNode = domainNode.getFirstChild();
123 String domainName = domainNameNode.getNodeValue();
124 domains.add(domainName);
125 }
126
127 //
128 NodeList moreTokenNodes = rootElement.getElementsByTagName(MORETOKEN_STRING);
129 if (moreTokenNodes.getLength() > 0) {
130 Element moreTokenElement = (Element) moreTokenNodes.item(0);
131 moreTokenElement.normalize();
132 Node tokenNode = moreTokenElement.getFirstChild();
133 String newMoreToken = tokenNode.getNodeValue();
134 if (moreToken != null && (newMoreToken.compareToIgnoreCase("null") == 0 || newMoreToken.length() == 0)) {
135 moreToken.setLength(0);
136 }
137 }
138 return domains;
139 }
140
141
142 public void deleteAttributes(String domain, String identifier) {
143 deleteAttributes(domain, identifier, new HashMap<String, String>());
144 }
145
146 /**
147 * Delete specified atrributes and/or values from item
148 *
149 * @param attributes set of attributes
150 * @return result of the operation
151 */
152 public void deleteAttributes(String domain, String identifier, Map<String, String> attributes) {
153 Map<String,String> parameters = getBaseParameters("DeleteAttributes");
154 parameters.put("ItemName",identifier);
155 parameters.put("DomainName",domain);
156 if (attributes != null) {
157 int i = 0;
158 for (Map.Entry<String, String> attr : attributes.entrySet()) {
159 String name = "Attribute."+ i + ".Name";
160 parameters.put(name,attr.getKey());
161 if (attr.getValue() != null) {
162 String value = "Attribute."+ i + ".Value";
163 parameters.put(value, attr.getValue());
164 }
165 ++i;
166 }
167 }
168 invoke(parameters);
169 }
170
171
172 public void putAttributes(String domain, String identifier, Map<String, String> attributes) {
173 putAttributes(domain, identifier, attributes, true);
174 }
175
176
177 public void putAttributes(String domain, String identifier, Map<String, String> attributes, boolean replace) {
178 Map<String,String> parameters = getBaseParameters("PutAttributes");
179 parameters.put("ItemName", identifier);
180 parameters.put("DomainName", domain);
181 int i = 0;
182 if (attributes.size() > 255) throw new IllegalArgumentException("more than 255 attributes not supported");
183 System.out.println("Writing " + attributes.size() + " attributes for " + identifier);
184 for (Map.Entry<String, String> attr : attributes.entrySet()) {
185 if (attr.getKey().length() > 700) throw new IllegalArgumentException("name '" + attr.getKey() + "' more than 700 bytes");
186 if (attr.getValue().length() > 700) throw new IllegalArgumentException("value '" + attr.getValue() + "' of name '" + attr.getKey() + "' more than 700 bytes");
187 String name = "Attribute."+ i + ".Name";
188 parameters.put(name,attr.getKey());
189 String value = "Attribute."+ i + ".Value";
190 parameters.put(value, attr.getValue());
191 i++;
192 }
193 //
194 if (replace) {
195 parameters.put("Replace","true");
196 }
197 invoke(parameters);
198 }
199
200 public void replaceAttributes(String domain, String identifier, Map<String, String> attributes) {
201 putAttributes(domain, identifier, attributes,true);
202 }
203
204
205 public Map<String, String> getAttributes(String domain, String identifier){
206 return getAttributes(domain, identifier, new HashMap<String, String>());
207 }
208
209 public Map<String, String> getAttributes(String domain, String identifier, Map<String, String> attributes) {
210 String NAME_STRING = "Name";
211 String VALUE_STRING = "Value";
212 String ATTRIBUTE_STRING = "Attribute";
213
214 Map<String,String> parameters = getBaseParameters("GetAttributes");
215 parameters.put("ItemName", identifier);
216 parameters.put("DomainName",domain);
217
218 int k = 0;
219 for (Map.Entry<String, String> attr : attributes.entrySet()) {
220 String name = "AttributeName."+k;
221 parameters.put(name,attr.getKey());
222 ++k;
223 }
224
225 //
226 Response response = invoke(parameters);
227 Element rootElement = (Element) response.rootElement;
228 NodeList attributeNodes = rootElement.getElementsByTagName(ATTRIBUTE_STRING);
229 Map<String, String> returnAttributes = new TreeMap<String, String>(String.CASE_INSENSITIVE_ORDER);
230 //
231 for (int i = 0; i < attributeNodes.getLength(); ++i) {
232 Node attributeNode = attributeNodes.item(i);
233
234 Element attributeElement = (Element)attributeNode;
235
236 NodeList nameNodes = attributeElement.getElementsByTagName(NAME_STRING);
237 Node nameNode = nameNodes.item(0);
238 nameNode.normalize();
239 Node attributeNameNode = nameNode.getFirstChild();
240 String attributeName = attributeNameNode.getNodeValue();
241 NodeList valueNodes = attributeElement.getElementsByTagName(VALUE_STRING);
242 Node valueNode = valueNodes.item(0);
243 valueNode.normalize();
244 Node attributeValueNode = valueNode.getFirstChild();
245 String attributeValue = attributeValueNode.getNodeValue();
246 returnAttributes.put(attributeName,attributeValue);
247 }
248 return returnAttributes;
249 }
250
251
252 protected Response invoke(Map<String,String> parameters) {
253 try {
254 Document document = getResponse(parameters);
255 Element rootElement = document.getDocumentElement();
256 Element responseStatus = (Element) rootElement.getElementsByTagName("ResponseStatus").item(0);
257 Node requestIdNode = responseStatus.getElementsByTagName("RequestId").item(0);
258 String requestId = null;
259 if (requestIdNode != null && requestIdNode.getFirstChild() != null) {
260 requestId = requestIdNode.getFirstChild().getNodeValue();
261 }
262 String boxUsage = null;
263 Node boxUsagesNode = responseStatus.getElementsByTagName("BoxUsage").item(0);
264 if (boxUsagesNode != null && boxUsagesNode.getFirstChild() != null) {
265 boxUsage = boxUsagesNode.getFirstChild().getNodeValue();
266 }
267 return new Response(document, rootElement, requestId, boxUsage, null, null);
268 } catch (RuntimeException e) {
269 throw e;
270 } catch (Exception e) {
271 throw new RuntimeException(e);
272 }
273 }
274
275
276 private static String getSignature(String data, String key) throws NoSuchAlgorithmException, InvalidKeyException {
277 SecretKeySpec signingKey = new SecretKeySpec(key.getBytes(),HMAC_SHA1_ALGORITHM);
278 Mac mac = Mac.getInstance(HMAC_SHA1_ALGORITHM);
279 mac.init(signingKey);
280 byte[] rawHmac = mac.doFinal(data.getBytes());
281 return new sun.misc.BASE64Encoder().encode(rawHmac);
282 }
283
284 private Map<String,String> getBaseParameters(String action) {
285 Collator myCollator = Collator.getInstance();
286 // Note that these parameters must be sent in order
287 Map<String,String> baseParameters = new TreeMap<String,String>(myCollator);
288 baseParameters.put("Action", action);
289 baseParameters.put("Version", version);
290 baseParameters.put("AWSAccessKeyId", accessId);
291 baseParameters.put("SignatureVersion", "1");
292
293 Date curTime = new Date();
294 SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ");
295 StringBuffer timeBuffer = new StringBuffer();
296 FieldPosition fp = new FieldPosition(NumberFormat.INTEGER_FIELD);
297 dateFormat.format(curTime, timeBuffer, fp);
298 timeBuffer.insert(timeBuffer.length()-2, ":");
299
300 baseParameters.put("Timestamp",timeBuffer.toString());
301 return baseParameters;
302 }
303
304 private Document getResponse(Map<String,String> parameters) throws UnsupportedEncodingException, MalformedURLException, IOException, NoSuchAlgorithmException, SAXException, ParserConfigurationException, InvalidKeyException {
305 String path = uri + getUrlPath(parameters);
306 URL url = new URL(path);
307 HttpURLConnection connection = (HttpURLConnection) url.openConnection();
308 connection.setDoOutput(true);
309 connection.setRequestMethod("GET");
310 connection.connect();
311
312 System.out.println("Connecting to " + url);
313 int respCode = connection.getResponseCode();
314
315 if (respCode == HttpURLConnection.HTTP_OK) {
316 InputSource inputSource = new InputSource(new BufferedReader(new InputStreamReader(connection.getInputStream())));
317 DocumentBuilderFactory documentBuilderFactory = DocumentBuilderFactory.newInstance();
318 DocumentBuilder documentBuilder = documentBuilderFactory.newDocumentBuilder();
319 return documentBuilder.parse(inputSource);
320 } else {
321 InputSource inputSource = new InputSource(new BufferedReader(new InputStreamReader(connection.getErrorStream())));
322 DocumentBuilderFactory documentBuilderFactory = DocumentBuilderFactory.newInstance();
323 DocumentBuilder documentBuilder = documentBuilderFactory.newDocumentBuilder();
324 Document document = documentBuilder.parse(inputSource);
325 Element rootElement = document.getDocumentElement();
326 Node requestIdNode = rootElement.getElementsByTagName("RequestID").item(0);
327 StringBuilder errorMessage = new StringBuilder("Failed to invoke input request '" + path + "'");
328
329 //
330 if (requestIdNode != null && requestIdNode.getFirstChild() != null) {
331 errorMessage.append(" with requestId " + requestIdNode.getFirstChild().getNodeValue());
332 Element errors = (Element) rootElement.getElementsByTagName("Errors").item(0);
333 if (errors != null) {
334 Element error = (Element) errors.getElementsByTagName("Error").item(0);
335 if (error != null) {
336 Node messageNode = error.getElementsByTagName("Message").item(0);
337 if (messageNode != null) errorMessage.append(" due to " + messageNode.getFirstChild().getNodeValue());
338 }
339 }
340 }
341 throw new RuntimeException(errorMessage.toString());
342 }
343 }
344
345
346 private String encode(String name){
347 try {
348 return URLEncoder.encode(name, DEFAULT_ENCODING);
349 } catch (UnsupportedEncodingException ex) {
350 throw new RuntimeException("Failed to encode '" + name + "'");
351 }
352 }
353
354
355 private String getUrlPath(Map<String,String> parameters) throws UnsupportedEncodingException, NoSuchAlgorithmException, InvalidKeyException {
356 StringBuilder queryString = new StringBuilder("?");
357 StringBuilder signatureData = new StringBuilder();
358 boolean first = true;
359 for (Map.Entry<String, String> param : parameters.entrySet()) {
360 if (first) {
361 first = false;
362 } else {
363 queryString.append("&");
364 }
365
366 signatureData.append(param.getKey() + param.getValue());
367 queryString.append(param.getKey() + "=" + URLEncoder.encode((String)param.getValue(), DEFAULT_ENCODING));
368 }
369
370 queryString.append("&Signature=" + URLEncoder.encode(getSignature(signatureData.toString(),secretKey), DEFAULT_ENCODING));
371 return queryString.toString();
372 }
373 }
374
Java Client Test
1 import junit.framework.TestCase;
2 import junit.framework.Test;
3 import junit.framework.TestSuite;
4
5 import java.util.List;
6 import java.util.Map;
7 import java.util.HashMap;
8
9 public class SdbDelegateTest extends TestCase {
10 private SdbDelegate fixture;
11 private String domainName;
12
13 public void setUp() {
14 this.fixture = new SdbDelegate();
15 this.domainName = "TestDomain";
16 }
17
18 public void testCreateDomain() {
19 fixture.createDomain(domainName);
20 assertTrue(fixture.listDomains().contains(domainName));
21 }
22
23 public void testDeleteDomain() {
24 fixture.deleteDomain(domainName);
25 assertFalse(fixture.listDomains().contains(domainName));
26 }
27
28
29 public void testGetPutAttributes() {
30 fixture.createDomain(domainName);
31 Map<String, String> attributes = new HashMap<String, String>();
32 attributes.put("StreetAddress", "705 5th Ave");
33 attributes.put("City", "Seattle");
34 attributes.put("State", "WA");
35 attributes.put("Zip", "98101");
36 fixture.putAttributes(domainName, "TCC", attributes, true);
37 Map<String, String> attributes2 = fixture.getAttributes(domainName, "TCC");
38 System.out.println("Retrieved " + attributes2);
39 assertEquals(attributes, attributes2);
40 }
41
42 public static Test suite() {
43 TestSuite suite = new TestSuite();
44 suite.addTestSuite(SdbDelegateTest.class);
45 return suite;
46 }
47
48 public static void main(String[] args) {
49 junit.textui.TestRunner.run(suite());
50 }
51 }
52
Erlang Client
1 %%%-------------------------------------------------------------------
2 %%% @author Shahzad Bhatti <bhatti@plexobject.com>
3 %%% @doc
4 %%% Business delegate to access SimpleDB
5 %%% @end
6 %%%-------------------------------------------------------------------
7 -module(sdb_delegate).
8 -include_lib("xmerl/include/xmerl.hrl").
9
10
11 %%%-------------------------------------------------------------------
12 %%% Public APIs
13 %%%-------------------------------------------------------------------
14 -export([list_domains/0, list_domains/1, list_domains/2, delete_domain/1, get_attributes/2, get_attributes/3, create_domain/1, put_attributes/3, put_attributes/4, delete_item/2, delete_attributes/2, delete_attributes/3, replace_attributes/3]).
15
16
17 %%%-------------------------------------------------------------------
18 %%% Test Methods
19 %%%-------------------------------------------------------------------
20 -export([test/0]).
21
22
23 %%%-------------------------------------------------------------------
24 %%% Configuration Functions %%%
25 %%%-------------------------------------------------------------------
26 uri() ->
27 "http://sdb.amazonaws.com?".
28
29
30 version() ->
31 "2007-11-07".
32
33 access_key() ->
34 "yourkey".
35
36 secret_key() ->
37 "secretkey".
38
39
40
41 %%%-------------------------------------------------------------------
42 %%% Public APIs to access SimpleDB
43 %%%-------------------------------------------------------------------
44 start() ->
45 crypto:start(),
46 inets:start().
47
48 stop() ->
49 init:stop().
50
51
52 list_domains() ->
53 list_domains(nil, nil).
54 list_domains(MoreToken) ->
55 list_domains(MoreToken, nil).
56
57 list_domains(MoreToken, MaxResults) ->
58 Base = base_parameters("ListDomains"),
59 Base1 = if MoreToken == nil -> Base; true -> Base ++ [["MoreToken", MoreToken]] end,
60 Base2 = if MaxResults == nil -> Base1; true -> Base1 ++ [["MaxResults", MaxResults]] end,
61 Xml = rest_request(Base2),
62 xml_values(xmerl_xpath:string("//DomainName/text()", Xml)).
63
64
65 create_domain(Domain) ->
66 Base = [["DomainName", url_encode(Domain)]|
67 base_parameters("CreateDomain")],
68 rest_request(Base).
69
70 delete_domain(Domain) ->
71 Base = [["DomainName", url_encode(Domain)]|
72 base_parameters("DeleteDomain")],
73 rest_request(Base).
74
75 replace_attributes(Domain, ItemName, Attributes) ->
76 put_attributes(Domain, ItemName, Attributes, true).
77 put_attributes(Domain, ItemName, Attributes) ->
78 put_attributes(Domain, ItemName, Attributes, false).
79 put_attributes(Domain, ItemName, Attributes, Replace) ->
80 Base = [["DomainName", url_encode(Domain)],
81 ["ItemName", url_encode(ItemName)]|
82 base_parameters("PutAttributes")] ++
83 encode_attributes(Attributes),
84 Base1 = if Replace == false -> Base; true -> Base ++ [["Replace", "true"]] end,
85 rest_request(Base1).
86
87
88 delete_item(Domain, ItemName) ->
89 delete_attributes(Domain, ItemName).
90
91
92 delete_attributes(Domain, ItemName) ->
93 delete_attributes(Domain, ItemName, nil).
94 delete_attributes(Domain, ItemName, AttributeNames) ->
95 Base = [["DomainName", url_encode(Domain)],
96 ["ItemName", url_encode(ItemName)]|
97 base_parameters("DeleteAttributes")] ++
98 encode_attribute_names(AttributeNames),
99 rest_request(Base).
100
101 get_attributes(Domain, ItemName) ->
102 get_attributes(Domain, ItemName, nil).
103 get_attributes(Domain, ItemName, AttributeNames) ->
104 Base = [["DomainName", url_encode(Domain)],
105 ["ItemName", url_encode(ItemName)]|
106 base_parameters("GetAttributes")] ++
107 encode_attribute_names(AttributeNames),
108 Xml = rest_request(Base),
109 xml_names_values(xmerl_xpath:string("//Attribute", Xml)).
110
111
112 %%%-------------------------------------------------------------------
113 %%% Private Functions
114 %%%-------------------------------------------------------------------
115
116 rest_request(Params) ->
117 Url = uri() ++ query_string(Params),
118 Response = http:request(Url),
119 case Response of
120 {ok, {{_HttpVersion, StatusCode, _ErrorMessage}, _Headers, Body }} ->
121 error_logger:info_msg("URL ~p Status ~p~n", [Url, StatusCode]),
122 {Xml, _Rest} = xmerl_scan:string(Body),
123 %%%error_logger:info_msg("Xml ~p~n", [Xml]),
124 case StatusCode of
125 200 ->
126 Xml;
127 _ ->
128 Error = xml_values(xmerl_xpath:string("//Message/text()", Xml)),
129 throw({Error})
130 end;
131 {error, Message} ->
132 case Message of
133 timeout ->
134 io:format("URL ~p Timedout, retrying~n", [Url]),
135 sleep(1000),
136 rest_request(Params);
137 true ->
138 throw({Message})
139 end
140 end.
141
142
143 query_string(Params) ->
144 Params1 = lists:sort(
145 fun([Elem1, _], [Elem2, _]) ->
146 string:to_lower(Elem1) > string:to_lower(Elem2) end,
147 Params),
148 {QueryStr, SignatureData} =
149 lists:foldr(fun query_string/2, {"", ""}, Params1),
150 QueryStr ++ "Signature=" ++ url_encode(signature(SignatureData)).
151
152
153 query_string([Key, Value], {QueryStr, SignatureData}) ->
154 QueryStr1 = QueryStr ++ Key ++ "=" ++ url_encode(Value) ++ "&",
155 SignatureData1 = SignatureData ++ Key ++ Value,
156 {QueryStr1, SignatureData1}.
157
158
159 encode_attributes(Attributes) when Attributes == nil ->
160 [];
161 encode_attributes(Attributes) ->
162 {Encoded, _} = lists:foldr(fun encode_attributes/2, {[], 0}, Attributes),
163 Encoded.
164
165 encode_attributes([Key, Value], {Encoded, I}) ->
166 KeyName = "Attribute." ++ integer_to_list(I) ++ ".Name",
167 KeyValue = "Attribute." ++ integer_to_list(I) ++ ".Value",
168 {[[KeyName, Key], [KeyValue, Value]|Encoded], I+1}.
169
170
171 encode_attribute_names(Attributes) when Attributes == nil ->
172 [];
173 encode_attribute_names(Attributes) ->
174 {Encoded, _} = lists:foldr(fun encode_attribute_names/2, {[], 0}, Attributes),
175 Encoded.
176
177 encode_attribute_names(Key, {Encoded, I}) ->
178 KeyName = "Attribute." ++ integer_to_list(I) ++ ".Name",
179 {[[KeyName, Key]|Encoded], I+1}.
180
181
182
183
184 %%%
185 % Converts a number into 2-digit
186 %%%
187 two_digit(X) when is_integer(X), X >= 10 ->
188 integer_to_list(X);
189 two_digit(X) when is_integer(X), X < 10 ->
190 "0" ++ integer_to_list(X).
191
192 abs_two_digit(X) when X < 0 ->
193 two_digit(0-X);
194 abs_two_digit(X) when X >= 0 ->
195 two_digit(X).
196
197 %%%
198 % Returns Coordinated Universal Time (Greenwich Mean Time) time zone,
199 %%%
200 timestamp() ->
201 {{_, _, _}, {_LocalHour, _LocalMin, _}} = LocalDateTime = calendar:local_time(),
202 [{{Year, Month, Day}, {Hour, Min, Sec}}] =
203 calendar:local_time_to_universal_time_dst(LocalDateTime),
204 Z = gmt_difference(),
205 integer_to_list(Year) ++ "-" ++ two_digit(Month) ++ "-" ++ two_digit(Day)
206 ++ "T" ++ two_digit(Hour) ++ ":" ++ two_digit(Min) ++ ":" ++
207 two_digit(Sec) ++ Z.
208
209
210 gmt_difference() ->
211 "-08:00".
212
213
214
215 %%%
216 % Returns HMAC encoded access key
217 %%%
218 signature(Data) ->
219 hmac(secret_key(), Data).
220
221 hmac(SecretKey, Data) ->
222 http_base_64:encode(
223 binary_to_list(crypto:sha_mac(SecretKey, Data))).
224
225
226 base_parameters(Action) ->
227 [["Action", Action],
228 ["AWSAccessKeyId", access_key()],
229 ["Version", version()],
230 ["SignatureVersion", "1"],
231 ["Timestamp", timestamp()]].
232
233
234
235 %%%
236 % This method retrieves node value from the XML records that are returned
237 % after scanning tags.
238 %%%
239 xml_values(List) ->
240 lists:foldr(fun xml_values/2, [], List).
241
242 xml_values(#xmlText{value=Value}, List) ->
243 [Value|List].
244
245
246 xml_names_values(List) ->
247 lists:foldr(fun xml_names_values/2, [], List).
248
249 xml_names_values(Xml, List) ->
250 [ #xmlText{value=Name} ] = xmerl_xpath:string("//Name/text()", Xml),
251 [ #xmlText{value=Value} ] = xmerl_xpath:string("//Value/text()", Xml),
252 [[Name, Value]|List].
253
254
255 sleep(T) ->
256 receive
257 after T ->
258 true
259 end.
260
261
262 %%%
263 % URL encode - borrowed from CouchDB
264 %%%
265 url_encode([H|T]) ->
266 if
267 H >= $a, $z >= H ->
268 [H|url_encode(T)];
269 H >= $A, $Z >= H ->
270 [H|url_encode(T)];
271 H >= $0, $9 >= H ->
272 [H|url_encode(T)];
273 H == $_; H == $.; H == $-; H == $: ->
274 [H|url_encode(T)];
275 true ->
276 case lists:flatten(io_lib:format("~.16.0B", [H])) of
277 [X, Y] ->
278 [$%, X, Y | url_encode(T)];
279 [X] ->
280 [$%, $0, X | url_encode(T)]
281 end
282 end;
283 url_encode([]) ->
284 [].
285
286
287 %%%-------------------------------------------------------------------
288 %%% Test Functions
289 %%%-------------------------------------------------------------------
290
291 test_list_domains() ->
292 list_domains().
293
294 test_create_domain() ->
295 create_domain("TestDomain"),
296 ["TestDomain"] = lists:filter(
297 fun(Elem) -> Elem == "TestDomain" end,
298 list_domains()).
299
300
301 test_delete_domain() ->
302 delete_domain("TestDomain"),
303 [] = lists:filter(
304 fun(Elem) -> Elem == "TestDomain" end,
305 list_domains()).
306
307 test_put_get_attributes() ->
308 create_domain("TestDomain"),
309 Attributes = lists:sort([
310 ["StreetAddress", "705 5th Ave"],
311 ["City", "Seattle"],
312 ["State", "WA"],
313 ["Zip", "98101"]
314 ]),
315 put_attributes("TestDomain", "TCC", Attributes),
316 Attributes = lists:sort(get_attributes("TestDomain", "TCC")).
317
318 test_put_delete_attributes() ->
319 create_domain("TestDomain"),
320 Attributes = lists:sort([
321 ["StreetAddress", "705 5th Ave"],
322 ["City", "Seattle"],
323 ["State", "WA"],
324 ["Zip", "98101"]
325 ]),
326 put_attributes("TestDomain", "TCC", Attributes),
327 delete_attributes("TestDomain", "TCC"),
328 sleep(1000), %% Let it sync
329 [] = get_attributes("TestDomain", "TCC").
330
331 test() ->
332 start(),
333 test_create_domain(),
334 test_delete_domain(),
335 test_put_get_attributes(),
336 test_put_delete_attributes(),
337 stop().
338
Note that in Erlang, I am hard coding time-zone difference, I wish Erlang had better support for time-zones, and URL encoding for UTF8.
Another word of caution, in order to use this code for real applications, you will need better handle on error processing especially because web services may timeout and you will need to retry.
You can download the code from here:
http://weblog.plexobject.com/SdbDelegate.java
http://weblog.plexobject.com/SdbDelegateTest.java
http://weblog.plexobject.com/sdb_delegate.erl
I am also adding the Erlang code to http://code.google.com/p/erlsdb/.
Further References:
http://awsdocs.s3.amazonaws.com/SDB/2007-11-07/sdb-dg-2007-11-07.pdf?
http://docs.amazonwebservices.com/AmazonSimpleDB/2007-11-07/DeveloperGuide/?
http://aws.amazon.com/simpledb
http://www.informationweek.com/news/showArticle.jhtml?articleID=204803008
You can send me comments or suggestions to improve this code at bhatti AT plexobject DOT com.
Update (Dec 17, 2007): I saw some comments on how unRESTful SimpleDB’s REST APIs are and I agree. Here is my attempt to RESTify them:
Create Domain
PUT http://sdb.amazonaws.com/DomainName
I am using PUT because I am defining the URL and per REST specs I should use PUT instead of POST. Also, I am relying on HTTP headers for authentication information.
Delete Domain
DELETE http://sdb.amazonaws.com/DomainName
List Domains
GET http://sdb.amazonaws.com?MaxDomains=XX&NextToken=YY
I am specifying parameters explicitly rather than using changing URI
Put Attributes
POST http://sdb.amazonaws.com/DomainName/ItemName
—POST BODY-BEGIN—
attributeName=attributeValue&…
—POST BODY-END—
I am specifying attributes as part of the POST body
Delete Attributes
DELETE http://sdb.amazonaws.com/DomainName/ItemName?attributeName1&…
Get Attributes
GET http://sdb.amazonaws.com/DomainName/ItemName?attributeName1&…
Query Items
GET http://sdb.amazonaws.com/DomainName/ItemName?queryExpression=…
Note that these URLs are based on Sam Ruby’s book on REST APIs. Also, Rails uses similar URLs for RESTful controllers.