Shahzad Bhatti Welcome to my ramblings and rants!

November 16, 2009

Applying Adaptive Object Model using dynamic languages and schema-less databases

Filed under: Java — admin @ 3:10 pm

Introduction to Adaptive/Active Object Model

Adaptive or Active Object Model is a design pattern used in domains that requires dynamic manipulation of meta information.
Though, it is quite extensive topic of research, but general idea from original paper of
Ralph Johnson is to treat meta information such as attributes,
rules and relationships as a data. It is usually used when the number of sub-classes is huge or unknown upfront and the system requires adding new functionality without downtime.
For example, let’s say we are working in automobile domain and we need to model different type of vehicles. Using an object oriented design would result in vehicle hierarchy such as follows:

In above example, all type hierarchy is predefined and each class within the hierarchy defines attributes and operations. Adaptive Object Modeling on the other hand use Object Type pattern, which treats classes like objects. The basic Adaptive Object Model uses type square model such as:

In above diagram, EntityType class represents all classes and instance of this class defines actual attributes and operations supported by the class. Similarly, PropertyType defines names and types of all attributes. Finally, instance of Entity class will actual be real object instance that would store collection of properties and would refer to the EntityType.

Java Implementation

Let’s assume we only need to model Vehicle class from above vehicle hierarchy. In a typical object oriented language such as Java, the Vehicle class would be defined as follows:

  1 /*
  2  * Simple Vehicle class
  3  * 
  4  */
 
  5 package com.plexobject.aom;
  6 
  7 import java.util.Date;
  8 
 
  9 public class Vehicle {
 10 
 11     private String maker;
 12     private String model;
 
 13     private Date yearCreated;
 14     private double speed;
 15     private long miles;
 
 16     //... other attributes, accessors, setters
 17 
 18     public void drive() {
 19         //
 
 20     }
 21 
 22     public void stop() {
 23         //
 
 24     }
 25 
 26     public void performMaintenance() {
 27         //
 28     }
 
 29     //... other methods
 30 }
 31 
 32 
 33

As you can see all attributes and operations are defined within the Vehicle class. The Adaptive Object Model would use meta classes such as Entity, EntityType, Property and PropertyType to build the Vehicle metaclass. Following Java code defines core classes of type square model:

The Property class defines type and value for each attribute of class:

  1 /*
  2  * Property class defines attribute type and value
  3  * 
  4  */
 
  5 package com.plexobject.aom;
  6 
  7 public class Property {
 
  8 
  9     private PropertyType propertyType;
 10     private Object value;
 11 
 12     public Property(PropertyType propertyType, Object value) {
 
 13         this.propertyType = propertyType;
 14         this.value = value;
 15     }
 16 
 17     public PropertyType getPropertyType() {
 
 18         return propertyType;
 19     }
 20 
 21     public Object getValue() {
 22         return value;
 
 23     }
 24     //... other methods
 25 }
 26 
 27

The PropertyType class defines type information for each attribute of class:

  1 /*
  2  * PropertyType class defines type information
  3  * 
  4  */
 
  5 package com.plexobject.aom;
 
  6 
  7 public class PropertyType {
  8 
  9     private String propertyName;
 
 10     private String type;
 11 
 12     public PropertyType(String propertyName, String type) {
 13         this.propertyName = propertyName;
 14         this.type = type;
 
 15     }
 16 
 17     public String getPropertyName() {
 18         return propertyName;
 19     }
 
 20 
 21     public String getType() {
 22         return type;
 23     }
 24     //... other methods
 
 25 }

The EntityType class defines type of entity:

  1 /*
  2  * EntityType class defines attribute types and operations
  3  * 
  4  */
  5 package com.plexobject.aom;
 
  6 
  7 import java.util.Collection;
  8 import java.util.HashMap;
  9 import java.util.Map;
 
 10 
 11 public class EntityType {
 12 
 13     private String typeName;
 14     private Map<String, PropertyType> propertyTypes = new HashMap<String, PropertyType>();
 
 15     private Map<String, Operation> operations = new HashMap<String, Operation>();
 16 
 17     public EntityType(String typeName) {
 
 18         this.typeName = typeName;
 19     }
 20 
 21     public String getTypeName() {
 22         return typeName;
 
 23     }
 24 
 25     public void addPropertyType(PropertyType propertyType) {
 26         propertyTypes.put(propertyType.getPropertyName(),
 27                 propertyType);
 
 28     }
 29 
 30     public Collection<PropertyType> getPropertyTypes() {
 31         return propertyTypes.values();
 
 32     }
 33 
 34     public PropertyType getPropertyType(String propertyName) {
 35         return propertyTypes.get(propertyName);
 36     }
 
 37 
 38     public void addOperation(String operationName, Operation operation) {
 39         operations.put(operationName, operation);
 40 
 41     }
 
 42 
 43     public Operation getOperation(String name) {
 44         return operations.get(name);
 45     }
 46 
 
 47     public Collection<Operation> getOperations() {
 48         return operations.values();
 49     }
 50     //... other methods
 
 51 }
 52 
 53

The Entity class defines entity itself:

  1 /*
  2  * Entity class represents instance of actual metaclass
  3  * 
  4  */
  5 package com.plexobject.aom;
 
  6 
  7 import java.util.Collection;
  8 import java.util.Collections;
  9 
 
 10 public class Entity {
 11 
 12     private EntityType entityType;
 13     private Collection<Property> properties;
 
 14 
 15     public Entity(EntityType entityType) {
 16         this.entityType = entityType;
 17     }
 18 
 19     public EntityType getEntityType() {
 
 20         return entityType;
 21     }
 22 
 23     public void addProperty(Property property) {
 
 24         properties.add(property);
 25     }
 26 
 27     public Collection<Property> getProperties() {
 28         return Collections.unmodifiableCollection(properties);
 
 29     }
 30 
 31     public Object perform(String operationName, Object[] args) {
 32         return entityType.getOperation(operationName).perform(this, args);
 
 33     }
 34     //... other methods
 35 }

The Operation interface is used for implementing behavior using Command pattern:

  1 /*
  2  * Operation interface defines behavior
  3  * 
  4  */
  5 package com.plexobject.aom;
 
  6 
  7 public interface Operation {
  8 
  9     Object perform(Entity entity, Object[] args);
 
 10 }

Above meta classes would be used to create classes and objects. For example, the type information of Vehicle class would be defined in EntityType and PropertyType and the instance would be defined using Entity and Property classes as follows. Though, in real applications, type binding would be stored in XML configuration or will be defined in some DSL, but I am binding programmatically below:

  1 /*
  2  * an example of binding attributes and operations of Vehicle
  3  * 
  4  */
 
  5 package com.plexobject.aom;
  6 
  7 import java.util.Date;
  8 
 
  9 
 10 public class Initializer {
 11 
 12     public void bind() {
 
 13         EntityType vehicleType = new EntityType("Vehicle");
 14         vehicleType.addPropertyType(new PropertyType("maker",
 15                 "java.lang.String"));
 
 16         vehicleType.addPropertyType(new PropertyType("model",
 17                 "java.lang.String"));
 18         vehicleType.addPropertyType(new PropertyType("yearCreated",
 
 19                 "java.util.Date"));
 20         vehicleType.addPropertyType(new PropertyType("speed",
 21                 "java.lang.Double"));
 22         vehicleType.addPropertyType(new PropertyType("miles",
 
 23                 "java.lang.Long"));
 24         vehicleType.addOperation("drive", new Operation() {
 25 
 26             public Object perform(Entity entity, Object[] args) {
 
 27                 return "driving";
 28             }
 29         });
 30         vehicleType.addOperation("stop", new Operation() {
 
 31 
 32             public Object perform(Entity entity, Object[] args) {
 33                 return "stoping";
 34             }
 35         });
 
 36         vehicleType.addOperation("performMaintenance", new VehicleMaintenanceOperation());
 37 
 38 
 39         // now creating instance of Vehicle
 40         Entity vehicle = new Entity(vehicleType);
 
 41         vehicle.addProperty(new Property(vehicleType.getPropertyType("maker"),
 42                 "Toyota"));
 43         vehicle.addProperty(new Property(vehicleType.getPropertyType("model"),
 
 44                 "Highlander"));
 45         vehicle.addProperty(new Property(vehicleType.getPropertyType("yearCreated"),
 46                 new Date(2003, 0, 1)));
 
 47         vehicle.addProperty(new Property(vehicleType.getPropertyType("speed"), new Double(120)));
 48         vehicle.addProperty(new Property(vehicleType.getPropertyType("miles"), new Long(3000)));
 
 49         vehicle.perform(
 50                 "drive", null);
 51 
 52     }
 53 }
 
 54 
 55

The operations define runtime behavior of the class and can be defined as closures (anonymous classes) or external implementation such as VehicleMaintenanceOperation as follows:

  1 /*
 
  2  * an example of operation
  3  * 
  4  */
 
  5 package com.plexobject.aom;
  6 
  7 class VehicleMaintenanceOperation implements Operation {
 
  8 
  9     public VehicleMaintenanceOperation() {
 10     }
 11 
 12     public Object perform(Entity entity, Object[] args) {
 
 13         return "maintenance";
 14     }
 15 }
 16 
 17

In real applications, you would also have meta classes for business rules, relationships, strategies, validations, etc as instances. As, you can see AOM provides powerful way to adopt new business requirements and I have seen it used successfully while working as consultant. On the downside, it requires a lot of plumbing and tooling support such as XML based configurations or GUI tools to manipulate meta data. I have also found it difficult to optimize with relational databases as each attribute and operation are stored in separate rows in the databases, which results in excessive joins when building the object. There are a number of alternatives of Adaptive Object Model such as code generators, generative techniques, metamodeling, and table-driven systems. These techniques are much easier with dynamic languages due to their support of metaprogramming, higher order functions and generative programming. Also, over the last few years, a number of schema less databases such as CouchDB, MongoDB, Redis, Cassendra, Tokyo Cabinet, Riak, etc. have become popular due to their ease of use and scalability. These new databases solve excessive join limitation of relational databases and allow evolution of applications similar to Adaptive Object Model. They are also much more scalable than traditional databases. The combination of dynamic languages and schema less databases provides a simple way to add Adaptive Object Model features without a lot of plumbing code.

Javascript Implementation

Let’s try above example in Javascript due to its supports of higher order functions, and prototype based inheritance capabilities. First, we will need to add some helper methods to Javascript (adopted from Douglas Crockford’s “Javascript: The Good Parts”), e.g.

  1 
  2 if (typeof Object.beget !== 'function') {
 
  3     Object.beget = function(o) {
  4         var F = function() {};
  5         F.prototype = o;
 
  6         return new F();
  7     }
  8 }
  9 
 
 10 Function.prototype.method = function (name, func) {
 11     this.prototype[name] = func;
 12     return this;
 13 };
 
 14 
 15 
 16 Function.method('new', function() {
 17     // creating new object that inherits from constructor's prototype
 
 18     var that = Object.beget(this.prototype);
 19     // invoke the constructor, binding -this- to new object
 
 20     var other = this.apply(that, arguments);
 21     // if its return value isn't an object substitute the new object
 
 22     return (typeof other === 'object' && other) || that;
 23 });
 24 
 
 25 Function.method('inherits', function(Parent) {
 26     this.prototype = new Parent();
 27     return this;
 
 28 });
 29 
 30 Function.method('bind', function(that) {
 31     var method = this;
 
 32     var slice = Array.prototype.slice;
 33     var args = slice.apply(arguments, [1]);
 34     return function() {
 35         return method.apply(that, args.concat(slice.apply(arguments,
 
 36             [0])));
 37     };
 38 });
 39 
 40 // as typeof is broken in Javascript, trying to get type from the constructor
 
 41 Object.prototype.typeName = function() {
 42     return typeof(this) === 'object' ? this.constructor.toString().split(/[\s\(]/)[1] : typeof(this);
 
 43 };
 44 
 45

There is no need to define Operation interface, Property and PropertyType due to higher order function and dynamic language support. Following Javascript code defines core functionality of Entity and EntityType classes, e.g.:

  1 
  2 var EntityType = function(typeName, propertyNamesAndTypes) {
 
  3     this.typeName = typeName;
  4     this.propertyNamesAndTypes = propertyNamesAndTypes;
  5     this.getPropertyTypesAndNames = function() {
 
  6         return this.propertyNamesAndTypes;
  7     };
  8     this.getPropertyType = function(propertyName) {
 
  9         return this.propertyNamesAndTypes[propertyName];
 10     };
 11     this.getTypeName = function() {
 12         return this.typeName;
 
 13     };
 14     var that = this;
 15     for (propertyTypesAndName in propertyNamesAndTypes) {
 
 16         that[propertyTypesAndName] = function(name) {
 17             return function() {
 18                 return propertyNamesAndTypes[name];
 
 19             };
 20         }(propertyTypesAndName);
 21         
 22     }
 
 23 };
 24 
 25 
 26 
 27 var Entity = function(entityType, properties) {
 28     this.entityType = entityType;
 
 29     this.properties = properties;
 30     this.getEntityType = function() {
 31         return this.entityType;
 32     };
 
 33     var that = this;
 34     for (propertyTypesAndName in entityType.getPropertyTypesAndNames()) {
 35         that[propertyTypesAndName] = function(name) {
 
 36             return function() {
 37                 if (arguments.length == 0) {
 38                     return that.properties[name];
 39                 } else {
 
 40                     var oldValue = that.properties[name];
 41                     that.properties[name] = arguments[0];
 42                     return oldValue;
 43                 }
 44             };
 
 45         }(propertyTypesAndName);
 46         
 47     }
 48 };

Following Javascript code shows binding and example of usage (again in real application binding will be stored in configurations):

  1 
  2 var vehicleType = new EntityType('Vehicle', {
 
  3     'maker' : 'String',              // name -> typeName
  4     'model' : 'String',
 
  5     'yearCreated' : 'Date',
  6     'speed' : 'Number',
  7     'miles' : 'Number'
 
  8 });
  9 
 10 var vehicle = new Entity(vehicleType, {
 11     'maker' : 'Toyota',
 
 12     'model' : 'Highlander',
 13     'yearCreated' : new Date(2003, 0, 1),
 14     'speed' : 120,
 
 15     'miles' : 3000
 16 });
 17 
 18 vehicle.drive = function() {
 19     }.bind(vehicle);
 
 20 
 21 vehicle.stop = function() {
 22     }.bind(vehicle);
 23 
 24 vehicle.performMaintenance = function() {
 
 25     }.bind(vehicle);

A big difference with dynamic languages is that you can bind properties operations to the objects at runtime so you can invoke them as if they were native. For example, you can invoke vehicleType.maker() to get maker property of the vehicle-type or call vehicle.drive() to invoke operation on vehicle object. Another difference is that a lot of plumbing code disappears with dynamic languages.

Ruby Implementation

Similarly, above example in Ruby may look like:

  1 require 'date'
 
  2 require 'forwardable'
  3 class EntityType
  4   attr_accessor :type_name
 
  5   attr_accessor :property_names_and_types
  6   def initialize(type_name, property_names_and_types)
  7     @type_name = type_name
 
  8     @property_names_and_types = property_names_and_types
  9   end
 10   def property_type(property_name)
 11     @property_names_and_types[property_name]
 
 12   end
 13 end
 14 
 15 
 16 class Entity
 
 17   attr_accessor :entity_type
 18   attr_accessor :properties
 19   def initialize(entity_type, attrs = {})
 
 20     @entity_type = entity_type
 21     bind_properties(entity_type.property_names_and_types)
 22     attrs.each do |name, value|
 23       instance_variable_set("@#{name}", value)
 
 24     end
 25   end
 26   def bind_properties(property_names_and_types)
 27     (class << self; self; end).module_eval do
 
 28       property_names_and_types.each do |name, type|
 29         define_method name.to_sym do
 30           instance_variables_get("@#{name}")
 
 31         end
 32         define_method name.to_sym do
 33           instance_variables_set("@#{name}", value)
 
 34         end
 35       end
 36     end
 37   end
 38 end
 
 39 
 66 
 67 
 68

We can then use Singleton, Lambdas and metaprogramming features of Ruby to add Adaptive Object Model support, e.g.

  1 vehicle_type = EntityType.new('Vehicle', {
 
  2     'maker' => 'String',             # class.name
  3     'model' => 'String',
 
  4     'yearCreated' => 'Time',
  5     'speed' => 'Fixnum',
 
  6     'miles' => 'Float'});
  7 
  8 
  9 vehicle = Entity.new(vehicle_type, {
 
 10     'maker' => 'Toyota',
 11     'model' => 'Highlander',
 12     'yearCreated' => DateTime.parse('1-1-2003'),
 
 13     'speed' => 120,
 14     'miles' => 3000});
 15 class << vehicle
 
 16   def drive
 17     "driving"
 18   end
 19   def stop
 
 20     "stopping"
 21   end
 22   def perform_maintenance
 23     "performing maintenance"
 
 24   end
 25 end
 26 
 27

Ruby code is a lot more succint and as Ruby supports adding or removing methods dynamically, you can invoke properties and operations directly on the objects. For example, you can invoke vehicleType.maker() to get maker property of the vehicle-type or call vehicle.drive() to invoke operation on vehicle object. Also, Ruby provides a lot more options for higher order functions such as monkey patching, lambdas/procs/methods, send, delegates/forwardables, etc. Finally, Ruby provides powerful generative capabilities to build DSL that can bind all properties and operations at runtime similar to how Rails framework work.

Schema-less Databases

Now, the second half of the equation for Adaptive Object Model is persisting, which I have found to be challenge with relational databases. However, as I have been using schemaless databases such as CouchDB, it makes it trivial to store meta information as part of the plain data. For example, if I have to store this vehicle in CouchDB, all I have to do is create a table such as vehicles (I could use Single Table Inheritance to store all types of vehicles in same table):

 curl -XPUT http://localhost:5984/vehicles
 curl -XPUT http://localhost:5984/vehicle_types

and then add vehicle-type as

 curl -XPOST http://localhost:5984/vehicle_types/ -d '{"maker":"String", "model":"String", "yearCreated":"Date", "speed":"Number", "miles":"Number"}'

which returns

 {"ok":true,"id":"bb70f95e43c3786f72cb46b372a2808f","rev":"1-3976038079"}

Now, we can use the id of vehicle-type and add vehicle a follows

 curl -XPOST http://localhost:5984/vehicles/ -d '{"vehicle_type_id":"bb70f95e43c3786f72cb46b372a2808f", "maker":"Toyota", "model":"Highlander", "yearCreated":"2003", "speed":120, "miles":3000}'

which returns id of newly created vehicle as follows:

 {"ok":true,"id":"259237d7c041c405f0671d6774bfa57a","rev":"1-367618940"}

Summary

It is often said in software development that you can solve any problem with another level of indirection. Adaptive Object Model uses another level of indirection to create powerful applications that meet increasingly changing requirements. When it is used with dynamic languages that support metaprogramming and generative programming, it can be used build systems that can be easily evolved with minimum changes and downtime. Also, Schema-less databases eliminates drawbacks of many implementations of AOM that suffer from poor performance due to excessive joins in the relational databases.

September 21, 2009

Installing Ubuntu Remix and Troubleshooting Network connections

Filed under: Computing — admin @ 10:00 am

I recently ordered ASUS Eeee PC 1005HA netbook that actually got lost in mail and had to reorder. Anyway, I finally received it this weekend and it comes with Windows XP that I decided to replace with Ubuntu. Though, there is a special distribution of Ubuntu called Remix or UNR, but support of netbooks on Ubuntu is still work in progress so it took longer than I expected. Here are the steps I went through to install and setup UNR on my ASUS netbook:

Download Ubuntu Remix

This was easy, I downloaded UNR from http://www.ubuntu.com/GetUbuntu/download-netbook and saved img file on my local netbook (which was running XP at that time).

Download USB Imager

Then, I downloaded USB Disk Imager for windows.

Creating UNR Image

After downloading imager, I opened the application, inserted my USB drive and copied the image, so far so good.

Changing BIOS to boot from USB

The ASUS reboots automatically from hard disk so I had to change the BIOS settings. I shutdown
the machine completely, then started while holding F2. It brought up BIOS settings and I changed the Boot sequence to boot from USB and then saved the settings with F10.

Installing UNR

After rebooting, the UNR loaded from the USB. First, I played without installing and figured out quickly that network isn’t working. I decided to install the UNR despite these issues. I allocated half of disk space about 70G to Linux and left Windows partition alone in case I fail. I then allocated swap space and then proceeded to install, which was fairly standard. After installation, I rebooted the machine and the GRUB loader showed me both Windows and UNR options.

Troubleshooting Network

Now, the fun started. Neither my wired nor wireless network was working. I found a number of forums with similar problems. I tried

 iwconfig
 iwlist scan
 lsmod

to see what’s installed and available but didn’t see the drivers. Also, “dmesg” wasn’t helpful and 

  sudo /etc/init.d/networking restart

didn’t help either. I then typed

 lspci

Which showed 

 02:00.0 Network controller: Atheros Communications Inc. AR9285 Wireless Network Adapter (PCI-Express) (rev 01)

I then switched to my Mac and I then looked for driver of AR9285. I found a good resource http://partner.atheros.com/Drivers.aspx and downloaded Linux driver and then copied to another USB drive.
I built the driver with

 tar -xzf 
 cd src
 make 
 sudo make install
 sudo insmod atl1e.ko

After rebooting, it fixed the wired network and I could then use the wired network to continue troubleshooting. I tried following instructions from http://wireless.kernel.org/en/users/Download, which suggested

 sudo apt-get install linux-backports-modules-jaunty

But it didn’t work for me. I then tried 

 apt-get install linux-backports-modules-$(uname -r)

And that didn’t work. Finally, I decided to upgrade to Karmic Koala by issuing this command:

 sudo do-release-upgrade -d

It took a while to download all packages, it then removed a bunch of obsolete packages and after reboot complained about a bunch of old configurations that are not compatible anymore. Nevertheless, my wireless started working, yeah. Next, I am going to install Regdb, CRDA, and IW to track any other wireless issues.

I still left option to dual boot on my netbook but I am definitely going to live in UNR for most part.

September 2, 2009

Introduction to CouchDB

Filed under: Computing — admin @ 6:51 pm

I have been following growth and popularity of CouchDB for a while and even attended an excellent talk by J Chris Anderson of http://couch.io. However, only recently I am getting chance to actually use it. I am building an internal Search Engine based on Lucene, but I am storing documents in CouchDB. Though, CouchDB is pretty easy to setup, but its documentation is sporadic. Here are basic steps to get it running:

Installation and Launch

I installed CouchDB on my MacPro notebook using:

 sudo port install couchdb

CouchDB is available for Linux distributions and you can use yum or apt to install it, though official binaries are not available for Windows. You can also setup to load it at startup on Mac usng:

 sudo launchctl load -w /opt/local/Library/LaunchDaemons/org.apache.couchdb.plist

Once you installed it, you can start the couchdb server using:

 sudo /opt/local/bin/couchdb

Alternatively, you can skip installation & launch and instead use hosting solution from http://hosting.couch.io using “booom-couch” password for private beta.

Verify Installation

Once couchdb is started you can point your browser to http://127.0.0.1:5984/ or type in:

 curl http://127.0.0.1:5984/

As CouchDB uses JSON format for communication, it would show something like:

 {"couchdb":"Welcome","version":"0.9.0"}

Alternatively, you can use curl to communication with couchd server:

 curl http://127.0.0.1:5984/

Creating a database

CouchDB is REST based service, and you can review all APIs at http://wiki.apache.org/couchdb/HTTP_Document_API. CouchDB uses PUT operation to create a database, e.g.

 curl -X PUT http://127.0.0.1:5984/guestbook

It will return

 {"ok":true}

Based on REST principles, PUT is used when adding a new data where the resource is specified by the client. However, if you call this API again with the same arguments, it will return in error, e.g.:

 {"error":"file_exists","reason":"The database could not be created, the file already exists."}

Adding documents

Each document is a JSON object that consists of name value pairs. Also, each document is specified a unique identifier or uuid. You can generate uuid in your application or get it from the CouchDB server. For example, to generate 10 UUIDs, call

 curl -X GET http://127.0.0.1:5984/_uuids?count=10

and it will return something like:

 {"uuids":["152019530472f7b0b364367bc2ec571d","cba55d13244afe7b924265760deccced","41a8d0d7093ac11827b3147565a08a80","281dc15503fffee17c9da332748e9288","90613ae77c78c8bd81849b728d648055","23c320522473bdd47071d56b72667172","bb8b72a9dc391e95ffd5e155d8bf7011","87b8da3e3cf0c16110e030a711dc26b3","cfdf87adc2cf4593a92e4edf38f2f557","dc80745c5cb478de48230e48efaf5ede"]}

You can then add a document using:

 curl -X PUT http://127.0.0.1:5984/guestbook/152019530472f7b0b364367bc2ec571d -d '{"name":"Sally", "message":"hi there"}'

It will return verification message:

 {"ok":true,"id":"152019530472f7b0b364367bc2ec571d","rev":"1-3525253587"}

Note, it generated a version of the document. Alternatively, you can use POST request to add document using server-generated UUID, e.g.

 curl -X POST http://127.0.0.1:5984/guestbook -d '{"name":"John", "message":"hi there"}'

That returns UUID and version of newly created object, e.g.

 {"ok":true,"id":"b4bb85ab50271f3d12d25feb219cb66e","rev":"1-657551114"}

Also, you can add binaries such as images to the CouchDB as well, e.g.

 curl -vX PUT http://127.0.0.1:5984/guestbook/6e1295ed6c29495e54cc05947f18c8af/image.jpg?rev=2-2739352689 -d@image.jpg -H "Content-Type: image/jpg"

Reading documents

CouchDB uses GET operation to read the document and you pass the id of the document, e.g.

 curl -X GET http://127.0.0.1:5984/guestbook/152019530472f7b0b364367bc2ec571d

which returns

 {"_id":"152019530472f7b0b364367bc2ec571d","_rev":"1-3525253587","name":"Sally","message":"hi there"}

Updating documents

CouchDB uses optimistic locking to update documents so this version number must be passed when we update document. Also, CouchDB is append-only database so it will create a new version of the document upon updated. For example, if you type same command again you would see:

 {"error":"conflict","reason":"Document update conflict."}

In order to update the document, the version must be specified, e.g.

 curl -X PUT http://127.0.0.1:5984/guestbook/152019530472f7b0b364367bc2ec571d -d '{"_rev":"1-3525253587", "name":"Sally", "message":"hi there", "date":"September 5, 2009"}'

This will in turn, create a new version and will return:

 {"ok":true,"id":"152019530472f7b0b364367bc2ec571d","rev":"2-1805813096"}

Deleting document/database

You can delete a document using DELETE operation, e.g.

 curl -X DELETE http://127.0.0.1:5984/guestbook/b4bb85ab50271f3d12d25feb219cb66e -d '{"rev":"1-657551114"}'

Similarly, you can delete a database using:

 curl -X DELETE http://127.0.0.1:5984/guestbook

Querying Documents

CouchDB uses Javascript based map and reduce functions to query and view documents, where map function takes a document object and returns (emits) attributes from the document. Here is simplest map function that returns entire document:

 function(doc) {
       emit(null, doc);
 }

Here is another example, that returns names of people who posted to guestbook:

 function(doc) {
     if (doc.Type == "guestbook") {
         emit(null, {name: doc.name});
     }
 }

Reduce function is similar to aggregation functions in most relatinal databases, for example to count all names you could define map function as

 function (doc) {
     if (doc.Type == "guestbook") {
         emit(doc.name, 1);
     }
 }

and reduce function as

 function (name, counts) {
     int sum=0;
     for (var i=0; i<counts.length; i++) {
         sum+=counts[i];
     }
     return sum;
 }

All Databases

You can list names of the database using:

 curl -X GET http://127.0.0.1:5984/_all_dbs

You can also get all documents for a particular database (guestbook):

 curl -X GET http://127.0.0.1:5984/guestbook/_all_docs

CouchDB also comes with a web based Futon application to create, update, and list databases and documents, simply go to http://127.0.0.1:5984/_utils/ and you will all databases in the system.
You can also control replication from that UI, which is pretty handy. Also, you can poll database changes using:

 curl -X GET 'http://127.0.0.1:5984/guestbook/_changes?feed=longpoll&since=2'

Also, you can get statistics using:

 curl -X GET http://127.0.0.1:5984/_stats/

And Config via:

 curl -X GET http://127.0.0.1:5984/_config

Replication

CouchDB is written in Erlang and uses many of internal features of Erlang such as replication of databases (that use Mnesia). In order to replicate, just create a database on another server, e.g.

 curl -X PUT http://127.0.0.1:5984/guestbook-replica

Then replicate using:

 curl -X POST http://127.0.0.1:5984/_replicate -H 'Content-Type: application/json' -d '{"source":"guestbook", "target":"http://127.0.0.1:5984/guestbook-replica"}'

Security

You can add user/password based basic authentication by editing /opt/local/etc/couchdb/local.ini file. You will then need to pass user/password when accessing CouchDB server, e.g.

 
 curl -basic -u 'user:pass' -X PUT http://127.0.0.1:5984/guestbook

Summary

I just started using CouchDB and I am still learning more advanced features and its capabilities in enterprise level environment. Though, it looks very promising, but I am keeping Berkely DB in the back pocket in case I run into severe issues.

August 15, 2009

Releasing Wazil.com

Filed under: Computing — admin @ 11:30 am

I just finished a brand new website Wazil.com and companion facebook app for posting yellow pages and classifieds. I am working on starting a local communities for this website that will show local search results based on your location. Please give it a try and post me your comments and suggestions.

August 6, 2009

How Twitters D.O.S brought my blog site to halt

Filed under: Computing — admin @ 12:17 pm

I noticed my blog was taking a long time to load and then realized that twitter’s denail of service attack effected my weblog. My blog shows latest tweets using javascript and as browsers generally block loading the contents when executing javascript, thus my blog was not showing the contents. I noticed that I was violating one of the commandments of Steve Souders, i.e., put your javascripts at the bottom. After moving the javascript calls to the bottom, my blog started loading happily though without the tweets for now. Also, Steve’s new book shows tons of ways to load javascript asynchronously but I haven’t added that to my blog yet.

July 26, 2009

Day 5 at #oscon 2009

Filed under: Computing — admin @ 11:00 am

July 24, 2009 that was Day 5 of OSCON 2009 for me started with yet another talk by Gunnar Hellekson on using open source for building government projects. This was followed by very entertaining talk by Erik Meijer on “Fundamentalist Functional Programming”. He talked about side-effect free programming and how most functional languages are not pure. He briefly described Monads features of Haskell and how LINQ is influenced by them. Finally, there were keynotes by Karl Schroeder and Mark Surman, which were not very inspiring.

There were not a lot of sessions on last day of the OSCON, I decided to attend The HTML 5 Experiments to learn a bit on new HTML5 tags. Bruce Lawson showed how he implemented his blog using some of HTML5 tags such as header, footer, section, article, time, etc. He also mentioned canvas feature that was interesting but was difficult for people that require assistance technology. Finally, video tags won’t be available anytime soon due to a lot of proprietary decoders.

I then skipped next session and headed to Tech Museum, which is must see if you are visiting San Jose. I then headed to the airport and flew back to Seattle in the evening. Overall, I enjoyed OSCON 2009, I wished there were more talks on functional programming and was disappointed when haskell talk was cancelled. Also, I wish more talks were a bit more hands on like talks on CoucheDB that showed examples of how to use the system instead of just listing out features.

Day 4 at #oscon 2009

Filed under: Computing — admin @ 9:29 am

Thursday, July 23 2009, which was Day 4 for me at OSCON 2009, started with keynote by Kirrily Robert, where she she deplored acceptance of women in open source projects. This was followed by lame keynote by Tony Hey from Microsoft, where the presenter showed bits of open source contributions by Microsoft. Finally, Simon Wardley talked about cloud computing that was pretty entertaining. I then proceeded to attend talk on JRuby on Google App Engine, which didn’t quite kept up to its name and a lot of talk focused on persistence. I attended talk on Eucalyptus, which is an open source project for building private EC2 based cloud. This was sort of marketing talk, but I got a couple of things out such as how Amazon throttles network traffic within a datacenter to 500mb/sec and between zones to 200mb/sec.

I then attended A Survey of Concurrency Constructs, which presented common constructs for concurrency such as locks, transactional shared memory, message passing, dataflow, futures, i-structures, etc. I liked dataflow due to its deterministic nature, but is difficult to implement. I-structures is also interesting, but is non-deterministic and requires ports that make it similar to actors. I also like Linda as it can simulate dataflow, actors and CSP. Finally, message-passing and actors model are poplar these days due to their implementation in Erlang and Scala languages. Ted mentioned how most of the solutions are 20-30 years old, you can read history of most of these solutions from his slides. This was bleak talk as none of the options presented satisfactory option, though his bias was towards JVM based technology and he was impressed with Jonas Boner’s work on AKKA.

Next, I attended talk on Clojure: Functional Concurrency for the JVM, which described functional nature of Clojure and brief overview of its features and syntax. I found calling Java code from Clojure a little verbose especially when you are using method chaining, e.g.

                 factory.newSaxParser().parse(src, handler)
 becomes
                 (.. factory new SaxParser (parse src handler))

Another interesting features of Clojure are its implementation of persistent datastorage and lazy evaluation. Finally, Clojure supports transactional memory for building concurrent applications but there is a little emperical data on its performance and usability. In fact, Ted Sueng mentioned porting some of open source applications to use transactional memory resulted in deadlocks so I am waiting for a little more evidence.

Next, I attended talk on Cassandra: Open Source Bigtable + Dynamo, which is another DHT similar to
Dynomite, Redis, Tokyo Tyrant, Voldemort, HBase, etc. Cassendra is an implementation of DHT based on Amazon Dynamo paper and supports consistent hashing, gossip, failure detection, cluster state, partitioning and replication. I liked the fact that there is no single master as in BigTable so it is easier to scale and uses bloomfilter to keep index of keys. You can read more on its features from the slides.

Last session I attended was “Design Patterns” in Dynamic Languages, where Neal Ford showed how GOF design patterns were created to overcome deficiencies of C++ and he described how dynamic languages like Ruby and Groovy make it trivial to use these patterns without all the ceremony. Neal showed how method_missing can be used to implement builder pattern (though, I prefer not to use method_missing). He showed how each method on array is easier than iterator, how closures can be used to implement command and strategy patterns. Neal then showed, how internel DSLs can be used to implement interpreter pattern. Other examples included decorator and adapter patterns that used invokeMethod feature of Groovy to delegate invocation. Finally, he showed using null object pattern for consistent interface and aridifier to keep your code DRY. You can read more from his slides.

July 25, 2009

Day 3 at #oscon 2009

Filed under: Computing — admin @ 9:44 pm

On the third day (Wednesday, July 22, 2009, the real conference started. The day began with the a couple of keynotes. First, Tim O’reilly talked about Government 2.0, data.gov and other open source organizations that are building applications for the newly opened data. This turned out to be theme of a number of keynote speakers and there was a lot of interest in sunlight labs, http://opensourceforamerica.org/, http://www.gov2summit.com/. Then Dirk Hohndel talked about netbooks and some of innovations from Intel to improve boot time. He deplored state of graphics on Linux that have changed a little in last twenty years. Finally, Mike Lopp, author of Rands in Repose blog talked about how well intentional evil people can ruin companies using Borland as an example.

I started the sessions with Testing iPhone apps with Ruby and Cucumber, which should have been called Testing iPhone GUI apps with Ruby and Cucumber. It was half decent, but the framework had a lot of dependencies that we didn’t go into. I would like to give it a try as testing on Objective-C sucks. I then attended Introduction to Animation and OpenGL on the Android SDK, which seemed too fast and the presenter rambled on miscleneous APIs of OpenGL that I could not follow.

On the second half, I started talk on Automating System Builds and Maintenance with Cobbler and Puppet. This was somewhat useful and I learned a bit to use Cobbler for creating system images and using Puppet for configuration. This was followed by Best practices for ‘scripting’ with Python 3. This was a good talk that described some good principles for writing scripts (as opposed to Python applications). These principles included using optparse for parsing arguments, layers of I/O to help testing (StringIO), using generators for performance and finally using templates for packaging as setup is hard to configure from scratch. I then attended Using Hadoop for Big Data Analysis, which was sort of marketing talk from Cloudera CEO and prsented a few projects that are using Hadoop such as log processing at rackspace, monitoring electircal grid and large hadron collider. Finally, I attended Distributed Applications with CouchDB, which was really good talk on CouchDB by J Chris Anderson from couch.io. It described architecture of CouchDB and features of CouchDB. Chris also gave password for private beta to http://hosting.couch.io, which was “booom-couch”. You can read detailed examples from his slides.

July 23, 2009

Day 2 at #oscon 2009

Filed under: Computing — admin @ 2:34 pm

On the second day of OSCon 2009, I started with PhoneGap tutorial. The PhoneGap is an ambitious project that provides Javascript based unified APIs to develop mobile applications for a variety of mobile platforms such as iPhone, Blackberry, Android, Windows Mobile, Nokia, Palm, etc (most of those are not yet support, but 1.0 is expected in a few months that will have support most of them). It competes with a number of other open source projects such as Joyent Smart platform, Big five, Corona, Nimblekit, Appcellerator, Rhodes, etc. The PhoneGap uses HTML, CSS and Javascript for development and relies on Webkit and HTML5 technologies and standards. Many of mobile platforms such as iphone, android, palmpre support Webkit, though Blackberry and Windows Mobile are exceptions. The PhoneGap uses a number of features of HTML5 such as caching, CSS transformation, fonts, local storage, etc. The PhoneGap uses XUI, which is a subset of jQuery as some of the platforms such as iPhone provide limited caching (25K) for Javascript. It uses selectors and CSS for animations. The session introduced Dashcode tool that comes with XCode to build web applications and then converting those web applications into native applications using PhoneGap. The presenation for this session is available from http://presentations.sintaxi.com/oscon/

For the second half I decided to attend “Scalable Internet Architectures” — more than 10 million consumers/day. It was interesting talk that discussed building scalable architectures from hardware and networking perspective. It empahsized awareness on end-to-end architecture including javascript, application, database, network and machines and stressed importance of including people from operations in the architecture of the system. The presenter suggested use of CDN for static contents and using peer-based HA instead of load balancers as it eliminates load balancers as point of contention or failures. The speaker also suggested use of reverse proxy cache such as Varnish or Squid. He also suggested setting up multiple DNS servers for each data center and registering local servers with local DNS so that they take advantage of shortest path routes and talk to local servers. Other suggestions included use of caching, avoiding 302 redirects, separtion of OLTP and OLAP databases, use of DHT. The speaker also pointed to a number of networking techniques such isolating network for different services to prevent starvation of bandwidth when one of the service is surging the network with high dataload by using mac based filtering.
The speaker mentioned a number of usability techniques to offload expensive operation or hinting users when something is going on in the background. He mentioned use of queuing technology for offload processing. Finally, the speaker talked about a number of lesson learned from scaling and some of big WTF moments from his consulting work. Overall, this talk summarized a lot of existing knowledge for building scalable applications (such as from Steve Souders work) with a couple of new networking techniques to tackle slashdot or denial of service attack. The slides from this talk are available at http://www.slideshare.net/postwait/scalable-internet-architecture.

July 22, 2009

Day 1 at #oscon 2009

Filed under: Computing — admin @ 5:25 pm

The first day of OSCon 2009 covered a number of tutorials and I decided to attend Google App Engine tutorial for the first half of the day. Google App Engine API follows CGI model of web development, i.e., it uses stdin and stdout files and assumes stateless applications. There is limit of 10MB response size and 30 requests per second, but it does not allow streaming. The tutorial started pretty slow and we spent first hour just installing the SDK and tutorial. The Google App Engine SDK is available from http://code.google.com/appengine/downloads.html. I downloaded Mac image and then dragged the image to my hard drive. I then double clicked the app icon for Google Appe Engine SDK, which installed the SDK under /usr/local/google_appengine. Once the SDK is installed, you have to install Google App Engine tutorials from http://code.google.com/p/app-engine-tutorial/.

After installing SDK and tutorial, I copied all files named tutorial? under the SDK. The rest of session covered those tutorials one by one, though we ran out of time in the end and completed only upto tutorial7. In order to install first tutorial, I went into tutorial1 directory, e.g.

 cd /usr/local/google_appengine/tutorial1

Then started local app server as follows:

 python ../dev_appserver.py .

When I pointed my browser to the http://localhost:8080, I was able to see “Hello World!”.

Next, I registered myself to http://appspot.com. After registering, I received an SMS message for confirmation and was able to fully register after entering the confirmation number. Next, I created an application-id on Google App Engine. You can only create 10 app-ids and you cannot delete app-ids, so be careful with ids. Also, you can also use your own domain instead appspot.com. For my testing purpose, I chose the id “shahbhat”.

Next, I changed app.yaml inside my local tutorial1 directory that describes how your application is configured. You may also notice index.yaml, which describes list of indices in the database, though Google App Engine can figure out what queries are being used and creates indices automatically. I changed application name in app.yaml to “shahbhat”, e.g.

 application: shahbhat

I then pushed my application to the Google App Engine by typing

 python ../appcfg.py update .

I was then able to go to http://shahbhat.appspot.com/ and see my application, Voila. You can also see your application usage from http://appengine.google.com/dashboard?app_id=shahbhat (you will have to change app_id parameter in your application).

Unfortunately, a lot of people had problems getting to that state so we wasted another half hour in break where other folks sort out configuration and deployment issues.
Next, I went through another tutorial to turn on authentication by setting:

 login: required

in app.yaml file. Next I added caching by adding expires options in the app.yaml. I was also able to use curl to test my applications and see headers to verify caching, e.g.

 curl --include http://localhost:8080

Which showed following when caching was not configured:

 Cache-Control: no-cache

When I configured the caching to 2d, I was able to see:

 Cache-Control: public, max-age=172800

The Google App Engine SDK also includes development that you can view by going to:

 http://localhost:8080/_ah/admin

The Google App Engine supports Django based templates, e.g.

 #!/usr/bin/env python
   
   import os
   from google.appengine.ext.webapp import template
   
   def main():
       template_values = {"foo" : [1,2,3]}
       template_file = os.path.join(
                    os.path.dirname(__file__), "index.html")
       body = template.render(
         template_file, template_values)
       print "Status: 200 OK"
       print "Content-type: text/html"
       print
       print body
   
   if __name__ == '__main__':
     main()

In addition, Google App Engine supports WSGI standard (PEP 333), e.g.

   import os
   import wsgiref.handlers
   
   from google.appengine.ext import webapp
   from google.appengine.ext.webapp import template
   
   class IndexHandler(webapp.RequestHandler):
   
     def get(self):
       template_values = {"foo" : 1}
   
       template_file = os.path.join(os.path.dirname(__file__), "index.html")
       self.response.out.write(template.render(template_file, template_values))
   
   
   def main():
     application = webapp.WSGIApplication([('/', IndexHandler)], debug=True)
     wsgiref.handlers.CGIHandler().run(application)
   
   
   if __name__ == '__main__':
     main()

Other tutorials included authentication APIs such as

 create_login_url(dest_url)
 create_logout_url(dest_url)
 get_current_user()
 is_current_user_admin()

The SDK also includes decorator to add authentication automitcally using 

 from gogole.appengine.ext.webapp.util import login_required
 ...
     @login_required
     def get(self):

Finally, we went over datastore APIs for persistence support, e.g.

   import os
   import wsgiref.handlers
   
   from google.appengine.ext import webapp
   from google.appengine.ext.webapp import template
   from google.appengine.ext import db
   
   class ToDoModel(db.Model):
     description = db.StringProperty()
     created = db.DateTimeProperty(auto_now_add=True)
     foo = db.FloatProperty(default=3.14)
     bar = db.IntegerProperty()
     baz = db.BooleanProperty(default=False)
     N = db.IntegerProperty()
     l = db.ListProperty(str, default=["foo", "bar"])
   
   
   class IndexHandler(webapp.RequestHandler):
   
     def get(self):
       todo = ToDoModel(description = "Hello World", bar=1, baz=True)
       todo.put()
   
   def main():
     application = webapp.WSGIApplication([('/', IndexHandler)], debug=True)
     wsgiref.handlers.CGIHandler().run(application)
   
   if __name__ == '__main__':
     main()

You can view the model by going to http://localhost:8080/_ah/admin/datastore. The data store supports a number of types such as string, boolean, blob, list, time, text. However, there are some limitations,
e.g. TextProperty can only store upto 500 bytes and Google App Engine will create index if needed, however it won’t create index on TextProperty. For each row, the datastore assigns a numeric id and UUID based key,
though you can provide your own key. Also, a row cannot exceed 1MB.
Unfortunately, we ran out of time At this time, so I had to go to http://code.googlecom/appengne/docs for further documentation. Overall, I thought it was good introduction to Google App Engine, but I was disappointed that instructor wasted a lot of time with setup that could have been used to cover rest of the tutorials.

For the second half of the day, I attended session on “Building applications with XMPP”. This was interesting session that showed usage of XMPP for a number of usecases such as IM, gaming, real-time social networking, , monitoring, etc. The session started with history of XMPP (Extensible Messaging and Presnce Protocol), its Jabber roots and its use of streaming XML. A number of factors contributed to the popularity of XMPP such as open source, XML, federeated network, low latency, etc. The XMPP is also very extensible and supports audio/video via jingle, geo location, TLS, SASL, etc. XMPP architecture is based on client server, where servers are decentralized and federated. XMPP identifies a user with jabber id that looks like email address and consists of local part, domain and resource, e.g. alise@wonderland.lit/TeaParty, where domain is mandatory, but local-part and resource are optional. There are three types of XMPP message stanzas, i.e., presence, IQ, message. The presence-stanza is asynchronous, but IQ stanza requires response. As opposed to web architecture that is based on short lived connections and stateless architecture, XMPP uses one long lived session and events are received asynchronously.

Next, the tutorial showed how to build a javascript client for jabber using sleekxmpp and Strophe library (alternatively you can use twistedword). The example used Bosh protocol to wrap XMPP protocol with HTTP protocol. Unfortunately, there was a lot of fast typing and I could not follow all that. I am waiting for the presenters to post the slides online so that I can use those examples in my own applications.

« Newer Posts — Older Posts »

Powered by WordPress