Shahzad Bhatti Welcome to my ramblings and rants!

June 1, 2011

Deploying Rails 3.0 App on Amazon EC2

Filed under: EC2 — admin @ 12:56 pm

It’s been a few years since I wrote a short HOW-TO on working with EC2 , but recently I tried to migrate the backend of Trading Floor – Facebook and iOS game I have been developing to EC2. So I am documenting the steps for setting up the EC2 for Rails 3.0.

Pre-requisites

I assume you already signed up for EC2, otherwise go to http://aws.amazon.com/ec2/ to signup. Also, you will need Java 5.0 or above, which you can download it from Oracle.

Download EC2 API Tools

First, download EC2 from http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351 and uncompress it in your root directory.

Create a X.509 Certificate

Next, create a X.509 certificate from the AWS Account section. You can then download your certificate and key safely, e.g. I saved them in .ec2 directory under my home directory. Note that you will not be able to download the key again, so don’t lose it.

Environment Variables

Next, I changed my shell as:

 export EC2_HOME=~/ec2-api-tools-1.4.2.4
 export PATH=$PATH:$EC2_HOME/bin
 export EC2_KEY_DIR=~/.ec2
 export EC2_PRIVATE_KEY=$EC2_KEY_DIR/pk-HFG55OCFPZARA6YHW5JGIE6JFD7EQE72.pem
 export EC2_CERT=$EC2_KEY_DIR/cert-HFG55OCFPZARA6YHW5JGIE6JFD7EQE72.pem
 

Where EC2_PRIVATE_KEY and EC2_CERT points to the X.509 key and certificate I downloaded from the Amazon.

Create a Key-Pair

Then I created a pair of keys as:

 ec2-add-keypair plexobject
 

Create a Security Group

I then created a security group for the server

 ec2-add-group web -d 'Web Server'
 ec2-authorize web -P tcp -p 22 -s 0.0.0.0/0
 ec2-authorize web -P tcp -p 80 -s 0.0.0.0/0
 ec2-authorize web -P tcp -p 443 -s 0.0.0.0/0
 

Finding a basic Ubuntu based AMI

Previously I used S3 based AMI, but Amazon now supports EBS based AMIs that has advantage that any changes to the root survive instances of EC2. I launched EC2 instance with basic Ubuntu 11.0 Natty from http://alestic.com/ as:

 ec2-run-instances ami-06ad526f --instance-count 1 --instance-type m1.small \
 --key plexobject --group web -z us-east-1d -m
 

Where -z describes the availability zone and -m turns on monitoring.

Installing Ubuntu Packages

I then proceeded to install basic packages such as Java, Curl, Git, Build, Ruby (1.8.7) and Rails (3.0.3) based on Rails/Ubuntu docs such as:

 sudo apt-get install openjdk-6-jdk
 sudo apt-get install mercurial
 sudo apt-get install curl git-core build-essential zlib1g-dev libssl-dev libreadline5-dev
 sudo  apt-get install libcurl4-openssl-dev 
 sudo apt-get install ruby
 sudo apt-get install rubygems1.8
 sudo gem install rubygems-update
 sudo update_rubygems
 sudo gem install rails
 

I then edited /etc/profile /etc/bash.bashrc and added environment variables

 export PATH=$PATH:/var/lib/gems/1.8/bin/
 export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk/
 

Next, I installed Sqlite and Mysql:

 sudo apt-get install sqlite3 libsqlite3-dev
 sudo gem install sqlite3-ruby
 sudo apt-get install mysql-server mysql-client
 sudo apt-get install libmysql-ruby libmysqlclient-dev
 

Next, I installed Apache and Passenger:

 sudo apt-get install apache2 apache2-mpm-prefork apache2-prefork-dev
 sudo apt-get install apache2-dev libapr1-dev libaprutil1-dev
 sudo gem install passenger
 sudo /var/lib/gems/1.8/gems/passenger-3.0.7/bin/passenger-install-apache2-module*
 

I then edited /etc/apache2/apache2.conf and added:

 LoadModule passenger_module /var/lib/gems/1.8/gems/passenger-3.0.7/ext/apache2/mod_passenger.so
 PassengerRoot /var/lib/gems/1.8/gems/passenger-3.0.7
 PassengerRuby /usr/bin/ruby1.8
 

and then restarted apache

 /etc/init.d/apache2 restart
 

Creating EBS Volume for Data

Next, I created an EBS volume to store all data such as database tables and Rails application from the AWS Console and then attached it to the instance as:

 ec2-stop-instances i-73ab181d 
 ec2-attach-volume  vol-612eaa0a -i i-73ab181d -d /dev/sdf
 ec2-start-instances i-73ab181d 
 

Note that you have to create the EBS volumen in the same availability zone as your instance. I then logged into my machine using

 ssh -i plexobject.pem ubuntu@ec2-50-19-134-251.compute-1.amazonaws.com
 

and then formatted the newly built EBS volume as:

 sudo fdisk -l
 sudo mkfs -t ext4 /dev/xvdf
 

I then edited /etc/fstab and added

 /dev/xvdf       /data   auto    defaults,nobootwait,noatime     0       0
 

and then rebooted machine

 sudo reboot
 

Moving Mysql Data Directory

Mysql installs data directory on the root volume in /var/lib/mysql directory, which I wanted to move to newly created volume. So I created a directory /data/mysql so I stopped mysql:

 sudo /etc/init.d/mysql stop
 

I then copied mysql data directory such as:

 sudo cp -R -p /var/lib/mysql/mysql /data/mysql
 sudo chown -R mysql:mysql /data/mysql/
 

I didn’t copy entire mysql directory, only mysql subdirectory. Next I edited /etc/mysql/my.cnf and changed datadir to /data/mysql directory and then edited /etc/apparmor.d/usr.sbin.mysqld and changed all /var/lib/mysql to /data/mysql. Finally I restarted AppArmor profiles as:

 sudo /etc/init.d/apparmor reload
 

Then restarted mysql:

 sudo /etc/init.d/mysql restart
 

I changed my root password and created a local mysql user as

 mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('mypass');
 mysql> GRANT ALL PRIVILEGES ON *.* TO 'tfuser'@'localhost' IDENTIFIED BY 'mypass' WITH GRANT OPTION;
 mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES ON dbname.* TO 'tfuser'@'localhost' IDENTIFIED BY 'mypass';
 

I copied my app to /data/trading_floor and changed permissions of all files to www-data

 sudo chown -R www-data:www-data /data/trading_floor
 

Then created /etc/apache2/sites-available/tf with

 
         ServerAdmin shahbhat@gmail.com
         ServerName tf.plexobject.com
         DocumentRoot /data/trading_floor/public/
         
                  AllowOverride all
                 Options -MultiViews
                 RailsEnv production
          
 
 

Finally, I restarted apache

 /etc/init.d/apache2 restart
 

Creating Elastic IP address

I wanted a permanent IP address for the server so I created EIP using AWS Console. Then associated my instance with the new IP address:

 ec2-associate-address 50.19.248.7 -i i-73ab181d
 

It rebooted your machine with the new IP address. I then changed DNS zone and pointed tf.plexobject.com to 50.19.248.7 (this may take hours to propagate). Next, I changed my Facebook app’s configuration and iOS app’s configuration to point to tf.plexobject.com.

Creating my EBS Image

Once I was happy with the server configuration, I created EBS image for future use. First, I detached data volume and then created image as follows:

 ec2-stop-instances i-f97ca197
 ec2-detach-volume vol-3f65d954
 ec2-create-image i-f97ca197 -n tf-20110601 -d 'Trading Floor Application Server'
 

I terminated my previous instance as

 ec2-terminate-instances -i  i-f97ca197 
 

and created instance with the new image

 ec2-run-instances ami-80837ae9 --instance-count 1 --instance-type m1.small --key tf --group web -z us-east-1d -m
 

After the launch, you would have to reattach the data volume

 ec2-stop-instances i-73ab181d 
 ec2-attach-volume  vol-612eaa0a -i i-73ab181d -d /dev/sdf
 ec2-start-instances i-73ab181d 
 

Summary

Voila, I had my game application running on the cloud. In order to cut per/hour cost I reserved instances for entire year. I am not quite done with my server and am now working on application specific configuration and adding better monitoring/backup. As, we have learned from recent Amazon Cloud outage that deploying your app on the cloud is only half the work, making it performant, scalable and fault tolerant is other half which is still manual work. Finally, I plan to release the Facebook app for Trading Floor and submit iOS app in a couple of weeks, be sure to try it and send me your suggestions.

April 25, 2011

Review of Guy Kawasaki’s book – “Enchantment: The Art of Changing Hearts, Minds, and Actions”

Filed under: Business — admin @ 12:23 pm

I recently read Guy Kawasaki’s book Enchantment: The Art of Changing Hearts, Minds, and Actions. This book shows how to engage with other people and build better relationships similar to Dale Carnegie’s book How to Win Friends & Influence People. Though, this book covers these topics in more professional context and it includes advice from several other business and management books. As, Guy is also a very savvy social media user, this book covers several tips on using modern networking tools to build personal relationships with others.

Enchantments

Guy describes enchantments as a way of delighting people with a product, service or organization, which is similar to the concept of Customer Delight popular in business literature. Guy suggests to start with a good product or service and fill people with the delight. This also reminded me Tony Hsieh’s book Delivering Happiness: A Path to Profits, Passion, and Purpose.

Likable and Trustworthy

Once you have a good product, you build the enchantments by being likable and trustworthy. The likability chapter covers several pointers such as smile, dress appropriately, firm handshake, accept others, yes attitude, and work in open environment. Guy encourages finding shared interests with other party and creating win-win situation when negotiating. On being trustworthy, Guy suggests giving people benefit of doubt, disclosing interests and positioning yourself. Some of these techniques seemed similar to what I have read from Stephen Covey’s book The 7 Habits of Highly Effective People and from agile development gurus.

Pre-Launch/Premortem

Guy gives a great set of tips on preparation before launching a product and suggests the product should be:

  • great
  • deep
  • intelligent
  • complete
  • empower
  • elegant

Product Launch

On launching a new product, Guy suggests telling personal stories, showing courage, planting many seeds and aspiring people by promising a better world. This chapter reminded me of how Steve Jobs promotes Apple products by promising better future, giving great demo, and simplifying the interface.

Overcoming Resistance

On overcoming resistance, Guy suggests creating perception of ubiquity and scarcity and finding a way to agree, which enhances your chances of being likable. I found the chapter on overcoming resistance a bit weak and encourage readers to look at Switch: How to Change Things When Change Is Hard and Fearless Change: Patterns for Introducing New Ideas.

Enchanting Influencers

Guy offers a great practical guidance on enchanting influencers such as working on grassroots, creating intrinsic motivation, paying it forward, and reciprocity. I liked his advice of saying “I know you’d do same for me” instead of saying “You’re welcome” in response to thank-you”.

Ecosystems

In order to create a grassroots support of your products or services, Guy recommends creating a product worthy of ecosystem and then lists several tools, which encourage exchange of ideas and collaboration such as user-groups, blogs, conferences, reward system, open architecture. Another key factor for ecosystem is having a diversified team, which different roles such as advocate, skeptic, visionary, adult, evangelist and rain maker.

Push Technology

This is one of best chapters in the book and shows how to use modern push technologies such as Presentations, Email, and Twitter. Guy recommends engaging many people fast and often. He also recommends giving them credit and providing a value for them. On presentations, Guy recommends customizing intro based on audience, selling dreams, dramatizing and rehearsing it. He suggests keeping the presentation short with 10-20-30 rule, where presentation has no more than 10 slides, takes 20 minutes and uses no less than 30-size font. For email, Guy suggests keeping it short (under six sentences) and asking for a specific action.

Pull Technology

On pull technologies, Guy suggests creating a website/blog with good content, refreshing contents frequently and having an about page. On Facebook, Guy suggests having a good landing page and being helpful. On Linked-in, Guy suggests having a great profile and reaching out to others actively.

Enchanting Employees

Guy also provides useful set of pointers on being a good employer such as engaging employees by providing MAP (Mastery, Autonomy, Purpose) and empowering employees to do the right things. He recommends instead of judging actions of others against their intentions, be harsh on yourself and judge your results against their intentions. He also suggests celebrating success and includes tips from Good Boss, Bad Boss: How to Be the Best… and Learn from the Worst such as protecting people from intrusions. Guy cites Michael Lopp’s advice from Managing Humans: Biting and Humorous Tales of a Software Engineering Manager such as setting ambitious goals, enabling, appreciating and providing feedback to the employees.

Enchanting Boss

On enchanting boss, Guy recommends:

  • make your boss look good
  • drop everything when boss asks for something
  • under-promise and over-deliver
  • prototype work by completing part of assignment and asking for feedback
  • show and broadcast progress while giving credit to colleagues who helped
  • form friendship
  • ask for mastership
  • deliver bad news early

Resist Enchanters

Finally on resisting enchanters, Guy suggests looking far in future, knowing your limits, having a skeptic attitude and not falling for example of one.

Summary

In this book, Guy Kawasaki provided a good collection of practical advice on building better interpersonal relationships and using tools from social media effectively. It shows that in order to build long lasting relationships, you have to be sincere and always be willing to help others. I found Guy’s pointers on push and pull technologies most helpful as he has created cult of followers on Twitter and Facebook and provided a number of tips from his personal experience.

February 9, 2011

Tips and Idioms for Developing Network based iOS Applications

Filed under: iPhone development — admin @ 12:21 pm

Over the last year, I built a number of iOS applications such as yogaspot, iPlexLotto and OptionsHouse (still in development), which are backed by the network services. I am documenting a few idioms and best practices that I have used in these applications along with some lessons learned:

Separation of concerns

The iOS APIs are based on MVC design pattern, which encourages separating the non-GUI code such as Models, Services, Helper classes from your views and uses Controllers to communicate between GUI and non-GUI code. As with building any other kind of application, keeping your design and code clean would make your application more maintainable and extendable. The iOS APIs make heavy use of protocols for interface segaration principle. Here are some of the usages of protocols in iOS APIs:

Data Sources

The iOS API use protocols for data sources, which then populate views such as table views, pickers and other views. In most application, your controller will be the data source but I used the data source protocols to create Wizard like behavior by implementing the data source interface in multiple classes and displaying different contents based on the state of application.

Delegates

The delegates are callback interface that are generally used for communication between classes or threads. As, protocols in Objective C can define both mandatory and optional behavior, delegates are also used to provide customized UI behavior. The iOS SDK uses delegates for customizing UI in a number of classes such as UISearchDisplayDelegate, UISearchBarDelegate, UIGestureRecognizerDelegate, UITableViewDelegate, etc. Here is an example of delegate for communication between classes:

 @protocol OrderCallbackDelegate
 @required
 - (void) addedOrder: (Order *)order;
 @end
 
Implementing Delegate

@protocol OrderCallbackDelegate
@interface OrderCallbackDelegate : NSObject {
}
@end

Referencing Delegate
 @interface SomeObject : NSObject {
     id delegate;
 }
 @property(nonatomic, retain) id delegate;
 @end
 

Encapsulation and Extensions

The iOS provide several options for hiding private details or extending existing classes. For example, you can use @private keyword in the class declaration to enforce encapsulation such as:

 
 @interface MyClass : NSObject {
     @private
     int myVar;
 }
 @end
 

The myVar attribute can only be accessed by the instance of MyClass.

Objective C supports inheritance and composition as other object-oriented languages, which I won’t cover here. However, the Categories feature of Objective C language is a more powerful way to extend an existing class without modifying it. For example, you can extend UIColor to provide additional colors as follows:

 @interface UIColor(MyBlueColor)
 +(UIColor *)myBlueColor;
 @end
 
 @implementation UIColor(MyBlueColor)
 +(UIColor *)myBlueColor {
     return [UIColor colorWithRed:0.2274 green:0.4666 blue:0.8235 alpha:1.000];
 }
 
 @end
 
 

Other examples include JSON parsing library, which extends NSString to support JSON parsing.
In addition to extension, you can also use categories for encapsulation by using categories syntax in the implementation file and adding private method and properites, e.g.

 @interface MyClass(private)                                                                                                                                                                                                                                                 
 //my private methods only visible in the implementation file
 @end
 

Code Structure

As with any other application, keeping your code organized in your iOS project helps readability and maintenance. One thing I don’t like about XCode is that when you create groups for organizing classes in your XCode project, it doesn’t create folders, so you will find all classes in Classes directory. So, I prefer creating folders on the file system and then add folders as groups to XCode. Here is an example of different groups I used in my projects:

  • Models
  • Network
  • Controllers
  • Utils
  • OtherSources
  • Parsing
  • Views
  • Resources

Dynamic Language Support

Objective C supports both static and dynamic language syntax and has powerful support of reflection. This gives you powerful tool to write concise code. For example, I could not find any XML or JSON binding library for Objective C, so I built helper classes that parsed XML and JSON and automatically bound the properties to the classes that used same names, e.g.

 typedef id (^objectFactoryBlk_t)();
 
 @interface JsonParser : NSObject {
 }
 + (void)bindJson:(NSData *)data toObject:(NSObject *) object;
 + (void)bindJson:(NSData *)data toArray:(NSMutableArray *) array mandatoryFields:(NSArray *)fields withFactory:(objectFactoryBlk_t) objectFactoryBlk;
 
 @end
 

And

 
 @interface XmlParser : NSObject {
     xmlDocPtr doc;
 }
 - (id)initWithXML:(NSData *) document;
 + (void)bindXml:(NSData *)xml toObject:(NSObject *) object;
 + (void)bindXml:(NSData *)xml toArray:(NSMutableArray *) array  mandatoryFields:(NSArray *)fields withFactory:(objectFactoryBlk_t) objectFactoryBlk;
 @end
 

I am skipping more details but you can download the implementation from Parser.zip.
Also, as Objective-C is dynamic languages, the compiler won’t complain if you call methods that are not defined, so you need to pay special attention to warnings when using dynamic language constructs or better turn warnings into errors by configuring your build. You can also use CLANG static analyzer to find improper dynamic method.

Network Services

Any network based iPhone app would rely on the backend network services and you would make these easier if these services are REST based. I recommend ASIHTTPRequest library, which supports asynchronous communication and concurrency.

Loading Status

When designing an modal dialog that interacts with the backend, it helps if you show loading message or use UIActivityIndicator when making network request (even though the network request would be asynchronous). For example, the OptionsHouse app shows loading status when executing trades as it may take a couple of seconds to communicate with the server.

XML vs JSON vs Binary

When you are developing an iPhone application, you will learn that despite the increasing powerful hardware it’s not same as desktop, laptops or servers we have in the office. Thus, you will have to learn how to optimize every aspect of the application and one of the key factor for network application is the protocol for the payload. I used both XML and JSON based services and found that JSON parsing was at least twice as fast compare to XML. Optionally, you can use Binary format such as binary plist, which further improves the performance of parsing.

Network Latency

An iPhone user might be connected on Edge, 3G, 4G or WiFi network and the network speed would vary. Though, you can make a general assumption that WiFi would be faster than 3G/4G, which would be faster than Edge, but I suggest that you test the speed within the app instead of relying on the connection. For example, you might be using WiFi at a local coffee shop or a conference, where the speed is slower than the edge or 3G.

Network Outages

On websites, the network services are often behind the web server so when they are down it’s easy to post a page for network outage. You will have to handle such outages in the application and notify the users about the outages.

Network Data Requests

A network based application requires fetching the data from the server and it’s best that you communicate with the services without blocking user interface. Both ASIHTTPRequest and NSURL supports asynchronous communcationions, where you register callback methods when the data arrives or network failure occurs. I built a thin layer on top of ASIHTTPRequest so that I could use blocks:

 typedef void (^requestCompletedBlk_t)(id payload);
 typedef void (^requestFailedBlk_t)(NSError *error);
 
 @interface ServiceBlocks : NSObject {
     requestCompletedBlk_t requestCompletedBlk;
     requestFailedBlk_t requestFailedBlk;
     ASIHTTPRequest *request;
     NSTimeInterval started;
     int retries;
 } 
 @property (copy, nonatomic) requestCompletedBlk_t requestCompletedBlk;
 @property (copy, nonatomic) requestFailedBlk_t requestFailedBlk;
 @property (retain, nonatomic) ASIHTTPRequest *request;
 @property (assign, nonatomic) NSTimeInterval started;
 @property (assign, nonatomic) int retries;
 
 @end
 
 @implementation ServiceBlocks
 @synthesize requestCompletedBlk, requestFailedBlk, request, started, retries;
 -(id)initWithRequest:(ASIHTTPRequest *) arequest completedBlock:(requestCompletedBlk_t) arequestCompletedBlk failedBlock:(requestFailedBlk_t) arequestFailedBlk {
     if(self = [super init]) {
         self.requestCompletedBlk = arequestCompletedBlk;
         self.requestFailedBlk = arequestFailedBlk;
         self.request = arequest;
         self.started = [[NSDate date] timeIntervalSince1970];
         self.retries = 0;
     }
     return self;
 }
 
 - (void)dealloc {
     [requestCompletedBlk release];
     [requestFailedBlk release];
     [request release];
                                                                                                                                                                      
     [super dealloc];
 }
 @end
 

Here is how you can use ASIHTTPRequest to make REST POST and GET requests:

 - (void)postJson:(NSData *) json completedBlock:(requestCompletedBlk_t)requestCompletedBlk failedBlock:(requestFailedBlk_t) requestFailedBlk {                       
     NSURL *url = [NSURL URLWithString:[[Configuration sharedConfiguration] getJsonServerUrl]];
     NSString *jsonBuffer = [[[NSString alloc] initWithData:json encoding:NSASCIIStringEncoding] autorelease];
     
     ASIHTTPRequest *request = [ASIHTTPRequest requestWithURL:url];
     request.connectionCanBeReused = YES;
     [request setUseKeychainPersistence:YES];
     [request setTimeOutSeconds:60];
     [request setValidatesSecureCertificate:NO];
     [request setDelegate:self];
     [request setShouldPresentAuthenticationDialog:YES];
     [request setDidFinishSelector:@selector(downloadCompleted:)];
     [request setDidFailSelector:@selector(downloadFailed:)];
     [request appendPostData:json];
 
     [request addRequestHeader:@"Content-Type" value:@"application/json"];
     [request setRequestMethod:@"POST"];
     [request buildPostBody];
     ServiceBlocks *blocks = [[ServiceBlocks alloc] initWithRequest:request completedBlock:requestCompletedBlk failedBlock:requestFailedBlk];
     [requests setValue:blocks forKey:[self toKey:request]];
     [blocks release];
     [request startAsynchronous];
 }
 
 
 - (void)get:(NSString *) uri completedBlock:(requestCompletedBlk_t) requestCompletedBlk failedBlock:(requestFailedBlk_t) requestFailedBlk {
     NSURL *url = [NSURL URLWithString:uri];                                                                                                                          
     ASIHTTPRequest *request = [ASIHTTPRequest requestWithURL:url];
     [request setUseKeychainPersistence:YES];
     [request setDelegate:self];
     [request setShouldPresentAuthenticationDialog:YES];
     [request setDidFinishSelector:@selector(downloadCompleted:)];
     [request setDidFailSelector:@selector(downloadFailed:)];
 
     ServiceBlocks *blocks = [[ServiceBlocks alloc] initWithRequest:request completedBlock:requestCompletedBlk failedBlock:requestFailedBlk];
     [requests setValue:blocks forKey:[self toKey:request]];
     [blocks release];
     [request startAsynchronous];
 }
 
 
 

Above methods setup downloadCompleted as a callback handler for successful completion of the request and downloadFailed for service failure, which are defined as follows:

 - (void)downloadFailed:(ASIHTTPRequest *)theRequest {
     ServiceBlocks *blk = [requests valueForKey:[self toKey:theRequest]];
     if (blk != nil) {
         NSTimeInterval interval = [[NSDate date] timeIntervalSince1970] - blk.started;
         NSLog(@"#####downloadFailed for %@ in %f seconds", [self toKey:theRequest], interval);
         
         if (blk.requestFailedBlk != nil) {
             blk.requestFailedBlk([theRequest error]);
         }
         [requests removeObjectForKey:[self toKey:theRequest]];
     } else {
         NSLog(@"#####downloadFailed could not find blocks for %@, available %@", [self toKey:theRequest], [requests allKeys]);
     }
     
     [[UIApplication sharedApplication] setNetworkActivityIndicatorVisible:NO];
 }                                                                                                                                                                    
 
 - (void)downloadCompleted:(ASIHTTPRequest *)theRequest {
     ServiceBlocks *blk = [requests valueForKey:[self toKey:theRequest]];
     if (blk != nil) {
         NSTimeInterval interval = [[NSDate date] timeIntervalSince1970] - blk.started;
         NSLog(@"################################downloadCompleted for %@  in %f seconds", [self toKey:theRequest], interval);
 
         NSData *content = [theRequest responseData];
         if (blk.requestCompletedBlk != nil) {
             blk.requestCompletedBlk(content);
         }
         [requests removeObjectForKey:[self toKey:theRequest]];
     } else {
         NSLog(@"################################downloadCompleted could not find blocks for %@, available %@", [self toKey:theRequest], [requests allKeys]);
     }
 
     [[UIApplication sharedApplication] setNetworkActivityIndicatorVisible:NO];
 
 }
 

Both methods lookup the user’s block that was passed upon request and then callback the proper block. In certain cases, I extended downloadFailed method to automatically retry the request upon certain failure types such as timeout errors.

Gateway and Proxy Patterns

I used Gateway and Proxy patterns to wrap remote services in a local class that defined methods using local Model classes and automatically converted Models into proper XML/JSON format for requests and responses. This kept the code clean and in a few cases, I had to switch from XML to JSON, which was made in these classes and rest of application remained unchanged.

Check-If-Modified

As both iPlexLotto and OptionsHouse refreshed data continuously in background, I needed an efficient way to check if the data is updated on the server. HTTP provides a great way to reduce bandwidth by using E-Tag or Last-Modified headers, and we used similar techniques to send modified timestamp as part of the response and used it in future requests. The server then compared the timestamp in the request with local state and sent response only if the data was modified.

Network Errors

A critical aspect of network based applications is handling network errors. There can be various reasons for the network errors such as phone might be in Airplane mode so no network requests should be made. In other cases network connection might be too slow or the service is down. Also, you may be continuously updating data in background and in some cases occasional network interruptions can be ignored. This requires careful consideration of the UI so that the application is smart about the environment and retries when possible instead of annoying users with popup alerts.

Caching

A quick way to improve performance is to cache contents for a certain period of time depending on the data. As iOS devices have limited memory, I used in-memory cache for small objects and used file based cache for larger objects. For example, I cached lottery results for a couple of hours, on the other hand for trading application I cached orders and positions for a few seconds. I didn’t’ cache quotes data as it changed frequently and showing stale quotes can result in financial loss. In addition to the user data, I also cached chart images or other meta-data that is not frequently changed. Here is a simple implementation of file based cache:

 
 #define K_CACHE_TIMEOUT_SECS 3600
 @interface FileCache : NSObject {
 }
 
 +(BOOL)saveObject:(NSString *)name withData:(id)entry;
 +(id)loadObject:(NSString *)name expired:(BOOL*)expired;                                                                                                                                                                                                                      
 
 @end
 
 
 #import "FileCache.h"
 
 @implementation FileCache
 
 +(id)loadObject:(NSString *)name expired:(BOOL*)expired {
     *expired = YES;
     NSString *docs = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) lastObject];
     NSString *filePath = [docs stringByAppendingPathComponent:[NSString stringWithFormat:@"%@.obj", name]];
     NSError *error;
     NSFileManager *fileManager = [NSFileManager defaultManager];
     NSDictionary *attrs = [fileManager attributesOfItemAtPath:filePath error:&error];
     id obj = nil;
     if (attrs != nil && [attrs count] > 0) {
         NSDate *date = [attrs valueForKey:@"NSFileModificationDate"];
         if (date != nil) {
             NSTimeInterval interval = [[NSDate date] timeIntervalSinceDate:date];
             if (interval <= K_CACHE_TIMEOUT_SECS) {
                 *expired = NO;
             }
             obj = [NSKeyedUnarchiver unarchiveObjectWithFile:filePath];
         }
     }
     return obj;
 }
 
 
 +(BOOL)saveObject:(NSString *)name withData:(id)entry {
     NSString *docs = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) lastObject];
     NSString *filePath = [docs stringByAppendingPathComponent:[NSString stringWithFormat:@"%@.obj", name]];
     return  [NSKeyedArchiver archiveRootObject:entry
                                               toFile:filePath];
 }
 @end
 

Using Timers for Repeatable Tasks

Both iPlexLotto and OptionsHouse continuously updated the data in background. For example, the OptionsHouse application fetched stock/option quotes, order transactions, positions in background. I used GCD's based Timers to schedule periodic download of the data:

 dispatch_time_t now = dispatch_walltime(DISPATCH_TIME_NOW, 0);
 dispatch_source_t timer = dispatch_source_create(DISPATCH_SOURCE_TYPE_TIMER, 0, 0, dispatch_get_global_queue(0, 0));                                                                                                                                                               
 int kPOLL_INTERVAL_SECS = 5;
 dispatch_source_set_timer(timer, now, [kPOLL_INTERVAL_SECS * NSEC_PER_SEC, 5000ull);
 dispatch_source_set_event_handler(timer, ^{
     // fetch data and update
     });
 

Above timer runs every 5 seconds and polls for new data and download updates for the user. In my applications, I activated the timer when the application comes in foreground (applicationDidBecomeActive method) and disable it when it goes in background (applicationWillResignActive method). Note that this syntax is much simpler compared to prior syntax of NSTimer and NSRunLoop.

Multi-tasking

Apple added support for multi-tasking in iOS SDK 4.0, which means an application can now be in one of these states

Active

When an application is in foreground and interacting with the user.

InActive

When an application is on its way to background and stops receiving the events.

Background

When an application is running in background.

Suspended

When application is still in memory but not executing.

Callback Methods

The iOS SDK added a number of callback methods for the state transfer such as:

 - (void)applicationWillResignActive:(UIApplication *)application {
     /*
      Sent when the application is about to move from active to inactive state. This can occur for certain types of temporary interruptions (such as an incoming phone
  call or SMS message) or when the user quits the application and it begins the transition to the background state.
      Use this method to pause ongoing tasks, disable timers, and throttle down OpenGL ES frame rates. Games should use this method to pause the game.
      */
 }
 
 - (void)applicationDidEnterBackground:(UIApplication *)application {
     /*
      Use this method to release shared resources, save user data, invalidate timers, and store enough application state information to restore your application to it
 s current state in case it is terminated later. 
      If your application supports background execution, called instead of applicationWillTerminate: when the user quits.
      */
 }
 
 - (void)applicationWillEnterForeground:(UIApplication *)application {
 }
 - (void)applicationDidBecomeActive:(UIApplication *)application {
 }
 

On the older devices without multi-tasking support, applicationWillTerminate is called after applicationDidEnterBackground.

Multi-Threading

The iOS SDK provides multiple ways to add concurrency support to your application.

NSThread

You can create NSThread as follows

 NSThread* myThread = [[NSThread alloc] initWithTarget:self
                                         selector:@selector(myThreadMainMethod:)
                                         object:nil];
 [myThread start];  // Actually create the thread
 

However, there is an easier way to create a thread by invoking a custom method on an object as follows:

 [NSThread detachNewThreadSelector:@selector(myThreadMainMethod:) toTarget:self withObject:nil];
 

Alternatively, you can start a background task as follows:

 [myObj performSelectorInBackground:@selector(doSomething) withObject:nil];
 

You will then define your method for doing the work as follows:

 
 - (void)myThreadMainRoutine
 {
     NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init]; // Top-level pool
  
     // Do thread work here.
  
     [pool release];  // Release the objects in the pool.
 }
 

Note that you will have to create an auto-release pool before doing the actual work. Though, these APIs are simple to use but managing threads manually is harder and you have to synchronize shared data, cleanup upon termination or when the app goes in background. See Apple's documentation for more details.

NSOperationQueue

NSOperationQueue is much easier way to manage concurrency as you don't have to create threads manually. For example, you can create an operation to save data asynchronously as follows:

 NSInvocationOperation *operation = [[NSInvocationOperation alloc] initWithTarget:self selector:@selector(saveData:) object:data];
 NSOperationQueue *queue = [[NSOperationQueue alloc] init];
 [queue addOperation:operation];
 

Instead of using NSInvocationOperation, you can define an operation class as follows:

 @interface SaveOperation : NSOperation {
     NSData *data;
 }
  
 @property(retain) NSData *data;
  
 - (id)initWithData:(NSData*)data;
  
 @end
 
 @implementation SaveOperation
  
 @synthesize data;
  
 - (id)initWithData:(NSData*)theData;
 {
     if ([super init]) {
         self.data = theData;
     }
     return self;
 }
  
 - (void)dealloc {
     [data release], data = nil;
     [super dealloc];
 }
  
 - (void)main {
     // saving
 }
  
 @end
 

You will then add the operation as follows:

 SaveOperation *op = [[SaveOperation alloc] initWithDataL:data];
 [queue addOperation:op];
 [op release];
 

You can also control the number of concurrent operations via:

 [queue setMaxConcurrentOperationCount:2];
 

Grand Central Dispatch

By far, the easiest way to add concurrency is to use Grand Central Dispatch (GCD) syntax and libraries. The GCD allows you to define tasks via function or blocks and schedules them to the available processors and automatically uses threads behind the scene. Though, it's similar to NSOperation, but the syntax is very terse and the performance is much better than using threads directly.

 dispatch_async(dispatch_get_global_queue(0, 0), ^{
     // save data
     NSLog(@"finished saving data");
  });
 

You can also nested blocks such as

 dispatch_async(dispatch_get_global_queue(0, 0), ^{
     // save data
     NSLog(@"finished saving data");
     dispatch_async(dispatch_get_main_queue(), ^{
         [self.tableView reloadData]
     });
     
  });
 

In addition to using predefined queues, you can also create your own queues, which execute the tasks in serial order such as:

 dispatch_queue_t myQueue = dispatch_queue_create("com.plexobject.myqueue", NULL);
 dispatch_async(myQueue, ^{
     // save data
     //  ...
     // now release the queue (optionally)
     dispatch_release(myQueue);
  });
 

Note that if we need to delete the queue, we must do it after all tasks are finished executed. See Apple's documenation for more details on queues.

Final word on Multi-threading

When using multi-threading explicitly or implicitly, you have to synchronize shared data. The IOS SDK provides several constructs such as @synchronized keyword, mutexes, locks, etc. to protect shared data. Nevertheless, this can be error-prone and can slowdown your application, so eliminate shared data if possible. Also, pay special attentions to exceptions within the thread method and cleanup threads upon app termination or when it becomes inactive. See Apple's documentation for more details. Also, any operations that require UI interaction must be run via main thread, though there are some exceptions such as some QuickTime APIs can be run in background.

Event Notifications

The iOS supports notifications for intercommunication between classes and threads, which promote loosely coupled design. In my applications, I extended notification by creating my own events such as:

 @interface MyEvent : NSObject {
     NSString *type;
 }
 
 @property(nonatomic, copy) NSString *type;
 -(void)fireEvent;
 @end
 
 #import                                                                                                                                                                                                                                                  
 
 #import "MyEvent.h"
 
 
 @implementation MyEvent
 
 @synthesize type;
 
 -(void)fireEvent {
     dispatch_async(dispatch_get_main_queue(), ^{
         [[NSNotificationCenter defaultCenter] postNotificationName:type object:self];
     });
 }
 

@end

Note that the event is being fired on the main thread so that UI controllers can respond to the event and update UI without any changes. The controller class would register for the event (typically in viewDidLoad) as follows:

 [[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(handleMyevent:) name:@"MyEvent" object:nil];
 

Similarly, controller would unregister for the event in viewDidUnload as follows:

 [[NSNotificationCenter defaultCenter] removeObserver:self name:@"MyEvent" object:nil];
 

The controller would then define the method handler for the event as follows:

 - (void) handleMyevent:(NSNotification *)notif {
     MyEvent *event = (MyEvent *) [notif object];
 }
 

You would then generate the event as follows:

 MyEvent *event = new MyEvent();
 event.type = @"MyEvent"
 event.fireEvent;
 

sending bug reports Automatically

Though, you can download crash reports from the Apple website, I added automatic reporting mechanism for bug reports and crashes. I used UncaughtExceptionHandler class to handle unexpected exceptions and automatically send bug reports. I don’t try to recover from unexpected exceptions as often the memory corruption leave the application in such unstable condition that it’s better to restart the application.

 
 - (void)validateAndSaveCriticalApplicationData:(NSException *)exception {
     NSMutableString *message = [[[NSMutableString alloc] initWithCapacity:200] autorelease];
     [message appendFormat:@"{name:'%@', ", exception.name];
     [message appendFormat:@"reason:'%@', ", exception.reason];
     [message appendFormat:@"userInfo:'%@', callStack:[", exception.userInfo];
     int count=0;
     for (NSObject *obj in exception.callStackReturnAddresses) {
         if (count > 0) {
             [message appendString:@","];
         }   
         [message appendFormat:@"callStack_%d:'%@'", obj];            
     }   
     [message appendString:@"], callSymbols:["];
     count=0;
     for (NSObject *obj in exception.callStackSymbols) {
         if (count > 0) {
             [message appendString:@","];
         }   
         [message appendFormat:@"callSymbol_%d:'%@'", obj];            
     }   
     [message appendString:@"]}"];
 
     [self sendBug:@"Crash report" withMessage:message];
 }
 

Persistence

The iOS SDK provides several options for persistence such as user preferences, file I/O, serialization, sqlite and CoreData. I used file based serialization for caching resources and objects that implemented NSCoding protocol and used CoreData for more complexed data. I will skip file based serialization as it’s fairly straightforward, but would list some tips on CoreData.

  • Setup versions for your data mapping files so that you can easily upgrade schema. You can add the version by selecting xcdatamodel file, choosing Design and then selecting “Add Model Version” from “Data Model” submenu. In order to auto-upgrade migration, define persistentStoreCoordinator accessor as
     
     - (NSPersistentStoreCoordinator *)persistentStoreCoordinator {
         if (persistentStoreCoordinator != nil) {
             return persistentStoreCoordinator;
         }   
         
         NSString *docs = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) lastObject]; 
         NSURL *storeUrl = [NSURL fileURLWithPath: [docs stringByAppendingPathComponent: @"MyData"]];
         
         NSError *error = nil;    
         NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:                                                                                                                                                                                                       
                                  [NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption,    
                                  [NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, nil];
         
     
         persistentStoreCoordinator = [[NSPersistentStoreCoordinator alloc] initWithManagedObjectModel:[self managedObjectModel]];
         if (![persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:storeUrl options:options error:&error]) {
             NSLog(@"Unresolved error %@, %@", error, [error userInfo]);
             abort();
         }    
         return persistentStoreCoordinator;
     }
     
  • Use a single NSManagedObjectContext per thread or queue. For example, I used different NSManagedObjectContext for background timer that updated the database than the NSManagedObjectContext that displayed the table using NSFetchedResultsController.
  • Merge changes when using multiple NSManagedObjectContext objects, e.g.
     
     NSNotificationCenter *dnc = [NSNotificationCenter defaultCenter];
     
     
     [dnc addObserverForName:NSManagedObjectContextDidSaveNotification object:insertManagedObjectContext
                               queue:nil usingBlock:^(NSNotification *saveNotification) {
                                   [[self readManagedObjectContext] mergeChangesFromContextDidSaveNotification:saveNotification];
         }];
     
     // update here
     
     [dnc removeObserver:self name:NSManagedObjectContextDidSaveNotification object:ctx];
     
     
  • In found problems with caching NSManagedObjects when using multiple NSManagedObjectContexts as iOS invalidate existing references when new updates come and you get wonderful error:
    The NSManagedObject with ID:#### has been invalidated.

I ended up creating transient copy of the object for caching purpose. Another factor you will have to consider is what information should be kept in memory and what information can be stored in the database. In general, if the data is small or frequently updated such as stock quote, I kept it in memory, however if the data is large and less often updated such as stock positions, I kept it in the database. In addition, I used the database as interface between UI and network, so the data would get updated asynchronously in background and stored in the database. The network code would then fire an event, which the UI subscribes and would then update the view from the database. This provided fast and efficient way to view large amount of data without waiting.

Performance

Performance is a key part of usage experience. Apple provides a number of tools such as Shark and Instruments to measure performance bottlenecks as well as investigate memory leaks and CPU activity, file I/O, database and other issues. You need to start measuring the performance on real devices early in the application lifecycle so that you can design the application properly.

Memory Management

As, Apple does not support garbage collection on iOS devices, the memory management is one of most painful aspects of the development. I felt that pain as I have been using language such as Java, Ruby, Erlang and Python for past many years that come with garbage collection. Here are some of the tips for memory management:

  • Use properties and synthesize keywords to automatically generate proper accessors/setters.
  • When assigning properties within the class use the syntax “self.property = aproperty” instead of “property = aproperty” so that you invoke your setters properly.
  • Use retain and release instead of dealloc method.
  • Use retain for all object references.
  • Use assign for primitive references.
  • Use copy for String types, GCD blocks and other instances where you want clone of the objects.
  • Use assign when you are using delegate pattern to hold another controller for callback so that you don’t end up with cyclic references.
  • Use AutoRelease Pools for separate threads.
     
     NSAutoreleasePool *pool = [ [ NSAutoreleasePool alloc ] init ];
      // ...
     [pool drain];
     
  • Use auto-release objects sparsely, for example if you only need an object in a method or a block and then just allocate it at the beginning and release it at the end (possibly in finally block of try-catch).
  • If a method name begins with alloc, copy, or new, then you must return allocated instance and caller would own the reference, otherwise you must return an auto-released object.

Use Memory-Leak detection feature of Instruments to find the objects that are not properly released. You can set NSZombieEnabled flag to true on your executable, which can help track the memory leaks when the app crashes. Finally, multi-threading further complicates the memory management. For example, you may pass auto-released object to another thread and by the time that the thread uses it, the first thread’s auto-release pool released it. So, you must retain the object in separate thread before processing it.

Testing

Though, iPhone Simulator is fast and easy way to test the application during the development, but there are significant difference between the iPhone Simulator and real devices. First, many of the APIs such as Location, Apple Store, Accelerator, Email are not supported on the Simulator, but most importantly the Simulator runs on blazing fast MacPro machine, whereas real devices are slow and limited. I highly recommend start your testing with real devices early in your app lifecycle. There is also a big difference in hardware between different generation of iPod touch, iPhone and iPad. For example, iPhone 3G has 620Mhz processor with 128M memory, iPhone 3GS has 833Mhz with 256M memory and latest iPhone 4 has 1Ghz with 512M memory. I suggest using older devices to really find performance and stability issues with the application. Also, you want it to run on most devices so you should target lowest device that you can, which you can configure it by setting target OS property of the build.

Though, Apple provides basic Unit Test support, but I found it very painful to use. We have used the UI Automation tool for functional testing with some success, but it lacks good documentation.

Conclusion

In this blog, I listed several techniques for designing and developing iOS applications, though most of the techniques can be easily applied to other mobile platforms. Though, hardware on mobile devices is becoming more powerful, but it is still constrained with limited memory. You have to pay an exceptional attention to the memory usage, performance and stability of the app and users are going to hate if it’s slow or crashes. The network apps have to further address the availability and speed of the network so that apps is not completely unusable in airport mode and with slow connection. In the end, testing on real devices and real environment is the key to create effective network based applications.


September 22, 2010

An implementation of Virtual Node Router with Consistent Hash algorithm

Filed under: Java — admin @ 1:02 pm

Since the Dynamo paper, published a few years ago, DHTs and consistent hash have become mainstream. Here is my implementation of a virtual node router that uses consistent hash algorithm for splitting requests to the virtual nodes:

 import java.nio.ByteBuffer;
 import java.security.MessageDigest;
 import java.util.Comparator;
 import java.util.Iterator;
 import java.util.Map;
 import java.util.SortedMap;
 import java.util.TreeMap;
 
 public class VirtualNodeRouter {
     interface HashCalculator {
         long calculateHash(String key);
     }
 
     private static class VirtualNode {
         final String nodeName;
         final int replicaNumber;
 
         VirtualNode(final String nodeName, final int replicaNumber) {
             this.nodeName = nodeName.toLowerCase();
             this.replicaNumber = replicaNumber;
         }
 
         boolean matches(String host) {
             return nodeName.equalsIgnoreCase(host);
         }
 
         @Override
         public String toString() {
             return nodeName + ":" + replicaNumber;
         }
     }
 
     private final HashCalculator hashFunction;
     private final SortedMap<Long, VirtualNode> virtualNodePoolByHash = new TreeMap<Long, VirtualNode>(
             new Comparator<Long>() {
                 public int compare(Long i, Long j) {
                     if (i > j) {
                         return 1;
                     } else if (i < j) {
                         return -1;
                     } else {
                         return 0;
                     }
                 }
             });
 
     public VirtualNodeRouter() {
         this(new HashCalculator() {
             public long calculateHash(String key) {
                 try {
                     MessageDigest sha1 = MessageDigest.getInstance("SHA1");
                     sha1.update(key.getBytes());
                     byte[] digest = sha1.digest();
                     return bytesToLong(digest);
                 } catch (Exception e) {
                     throw new RuntimeException(e);
                 }
             }
         });
     }
 
     public VirtualNodeRouter(final HashCalculator f) {
         this.hashFunction = f;
     }
 
     /**
      * Adds a node with one replica
      * 
      * @param node
      *            - node name
      */
     public void add(String node) {
         add(node, 1);
     }
 
     /**
      * Adds a node to the available pool
      * 
      * @param node
      *            - node name
      * @param replicas
      *            - # of replicas - increase # of replicas based on the
      *            computing power of the machine
      */
     public void add(String node, int replicas) {
         // Note: You can call this method incrementally by adding more replicas,
         // so that you don't cause DOS on
         // your own services
         int existingReplicas = getReplicas(node);
 
         for (int i = 0; i < replicas; i++) {
             VirtualNode virtualNode = new VirtualNode(node, i
                     + existingReplicas);
             virtualNodePoolByHash.put(hashFunction.calculateHash(virtualNode
                     .toString()), virtualNode);
         }
     }
 
     /**
      * remove the node from available pool
      * 
      * @param node
      */
     public void remove(String node) {
         Iterator<Long> it = virtualNodePoolByHash.keySet().iterator();
         while (it.hasNext()) {
             Long key = it.next();
             VirtualNode virtualNode = virtualNodePoolByHash.get(key);
             if (virtualNode.matches(node)) {
                 it.remove();
             }
         }
     }
 
     public String getNode(String key) {
         if (virtualNodePoolByHash.isEmpty()) {
             return null;
         }
         long hash = hashFunction.calculateHash(key);
         for (Map.Entry<Long, VirtualNode> e : virtualNodePoolByHash.entrySet()) {
             if (hash < e.getKey()) {
                 return e.getValue().nodeName;
             }
         }
         SortedMap<Long, VirtualNode> tailMap = virtualNodePoolByHash
                 .tailMap(hash);
         hash = tailMap.isEmpty() ? virtualNodePoolByHash.firstKey() : tailMap
                 .firstKey();
         return virtualNodePoolByHash.get(hash).nodeName;
     }
 
     public void dump() {
         for (Map.Entry<Long, VirtualNode> e : virtualNodePoolByHash.entrySet()) {
             System.out.println("  " + e.getKey() + " => " + e.getValue());
         }
     }
 
     public int getReplicas(String nodeName) {
         int replicas = 0;
         for (VirtualNode node : virtualNodePoolByHash.values()) {
             if (node.matches(nodeName)) {
                 replicas++;
             }
         }
         return replicas;
     }
 
     private static long bytesToLong(byte[] b) {
         ByteBuffer bb = ByteBuffer.wrap(b);
         return bb.getLong();
     }
 }
 

The virtual nodes are added by specifying the node name and the number of replicas. You can add more virtual nodes for the more powerful machines than the low-end machines. You can call remove method when the node goes down or is unavailable. The getNode is called when a request needs to be routed. For example, if you are caching results of a service, you can use the key of cache to find the virtual node and then store the cache value at that node. You can read following resources to learn more about the consistent hash algorithm:


September 16, 2010

My impression of Diaspora codebase

Filed under: Computing — admin @ 5:07 pm

I briefly reviewed the Diaspora source code, which is all written in Ruby on Rails 3.0. Here are my initial thoughts on the codebase:

What I liked:

  • It’s all open source — YES!
  • Diaspora uses latest Rails 3.0 APIs such as global respond_to in controllers, thin controllers, and fat models with new query APIs. The code uses RSpec for tests and Factory-Girl for creating test objects instead of fixtures, which are much easily managed. There are a few Selenium tests, but most of the tests are in RSpec.
  • Diaspora is built using latest technologies and standards such as PKI (OpenSSL), HTML5, VCard, Websockets, Microformat, XRDS, PubSubHubbub along with popular libraries such as EventMachine, HAML, jQuery, Fancybox, Sprinkle, Bundler, Blueprint, etc.
  • Deployment scripts and documentation – Though, there isn’t any documentation on overall architecture or comments in the code, but I found installation documentation very helpful. Also, the deployment rake tasks and configurations are all included in the source code.

What I disliked:

Though, I found the code to be fairly easy to read and consistent in style I found following problems related to the performance, scalability and modularity.

  • Service API – Diaspora uses Rails controller for serving HTML as well as JSON, XML and XRDS requests but I would have preferred separate services for the API with much more defined contract.
  • Pagination – I found sporadic use of pagination in the code but a number of classes use Rails’ builtin relationships without any pagination. It’s like when you ask for banana you get the gorilla holding the banana. In my experience, this has been problem with all O/R mapping tools, which give you nice syntax for fetching related objects until your server runs out of memory or your database dies on you.
  • Before/after filters – I found a number of such filters in the code, which I have found to be another common issue with the scalability when before/after filters require a lot of overhead.
  • Asynchronous Messaging – I didn’t see any use of asynchronous messaging as a lot of requests such as adding/removing a friend can be done asynchronous.
  • Modularity – Diaspora code uses modules and models for the business logic, but I found a couple of models such as user to be too big, which can be divided into suitable modules.
  • MongoDB – I have nothing against MongoDB but I found a lot of code depends on MongoDB. I would have preferred using data access service instead, which completely encapsulates underlying storage technology and allows you to replace it without modifying all the code.

Conclusion

Despite the Knuth’s advice: “Forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil,” you need the architecture for scalability and performance when you are building the platform to replace Facebook. As social networking tools depend much more on Network effects or Metcalfe’s law instead of best technology, I hope early release of the software allows it to capture more users who use it. I was somewhat disappointed that identi.ca has not caught much attraction as open source alternative to Twitter. And I hope that Diaspora succeeds in becoming a good alternative to Facebook.

August 12, 2010

Implementing a Single Sign-on solution for WordPress (PHP) and Rails applications using Central Authentication Service (CAS)

Filed under: SSO — admin @ 2:36 pm

I am starting a new Rails project that would be using WordPress for CMS functionality and I needed a way to share the user sessions between WordPress and Rails. Though, I found a couple of WordPress plugins, but I decided to use Central Authentication Service (CAS). I found two server implementations, where the first one was in Java and second one was in Ruby. I had some experience with the Java based CAS server back in 2005, so I decided to use it. Here are the key installation and configuration steps for setting up the Single-Sign-On solution:

Create Rails Project

I had already installed Rails 2.3, so I created a simple project using:

 rails ssoapp
 

Install WordPress

I downloaded latest 3.0 version of WordPress and unzipped it under ssoapp/public directory.

Configuring PHP for Apache

I decided to use Apache for hosting WordPress and Rails applications. As I was using Mac, it already had Apache installed on my machine. All, I had to do was to uncomment following line from /etc/apache2/httpd.conf file:

 LoadModule php5_module        libexec/apache2/libphp5.so
 

I then restarted Apache using

 sudo /usr/sbin/apachectl restart
 

I created a database for wordpress and defined a username/password as follows:

 create database wp
 CREATE USER 'wp'@'localhost' IDENTIFIED BY 'wp';
 GRANT ALL ON *.* TO 'wp'@'localhost';
 

I specified database settings in wp-config.php as follows:

 define('DB_NAME', 'wp');
 
 define('DB_USER', 'wp');
 
 define('DB_PASSWORD', 'wp');
 
 define('DB_HOST', 'localhost');
 

and created an admin account by pointing my browser to http://localhost/wp/wp-admin/install.php.

Installing Phusion Passenger on Apache

I installed passenger gem as follows:

 gem install passenger
 

And installed passenger module for Apache using:

 passenger-install-apache2-module
 

I then added following lines to /etc/apache2/httpd.conf file:

 LoadModule passenger_module /Users/sbhatti/.rvm/gems/ruby-1.9.1-p378/gems/passenger-2.2.15/ext/apache2/mod_passenger.so                                                                                                                                                                                          
 PassengerRoot /Users/sbhatti/.rvm/gems/ruby-1.9.1-p378/gems/passenger-2.2.15
 PassengerRuby /Users/sbhatti/.rvm/rubies/ruby-1.9.1-p378/bin/ruby
 

I created a file called ssoapp.conf under /etc/apache2/users with following contents:

 <VirtualHost localhost:80>
       ServerName localhost
       DocumentRoot /Users/sbhatti/src/yoga-rails/public
       RailsEnv development
       <Directory /Users/sbhatti/src/yoga-rails/public>
          Allow from all
          Options Indexes -MultiViews
       </Directory>
       <Location /wp>
          PassengerEnabled off 
          AllowOverride all 
       </Location>
       <Location /cas>
          PassengerEnabled off 
          AllowOverride all 
       </Location>
 </VirtualHost>
 

Configuring SSL for Apache

I needed SSL for some functionality so I created self-signed keys and certificates a follows:

 openssl genrsa -des3 -out server.key 1024
 openssl req -new -key server.key -out server.csr 
 

and copied those files to /etc/apache2 directory. I then modified /etc/apache2/httpd.conf by uncommenting following line:

 LoadModule ssl_module libexec/apache2/mod_ssl.so
 

and adding

 Listen 443
 

I then modified /etc/apache2/users/ssoapp.conf and added following contents:

 <IfModule mod_ssl.c>
   AddType application/x-x509-ca-cert .crt
   AddType application/x-pkcs7-crl .crl
 
   SSLProtocol all -SSLv2
   SSLPassPhraseDialog builtin
   SSLSessionCache dbm:/var/run/ssl_scache
   SSLSessionCacheTimeout 300
   SSLMutex file:/var/run/ssl_mutex
   SSLRandomSeed startup builtin
 <VirtualHost localhost:443>
       ServerName localhost
       DocumentRoot /Users/sbhatti/src/yoga-rails/public
     ServerAdmin sbhatti@peak6.com
     ErrorLog /tmp/error_log
     TransferLog /tmp/access_log
     SSLEngine on
     SSLProtocol all -SSLv2
     SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
     SSLCertificateFile /etc/apache2/server.crt
     SSLCertificateKeyFile /etc/apache2/server.key
     <Files ~ "\.(cgi|shtml|phtml|php3?)$">
       SSLOptions +StdEnvVars
     </Files>
       <Directory /Users/sbhatti/src/yoga-rails/public>
          Allow from all
          Options Indexes -MultiViews
         SSLOptions +StdEnvVars
       </Directory>
       <Location /wp>
          PassengerEnabled off
          AllowOverride all
       </Location>                                                                                                                                                                                                                                                                                                          
       <Location /cas>
          PassengerEnabled off 
          AllowOverride all 
       </Location>
     SetEnvIf User-Agent ".*MSIE.*" \
     nokeepalive ssl-unclean-shutdown \
     downgrade-1.0 force-response-1.0
     CustomLog /var/log/httpd/ssl_request_log \
     "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
 </VirtualHost>
 </IfModule>
 

Basically, above configuration allows all HTTP and HTTPS requests be sent to my SSO rails app except /wp requests, which are sent to the WordPress.

Password Encoding

I decided to use WordPress users table (wp_users) as primary users table. WordPress by default uses PHPPass library for encrypting passwords. The latest WordPress uses combination of salt, MD5 and iterations for encrypting, but I wanted to use simple MD5 for now, so I edited ./wp-includes/pluggable.php and replaced default hash encoding to MD5 as follows:

 function wp_hash_password($password) {
    return md5($password, FALSE);
 }
 

I then reset the password in Mysql database as follows:

 update wp_users set user_pass = md5('hello') where user_login = 'admin';
 

Setting CAS Server

I downloaded and unzipped CAS Server. I needed to configure CAS server to use the wp_users table for authenticating users, so I added following dependencies to the cas-server-webapp/pom.xml:

         <dependency>
                 <groupId>mysql</groupId>
                 <artifactId>mysql-connector-java</artifactId>
                 <version>5.0.5</version>
         </dependency>
         <dependency>
                 <groupId>commons-dbcp</groupId>
                 <artifactId>commons-dbcp</artifactId>
                 <version>1.4</version>
         </dependency>
         <dependency>
                 <groupId>${project.groupId}</groupId>
                 <artifactId>cas-server-support-jdbc
                 </artifactId>
                 <version>${project.version}</version>
         </dependency>
 

I then added following beans to the cas-server-webapp/src/main/webapp/WEB-INF/deployerConfigContext.xml:

         <bean id="md5PasswordEncoder"
                 class="org.jasig.cas.authentication.handler.DefaultPasswordEncoder">
                 <constructor-arg index="0" value="MD5" />
         </bean>
         <bean id="wpDataSource" class="org.apache.commons.dbcp.BasicDataSource">
                 <property name="driverClassName">
                         <value>com.mysql.jdbc.Driver</value>
                 </property>
                 <property name="url">
                         <value>jdbc:mysql://localhost:3306/wp
                         </value>
                 </property>
                 <property name="username">
                         <value>wp</value>
                 </property>
                 <property name="password">
                         <value>wp</value>
                 </property>
         </bean>
 

and removed default authentication class org.jasig.cas.authentication.handler.support.SimpleTestUsernamePasswordAuthenticationHandler and added following bean instead:

         <property name="authenticationHandlers">
                 <list>
         ...
                         <bean
                                 class="org.jasig.cas.adaptors.jdbc.SearchModeSearchDatabaseAuthenticationHandler">
                                 <property name="tableUsers">
                                         <value>wp_users</value>
                                 </property>
                                 <property name="fieldUser">
                                         <value>user_login</value>
                                 </property>
                                 <property name="fieldPassword">
                                         <value>user_pass</value>
                                 </property>
                                 <property name="dataSource" ref="wpDataSource" />
                                 <property name="passwordEncoder" ref="md5PasswordEncoder" />
                         </bean>
 
                 </list>
         </property>
 

Above bean configuration specifies MD5 for hashing passwords.

Creating WAR file for CAS-Server

I then created the CAS war file by typing:

 mvn clean install
 

Install and Configure Tomcat

The CAS-server requires Java application server so I downloaded and installed latest Apache Tomcat. Next, I needed to setup SSL for secure login so I created another set of certificate and private key using keytool as follows:

 keytool -genkey -alias ssoapp -keypass ssoapp -keystore ssoapp.ks -storepass ssoapp
 

I copied ssoapp.ks to the conf directory under Apache tomcat and modified conf/server.xml and added following contents:

         <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
                 maxThreads="150" scheme="https" secure="true" clientAuth="false"
                 sslProtocol="TLS" keystoreFile="${catalina.home}/conf/ssoapp.bin"
                 keystorePass="ssoapp" />
 

I then started the Apache Tomcat using:

 bin/startup.sh 
 

Configuring CAS-client for WordPress

First, I downloaded phpCAS client library and unzipped it under public directory of my Rails app. I used wpCas plugin for WordPress, which uses phpCAS library for connecting to the CAS server. I installed the plugin under wp-contents/plugins directory and logged in to WordPress as admin. I clicked the plugins link from left navigation and then activated the plugin. After activating it, I clicked the settings link from the left navigation and configured the CAS as follows:

Now, the moment of truth. I opened another instance of browser and pointed to http://localhost/wp, which showed my blog.

I then clicked on “Log in” link, which showed me certificate warning due to self-signed certificate. After accepting it, it took me to the CAS login page, e.g.

Voila, after typing my username and password, it took me back to the WordPress.

Setting up CAS-Client for Rails

I used rubycas-client plugin for Rails and installed it as follows:

   ./script/plugin install git://github.com/gunark/rubycas-client.git
 

I then modified config/environment.rb file and added:

   require 'casclient'
   require 'casclient/frameworks/rails/filter'                                                                                                                                                                                                                                                                              
   CASClient::Frameworks::Rails::Filter.configure(
     :cas_base_url => "https://localhost:8443/cas"
   )
 

One gotcha in above configuration is that you have to place it at the end of configuration file as you get load error if you place it inside initializer block.
I then added before_filter CASClient::Frameworks::Rails::Filter to all controllers that needed authentication and added CASClient::Frameworks::Rails::Filter.logout(self) where I needed to logout. After changing my controller, I went back to the Rails app and sure enough it worked like a charm!

Summary

Though, there were many small pieces to get this working, but I am happy with this solution. This would help create seamless experience to the users, though the other half of the experience depends on unified user interface.

References


August 10, 2010

NoSql databases bring “Stored Procedures” back in fashion

Filed under: Business — admin @ 5:26 pm

One of best tip I learned from my post-graduate research in Parallel & Distributed area was to bring the computation closer to the data. However, most applications in the real world are designed as three or more tiers that separate databases from the application server, where the business logic resides. Though, stored procedures have long been used in client server architecture, dataware services, reporting, and other forms to run the business logic closer to the database, but they are generally shunned due to the maintenance issues. I find it interesting that NoSQL databases are bringing back the stored procedures in the form of map/reduce queries. NoSQL databases come in various forms such as key-value stores, document stores, column stores, and graph stores. They are primarily influenced by the Brewer’s CAP Theorem and use BASE (basically available, soft state, eventually consistent) transactions as opposed to ACID (atomicity, consistency, isolation, durability) transactions. NoSQL databases are designed for horizontal scalability and are able to support large data by partitioning it. NoSQL offer rich queries based on map/reduce, which are generally written in javascript or other scripting languages. These queries provide powerful mechanism to define the business logic for filtering or aggregating results, which are then executed inside the database or closer to the data. Thus, NoSQL databases are able to provide much better performance as a side effect if the application logic is transferred to the host where the data resides. Everything old is new again and stored procedures are back in the fashion.

References:

August 5, 2010

Implementing Apple Remote Push Notification using Objective-C and Ruby

Filed under: iPhone development — admin @ 3:44 pm

Recently, I added the remote push capabilities to an iPhone/iPad application at work. It turned out to be a bit more involved process than I expected so I would document key steps of the process.

Overview of Remote Push Notification

The Apple Push Notification works like SMS, where the application providers can send short messages to the users (upto 256 bytes). The message consists of JSON structure as follows:

 { 
   "aps" : {  "alert" : "your-message", "sound" : "filename" : "badge" : 5,     },     
   // optionally more structures, e.g. {"acme1" : "bar"}
 }
 

The alert key defines the contents of the message, which is displayed to the user. You can play sound by specifying the sound filename (which is already bundled with the app), however you can leave it blank or use ‘default’ for playing default sound. Finally, the badge will allow notification to show a number next to your application icon. Also, you can optionally add action buttons that can take user to a particular screen in your application (though I won’t show it here).

The format for sending notification is as follows:

You can read Apple documentation for more inforamtion on the binary format.

The architecture of Push Notification consists of three key pieces:

  • iOS Application – that enables push notification and registers the device for notification
  • Application Server – is responsible for generating the messages and publishing them to the Apple Push Notification Service (APNS)
  • Apple Push Notification Service (APNS) – is responsible for delivering the messages to the devices where the application is installed

Here is the architecture diagram from Apple documentation on Push Notification:

Enabling Push Notification for the application

I assume you have already created an application with developer certificate, app-id and provisioning profile. However, Apple Push Notification requires another set of certificates for development and production. So, login to the iPhone developer site and select iPhone Provisioning Portal from the upper right side and then choose App IDs link.

Select “Enable for Apple Push Notification service” checkbox, then click on Configure link:

It will guide you through creating certificate, where you will have to create a certificate request by opening “Keychain Access”, then selecting “Certificate Assistant” option from the menu and choosing “Request a Certificate from Certificate Authority”. You can then upload the generated request file to the portal. Once the certificate is generated you should see the option to download the certificate, e.g.

After downloading the certificate, you can double click or drag it to the “Keychain Access” and it will add the certificate to the Keychain.

Publishing Push Notifications on the Application Server

The push notification requires a server that connects to the APNS service and sends notification messages. You will need to use same set of keys and certificates that you created on the Apple development portal. So open “Keychain Access” and select Keys from the left options, then select the private key for the push notification. You can then export the key by right clicking, which will create a file with .12 file extension. Note that it will ask you for a password for encrypting the key (type in anything as we will remove it later). Next, select “My Certificates” from the left options and select the certificate for push notification. Then repeat the process of exporting by right clicking and saving the certificate. You will see two files with .12 extensions, which can be converted into .pem files as follows:

 openssl pkcs12 -clcerts -nokeys -out apns_cert.pem -in 
 openssl pkcs12 -nocerts -out apns_key.pem -in 
 

You can remove password with the the following command:

 openssl rsa -in apns_key.pem -out apns_key_unenc.pem 
 

Then merge two files with following command:

 cat apns_cert.pem apns_key_unenc.pem > apns.pem
 

I used Rails on the server side, so I defined a few models and controllers to store devices and messages, where the device model had an attribute ‘token’ and the message model had three attributes: alert, badge and sound. Each device also has an attribute deactivated_at, which is set to the date when the device is disabled. Next, I created a short Ruby library for sending notifications. The library provides two methods send_message for sending notifications and get_feedbacks for checking disabled devices so that you can remove them from your database. The send_message method accepts a message model as an argument and then opens a connection to the APNS service. Apple recommends using a single connection for publishing notifications to all devices, so it adds all devices within a single a connection.

 require 'socket'                                                                                                                                                                                                                                                                                                           
 require 'openssl'
 
 class ApnsPublisher
   def self.send_message(apns_message)
     json_payload =  {"aps" => {"alert" => apns_message.alert, "sound" => apns_message.sound, "badge" => apns_message.badge}}.to_json.to_s
 
     open_connection(APNS_SERVER, 2195) do |conn, sock|
       ApnsDevice.find_in_batches(:batch_size => 500 ) do |devices| 
         devices.each do |device|
           next unless device.deactivated_at.nil?
           unless ApnsDevicesMessage.find_by_apns_device_id_and_apns_message_id(device.id, apns_message.id)
             token = device.token.gsub(/\s+/,'')
             byte_token = [token].pack("H*")
             message = "\0\0 #{byte_token}\0#{json_payload.length.chr}#{json_payload}"                                                                                                                                                                                                                                      
             raise "message #{message} is too big" if message.size.to_i > 256 
             conn.write(message)
             f.write(message)
             ApnsDevicesMessage.create(:apns_device_id => device.id, :apns_message_id => apns_message.id, :delivered_at => Time.new.utc)
           end                                                                                                                                                                                                                                                                                                              
         end 
       end 
     end 
   end 
 
   def self.get_feedbacks()
     open_connection(APNS_FEEDBACK_SERVER, 2196) do |conn, sock|
       while line = sock.gets
         line.strip!
         feedback = line.unpack('N1n1H140')
         token = feedback[2].scan(/.{0,8}/).join('').strip
         device = ApnsDevice.find_by_token(token)
         if device
           device.update_attribute(deactivated_at, Time.at(feedback[0])
         end 
       end 
     end 
   end 
 private
   def self.open_connection(host, port, passphrase='')
     cert = File.read(APNS_CERT_FILE)
     ctx = OpenSSL::SSL::SSLContext.new
     ctx.key = OpenSSL::PKey::RSA.new(cert, passphrase)
     ctx.cert = OpenSSL::X509::Certificate.new(cert)
   
     sock = TCPSocket.new(host, port)
     ssl = OpenSSL::SSL::SSLSocket.new(sock, ctx)
     ssl.sync = true
     ssl.connect
     yield ssl, sock 
   
     ssl.close
     sock.close
   end 
 end
 

You can use delayed_job plugin to run the get_feedbacks method periodically. Also, I added a service for registering and adding the devices, which is called from the iPhone devices (but not shown here).

Configuration

You will need to define following properties for development:

 APNS_SERVER = "gateway.sandbox.push.apple.com"
 APNS_FEEDBACK_SERVER = "feedback.sandbox.push.apple.com"
 APNS_CERT_FILE = File.join(RAILS_ROOT, 'config', 'apns_dev.pem')
 

and following properties for production:

 APNS_SERVER = "gateway.push.apple.com"
 APNS_FEEDBACK_SERVER = "feedback.push.apple.com"
 APNS_CERT_FILE = File.join(RAILS_ROOT, 'config', 'apns_prod.pem')
 

Registering devices for notification

You can register the device by adding following method in your didFinishLaunchingWithOptions of the primary delegate class:

 
         [[UIApplication sharedApplication] 
                 registerForRemoteNotificationTypes:( UIRemoteNotificationTypeAlert |UIRemoteNotificationTypeBadge | UIRemoteNotificationTypeSound)];
 
 

and then registering with following callback methods:

 #pragma mark push notifications
 - (void)application:(UIApplication *)app didRegisterForRemoteNotificationsWithDeviceToken:(NSData *)devToken {
         NSString *token = [[devToken description] stringByTrimmingCharactersInSet:[NSCharacterSet characterSetWithCharactersInString:@"<> "]];
         token = [token stringByReplacingOccurrencesOfString:@" " withString:@""];
         DeviceRegisterer *registrar = [[DeviceRegisterer alloc] init];
         [registrar registerDeviceWithToken:token];
 }
 
 
 - (void)application:(UIApplication *)app didFailToRegisterForRemoteNotificationsWithError:(NSError *)err {
         NSLog(@"failed to regiser %@", err);
 }
 
 - (void)application:(UIApplication *)application didReceiveRemoteNotification:(NSDictionary *)userInfo {
         NSLog(@"notification options %@", userInfo);
 }
 
 

I used ASIHTTPRequest library for invoking REST service I wrote for registering devices (on Rails side), e.g.

DeviceRegisterer.h

 @class ASIFormDataRequest;                                                                                                                                                                                                                                                                                                 
 
 
 @interface DeviceRegisterer : NSObject {
         ASIFormDataRequest *request;
 
 }
 
 @property (retain, nonatomic) ASIFormDataRequest *request;
 
 - (void)registerDeviceWithToken:(NSString *)token;
 
 @end
 

DeviceRegisterer.m

 
 #import "DeviceRegisterer.h"
 
 #import "ASIFormDataRequest.h"
 
 @implementation DeviceRegisterer
 @synthesize request;
 
 - (void)registerDeviceWithToken:(NSString *)token {
         if (self.request == nil) {
                 NSURL *url = [NSURL URLWithString:[NSString stringWithFormat:@"%@/admin/apns_devices.json", API_BASE_DOMAIN]];
                 [self setRequest:[ASIFormDataRequest requestWithURL:url]];
                 [request setPostValue:token forKey:@"token"];
 
                 [request setTimeOutSeconds:30];
                 [request setDelegate:self]; 
                 [request setDidFailSelector:@selector(registerFailed:)];
                 [request setDidFinishSelector:@selector(registerFinished:)];
                 
                 [request startAsynchronous];            
         }
 }
 
 
 - (void)registerFailed:(ASIHTTPRequest *)theRequest {
         NSLog(@"registerFailed %@", [theRequest error]);
 }
 
 - (void)registerFinished:(ASIHTTPRequest *)theRequest {
         NSLog(@"registerFinished %d",[theRequest postLength]);
 }
 
 - (void)dealloc {
         [request cancel];
         [request release];
         [super dealloc];
 }
 
 @end
 

Summary

In nutshell, push notification is powerful feature that can help your users engage with your application, though it must be used with caution so that users are not annoyed and in turn remove your application or disable it. It also requires a lot of moving parts for sending notification, registering, getting feedback on the devices. I found that Apple does not provide a great debugging options when testing the push notification. For example, first problem I encoutered with testing was that my profile on XCode was old and wasn’t updated after I enabled push notification. I had to delete my old profile and then refresh it from the Organizer. Also, when you send notifications to the APNS, you don’t get any response code or errors. This caused some frustration when I wasn’t getting messages due to slightly wrong JSON format. Fortunately, when your device is connected you can select the device from the XCode Organizer and view the Console tab for debugging information. I was able to view the error (which a bit vague) and then got everything working after fixing the JSON structure.


July 25, 2010

Tutorial days from OSCON 2010

Filed under: Computing — admin @ 10:21 pm

I had fun at OSCON last year, so I decided to go back this year. It’s incredible experience being part of hundreds of developers (more than 2500), who are excited about open source and upcoming technogolies. Though, there were a number of tracks, but I was mainly interested in Mobile Computing, Cloud Computing, No-SQL, and Scala. There were also interesting sessions on Emerging Languages and Hardware Hacking with Arduino, but I didn’t get chance to attend them.

Tutorials Day One

I arrived in Portalnd on Sunday night and the first session I attended on Monday morning was on Android. I saw fair number of Android phones at the conference and people were excited about Android. There was a lot of bashing of Apple, but I won’t get into it. Here are some of the highlights from the Android session:

Android for Java Developers

Android is a specially designed O/S based on Linux that uses a number of open source libraries and custom framework for building mobile applications. It even comes with a command line tool adb that opens shell directly on the device or emulator. Though, Android applications are written in Java but the byte-codes are converted into Dalvik instruction sets (.dex files). Dalvik is a register based Java VM as opposed to Sun’s stack based and uses JavaSE minus Swing/AWT APIs. You use Dalvik generated executable and resources (images, audio, configurations, etc.) to build the application file (APK), which is then signed by self-generated certificate. The best part of Android development is their deployment, which is an order of magnitude easier compare to Apple iOS.

Android SDK

Before developing, you would need to download the Eclipse IDE, Android SDK and then install Eclipse plugin from https://dl-ssl.google.com/android/eclipse/.

Creating Android Device

One of the downside of Android is variations of the capabilities of all Android devices, however you don’t need to own all variations. All you need is to create Android device by giving name, target version, SD card, skin and other hardware limitations. Android comes with emulator as opposed to simulator with iPhone SDK and follows hardware much more closely. On the downside, it takes a long time to start the Android emulator, but on the upside you can emulate call, SMS, and launch multiple emulators, which can other emulators. Note, the android device file definition goes into ~/.android/avd directory.

Hello World

You can create a new project and specify the build target, application name, package name, and main activity class. An activity class represents a controller class, which you define for each screen. Android uses UI to generate layout file (R file). If your application need resources such as images and property files, you store them in res directory. You can also create resource files for storing strings to make your application easily localized for various languages.

Activities

Activities represent screens and are managed by the activity managers. The activity can be in one of five states: starting, running, paused, stopped, and destroyed. The Activity class defines callback methods such as onCreate, onStart, onRestoreInstanceState, and onResume when the state changes.

Intent

Intent represents events or actions, which can be explicit or implicit.

Services

One of the key advantage of Android has been its support of multi-tasking and you can have a background processing using services. Services also have lifecycle, but simpler than activities.

Content Providers

Content providers allow sharing data between applications such as contacts, mediastore, settings, etc.

Broadcast Receivers

These allow pub-sub based mechanism for system events such as SMS messages.

Architecture of Android applications

Here are some of the tips that Marko Gargenta gave when designing application:

  • Isolate I/O operations such as network or disk operations into separate tasks or background services, which either use notification or database to communicate with interactive application. For example, in his sample twitter application, he used background service to poll tweets and stored them to the database. The activities then polled tweets from the database and also subscribed to the notification when new tweet arrives.
  • Use layout for screen design as it is more declarative and separate text values from the layout and use string resources.
  • Due to the variations of the Android devices, use layout weight and density intensity pixel (dp or sp) instead of fixed pixels (px) for components.
  • Android SDK provides GUI tool for designing layout and you will need to bind the UI components back to the activity classes, so use consistent naming convention for both layout file and the activity file, e.g.
       public class Twitter extends Activity {
         EditText editTweet;
         Button buttonUpdate;
         public void onCreate(...) {
           editTweet = (EditText) findViewById(R.id.editTweet);
           editButton = (Button) findViewById(R.id.editButton);
           buttonUpdate.setOnClickListener(this)
             String tweet = editTweet.getText().toString();
           }
         }
       }
     

Adapters

Android allows easily access large datasets as arrays that can be displayed on the screen.

Logging

Android uses custom libc and Java for Logging and you can add logging as:

   Log.debug("ClassName", "button clicked");
 

You can use “adb logcat” command to view logs, e.g. “adb logcat Twitter:* *:S”

Security Permissions

Any permissions that user need must be defined in the manifest file.

Advanced components

Android comes with a number of advanced components such as Map, Menus, Graphics, Animations, Multimedia, Preferences, Sqlite databases, etc.

Cloud to Device Push

This is a new service similar to iPhone push notification.

Debugging

Marko suggested use of Eclipse debugger, logcat, hierarchy viewer and traceview for debugging.

For more information on Android session, Download the slides.

The Seductions of Scala

For the second half of the day, I attended Dean Wampler’s session on Scala. I have been long interested in Scala and have done a little development on my own. As OSCON offered a lot of sessions on Scala, I took the opportunity to learn more on Scala. Dean highlighted concurrency, concise code, and correctness, better object model as major benefits of Scala.

Introduction

Scala comes with a number of features for concise code such as implicit type conversion, properties with uniform access principle and optional semicolons and paranthesis when function arguments are one or empty. Scala allows objects to act as function using apply method, e.g.

   class Logger(val level: Level) {
         def apply(message: String) = {
                 log(level, message)
         }
   }
 
 val error = new Logger(ERROR) ...
 error("Network error.")
 

Also, Scala treats primitive types as objects, but are comiled down as primitivies. Scala also treats functions as objects, e.g. you can create list or map without new

 val list = List(1, 2, 3, 4, 5)
 val map = Map("name" -> "Dean", "age" -> 39)
 

Above list syntax is same as 1::2::3::4::5::Nil. Scala also allows any symbol for function name so you can define functions that look like operator overloading. Scala uses infix operator notation, for example, following two expressions are equivalent:

 "hello" + "world" 
 "hello".+("world")
 

Scala gives you higher level operations such as map, filter, fold/reduce, e.g.

 val list = "a" :: "b" :: Nil
 list map {
         s => s.toUpperCase
 }
 

Generics

Martin Ordesky added limited support of Generics in Java, but he added fully functional generics support in Scala, e.g.

 class List[A] { ...
 def map[B](f: A => B): List[B]
 ... 
 }
 

Traits

One of nice feature of Scala is its support of Traits, which are interfaces with implementation and are similar to Ruby mixins. Here is an example:

 trait Logger { def log(level: Level,
 message: String) = { Log.log(level, message)
 } }
 val dean = new Person(...) extends Logger
 dean.log(ERROR, "Bozo alert!!")
 

Scala also defines traits for functions to convert them into objects, e.g.

 trait Function1[A,R] extends AnyRef {
 def apply(a:A): R
 ... }
 

User-defined factory methods

You can define functions as factory-methods to instantiate objects, e.g.

  val persons = Map("dean" -> deanPerson, "alex", -> alexPerson)
 

DSL

Scala offers powerful semantics to define internal DSLs, e.g. you can create your own controls, e.g.

 import java.io._ object Loop {
 {...}
 }
 def loop(file: File, f: (Int,String) => Unit) =
 ...
 loop (new File("...")) { (n, line) => ...
 }
 

Options as alternative to Null

Scala avoid NullPointerExceptions by wrapping nulls into options, e.g.

 abstract class Option[T] {...} case class Some[T](t: T)
 extends Option[T] {...} case object None
 extends Option[Nothing] {...}
 

Case Classes

Case classes provide succint syntax for creatng Javabeans.

For comprehensions

Scala provides for comprehensions, which are similar to Python generators, e.g.

 val l = List( Some("a"), None, Some("b"), None, Some("c"))
 for (Some(s) <- l) yield s
 

Actors

Scala provides Actor based concurrency similar to Erlang, though there are multiple implementations and Akka seems to provide better implementation than what comes with Scala. Here is an example:

 case class Point(
 x: Double, y: Double)
 abstract class Shape { def draw()
 }
 ....
 package shapes import scala.actors._, Actor._ object ShapeDrawingActor
 extends Actor { def act() {
 loop { receive {
   case s:Shape =>
   s.draw()
 ... }
 } }
 }
 

Tutorials Day Two

On day two, I attended all-day Scala summit, which covered various topics for practical Scala.

Why Scala?

The summit started with session on "Why Scala?" by Alex Payne and Dean Wampler. Dean repeated some of same concepts from Monday's session on Scala's conciseness, concurrency, correctness, infix operator, type inference, case classes, etc. Dean then gave some examples of actors using Akka, where he calls multiple services using actors and then gather the results, e.g.

 val futures = for { s   <- services
 server <- allServersFor(s) }
 yield (server !!! HeartBeat)
 Futures.awaitAll(futures) val results = for {
 future <- futures
 result <- future.result } yield result val all = results reduceLeft(
 (r1, r2) => r1 merge r2 ) compact(render(all))
 }}
 

Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Actors

Jonas Bonér then gave brief overview of Akka, which provides a number ofabstractions for concurrency such as actors, STM, and agents. Jonas gave introduction to actors, which provide concurrency based on message-passing, shared-nothing, and mailbox. In Akka, actors can be thread based or event based, where event based actors are very light-weight and you can create millions of them (each actor takes about 600 bytes as opposed to 300 bytes in Erlang).

factory-methods

Akka uses factory-methods to hide type of actors, e.g.

 val counter = actorOf[Counter]  // counter is ActorRef
 actor.start
 actor.stop
 
 

Jonas suggested use of actorOf as opposed to "new Counter" syntax as it avoids calling methods on the objects directly. Akka uses !, !! and !!! notations to send messages, where ! is just fire and forget, !! collects results using Futures and !!! returns Future, e..g

 counter ! Tick // send message -- fire-forget
 val result  = (actor !! Message).as[String] // uses Future under the hood with timeout
 val resultOption = actor !! Message
 val result = resultOption.getOrElse(defaultResult)
 val result = resultOption.getOrElse(throw new Exception("time out"))
 val future = actor !!! Message
 future.await
 val result = future.get
 
 Futures.awaitOne(List(fut1, ..))
 Futures.awaitAll(List(fut1,))
 
 

You use self.reply to reply back to the sender and access sender using self.sender or self.senderFuture.

Immutable Messages

In order to keep actors free of side effects, messages must be immutable using case classes, tuples or lists, e.g.

 - case class Register(user: User)
 - actor ! Register(user)
 - actor ! (username, password)
 - actor ! List("xxx", "yy")
 

Dispatchers

Akka comes with a number of dispatches such as event based, thread based, reactor, etc. See Dispatchers class for more information.

Queues

Akka also comes with various queue types such as unbounded LinkedBlockingQueue, bounded LinkedBlockingQueue, etc.

ActorRegistry

ActorRegistry provides lookup methods for actors such as ActorRegistry.actorsFor.

Fault tolerance

Akka borrows concepts of hierarchy of supervisors for managing actors or processes from Erlang. Erlang's philosophy for fault tolerance is let it crash and the supervisor automatically starts failed process or group of processes. You use link(actor), unlink(actor), startLink(actor) to connect actors with supervisors and trap events using trapExit = List(classOf[ServiceException], classOf[PersistentException]), e.g.

 class Supervisor extends Actor {
   import self._
   trapExit = List(classOf[Throwable])
 }
 
 class FaultTolerantService extends Actor
   override def preRestart
   override def postRestart
 

Remote Actors

You can start a node, which is remotely accessible using:

 RemoteNode.start("localhost", 9999)
 spawnLinkRemote[MyActor]("darkstar", 9999)
 
 RemoteNode.register("service:id", )
 

STM, Transactors, Modules, Camel, Storage

We ran out of time for the rest of contents, but you can read more from the Slides.

Simple Build Tool

Next, Mark Harrah presented SBT, which everyone raved at the summit. SBT uses Scala based DSL for writing build scripts and internally uses Ivy for managing dependencies. You can create a new project by creating a new directory and typing sbt. You can set various properties in sbt shell such as target version, e.g.

 set build.scala.versions 2.80
 reload
 

You can easily create custom tasks in sbt by extending DefaultProject, e.g.

 import sbt._
 class MyProject(info: ProjectInfo) extends DefaultProject(info) {
   lazy val hi = task { println("Hi"); None}
   lazy val goodbye = task { println("Bye"); None} dependsOn(hi)
 }
 

You can also target tasks for just test using

 import sbt._
 class MyProject(info: ProjectInfo) extends DefaultProject(info) {
   val sc = "org.scala-tools.testing" %% "scalacheck" % "1.7" % "test"
 }
 OR
 import sbt._
 class MyProject(info: ProjectInfo) extends DefaultProject(info) {
   val sc = "org.scala-tools.sbt" %% "launcher-interface" % "0.74"
   val tt = "org.scala-tools.sbt" %% "launcher-interface" % "0.74" % "test"
 }
 

You can define main application as follows:

 import xsbti._
 class HW extends AppMain {
   def run(config: AppConfiguration): MainResult = {config.arguments foreach println; new Exit {def code = 0}}
 }
 

You generate executable jar by typing publish-local in sbt shell. You can define plugins as follows:

 class Plugins(inf: ProjectInfo) extends PluginDefinition(info) {
 val android = "org.scala-tools.sbt" % "sbt-android-plugin" % "0.5.0"
 }
 

Finally, sbt allows you to create processors, which behave like scaffolding in Rails, e.g.

 import xsbti._
 import processor._
 class HW extends BasicProcessor {
   def apply(project: Project, args: String) {
     import project._
         val contents = "This is " + name + "" + version + "\n" + args + "\n"
         FileUtilities.write(info.projectPath / "README" asFile, contents, log)
   }
 }
 

When you type publish, it will create README file for the project. That was pretty much the introduction to the sbt.

Specs & Scala, Tips and Tricks for a Friendly DSL Syntax

Eric Torreborre talked about Spec, which a BDD based testing tool for Scala. Spec provides support for BDD, Structures, Matchers, ScalaCheck, Mocks, Runners and databases. You use matchers to compare strings or XML contents, e.g.

 class Reverse2Spec extends  Specficiation {
   reverse("") must_== ""
 ...
 
 

You can restrict scope by defining tag method, e.g.

 class Example(des : String) {
   def in(arg: Any) = expectations
   def tag(t : String) = this
 }
 

Scala DSL

Spec uses a number of tricks for simplifying the syntax such as implicit parameters, operators, lazy evaluation.

 "With a 3 tests ok" in {
   prop 
 }
 //Some paraemeters can be implicit
 implicit val defaultparams = new Params
 
 "this should not explode" in {
   error("boom")
 }
 def in(e: => Any) =                                     // parameters are evaluated lazily when you use it
 

It also uses principles such as add principle by adding new functionality, e.g.

 result + 1
 result.pp + 1
 

Spec also supports table similar to Fit and Fitness for writing concise tests. Overall, I was impressed with wide set of tools for writing tests.

Lift: Quick and Fun

Lift is a Scala based web framework for writing secure, typesafe, concise, and interactive (like desktop) applications. It abstracts much of plumbing of HTTP, which I personally don't like as I have found web frameworks that does that results in leaky abstractions. Lift also uses stateful web applications, which require sticky sessions, which is another area that I have found to be problematic for scalability and upgrade. Here is an example of Lift chat server:

 package code.comet
 import net.liftweb._
 import http._
 import actor._
 import scala.xml.
 object ChatServer extends LiftActor withListenerManager {
         private var msgs = List("Welcome")
         def createUpdate = msgs
         override def lowPriority = {
                 case s: String => msgs ::= s; updateListeners()
         }
 }
 
 class Chat extends CometActor withCometListener {
   private var msgs: List[String] = Nil
   def regiserWith = ChatServer
   override def lowPriority = {
      case l: List[String] = msgs = l; reRender(false)  // don't use reRender
   }
   def line(in: NodeSeq) : NodeSeq = msgs.reverse.flatMap(m => bind("chat", in, "item" -> m))
   def render = bind("chat", "line" -> line _)
 }
 

In Lift, every component has GUID and version that was used to render and then sets up long poll and then receive deltas (every 100ms). You can use sbt to deploy jetty and prepare war file, e.g.

 sbt
 >jetty-run
 >prepareWeb                                                     // uses JRebel to reload classes
 

Rewiring Android with Scala

This was another interesting talk by Nathan Hamblen for using Scala for writing Android applications. The key selling point of Scala has been conciseness of the language, and you can write simple code such as:

 dialog.setOnShowListener { di: DialogInterface => 
   runSomeCode() 
 }
 

instead of

 dialog.setOnShowListener(
   new DialogInterface.OnShowListener() {
     public void onShow(DialogInterface interface) {
       runSomeCode();
     }
   }
 );
 

or

 future { runSomeCode(myObject) }
 

instead of

 new AsyncTask () {
   protected Integer doInBackground(MyObject... objs) {
     runSomeCode(objs[0]);
   }
 }.execute(myObject);
 
 

Nathan showed how you can define Scala traits to add lazy handlers for android code, e.g.

 trait ScalaActivity extends Activity {
 ...
 lazy val handler = new Handler
 def post(block: => Unit) { 
   handler.post(new Runnable{
     def run { block }
   })
 }
 

Or you can extend APIs, e.g.

 implicit def f2cancel(block: DialogInterface => Unit) = 
   new DialogInterface.OnCancelListener {
     def onCancel(dialog: DialogInterface) { 
       block(dialog) 
     }
   }
 ...
 new AlertDialog.Builder(this)
   .setOnCancelListener { 
     di: DialogInterface => finish() 
   }
 

Nathan also showed a plugin (sbt-android-plugin) to create type-safe layout instead of using R.java file generated by Android, which you can get it from git clone git://github.com/meetup/meetabout.git. On the downside, Scala based android applications require Scala jar files and the size of application becomes considerable large. Though, you can use tools to extract the classes that you need, but it would still be larger than Java code.

Scala in Practice

Alex Payne and Coda Hale had a section on Scala in practice, but it was only Q/A session. I was a bit disappointed that they didn't come prepare with actual usage or war stories from their work environment.

High Wizardry in the Land of Scala

The last section of the day was a bit on type and category theory, which was interesting but mostly theortical. Daniel Spiewak explained difference between kind and type system. The only tip from the session I got was that Values are to types as types are to kinds. Finally, Daniel explained that newly released 2.8.0 version of Scala supports continuation but it's all broken and useless.

Summary

Overall, I found sessions on both Android and Scala were well worth the time and it peaked my interest in both. I think the ecosystem of Scala has matured and there is better tools support with the new version (2.8). I am going to try to influence co-workers into using it for new development. I am also going to start Android project pretty soon but I am a bit hesitant on writing in Scala due to increased application size.

June 3, 2010

A few lessons from Seth Godin’s book – Linchpin: Are You Indispensable?

Filed under: Business — admin @ 10:25 pm

I just finished reading Seth Godin’s new book Linchpin: Are You Indispensable?. Seth shows how the white-collar jobs, which supposed to save the middle class are being eliminated either by machines or outsourcing with cheap labors. He shows that you can either continue to live your life as a faceless cog or choose to become Linchpin. Here are some of the lessons I learned from this book:

Industrial Revolution is Over

The race to make average stuff for average people in huge quantities is almost over.

This book shows the industrial revolution is changing and in order to survive in the new era of economy, you have to become linchpin or indispensable. In last three hundred years, the industrialization began by standardizing the tasks so that it can be performed by easily replaceable labor or so called cogs. It relied on two layers: management and labor, where management breaks production of goods into tiny tasks, which are performed by the labor. The management wins when it can get the most work for the least pay. The system taught workers to follow the instructions and you don’t have to think. Though, that system worked but has been falling apart in the face of competition, outsourcing and globalization. The attendance-based compensation (ABC) is over. Th old American dream that taught to keep your head down, follow instructions, work hard and you will be rewarded is dead. The mass production treats everything such as labor and material as interchangeable. However, in global market, the competition is fierce and cheap strategy doesn’t scale very well.
Instead of easily replaced laborers or cogs, you can choose to become Linchpin by differentiating yourself from the rest and focusing on humanity, connection and art. The web has made it easier to be productive and create or invent. The new American dream is to be remarkable, generous, create art and connect with people.


Education System is a Sham


In capitalist market, the companies make money by hiring obedient and competent workers as cheaply as you can and using productivity advantage to earn more profit. Andrew Carnegie saw that limited amount of education to get them to cooperate. The school system throughout the world encourages mediocre obedience and is driven by fear as when we learn things in fear. Seth shows public school system is designed to prepare us for factories, where we are just replaceable cogs and care little about our jobs or customers. The same factory model created consumer culture that uses consumption as a shortcut to happniess. Instead, school should teach solving interesting problems and leading.


Becoming a Linchpin


In order to become a linchpin or indispensable, you must embrace an artist and genius within you. Seth recommends avoiding asympototic goals such as bowling, where there is a ceiling of how good you can be. Also, for an artist, the economy is not just zero sum game, instead he/she can increase the pie. Seth cites Richard Florida’s survey of top ten reason for employees to do best work as follows:

  • challenge and responsibility
  • flexibility
  • stable work environment
  • money
  • professional development
  • peer recognition
  • stimulatng colleagues and bosses
  • exciting job content
  • organization culture
  • location and community

All of above reasons except money are internal that we can control. Seth encourages readers to find the work that suits your passion. He uses Emotional labor term, originally coined by Arlie Hochschild to connect with the work. Though, you may get a little compensation in return of emotional labor, but you get inward reward. Instead of day’s work for day’s job or the poverty mentality that treats life as zero sum game, you give gift and build bonds. Seth shows that the easier work is to quantify, the less it’s worth and more humanity you bring to your work, the better results you will receive. Seth cites Krulak’s law for building strong relations with your customers, i.e.,

The closer you get to the front, the more power you have over the brand.
 


Resistance to Change


Seth gives plenty of examples and demonstrates that real artists ship, however shipping is hard due to trashing/tweaking and coordination. According to Seth, the biggest resistance to the change is our lizard brain. He explains how we all have two brains: primeval brain or lizard brain and gray matter or recently developed brain. The lizard brain has animal instincts such as hungry, scared, angry and horny, whereas newer brain allows big thoughts, generosity, speech, and art. Lizard brain seek compfort and obedience, and avoids risks, public speaking and generosity.


Good is enemy of perfect


Seth encourages readers to become excellent and not perfect as art is never defect-free. He cites Bre Pettis, who says that there are three states of being: not knowing, action and completion. He says accept that everything is draft as it helps to get it done.


Generosity


Exchanging gifts is an ancient tradition. Seth shows that artists who give gifts win as becoming a linchpin is not an act of selfishness. Seth also shows how usury was prohibited in Bible as interest-free loan was kind of gift. This changed when Martin Luther lifted the sanction to get support for the Protestant Reformation. Seth writes:

For the last five hundred years, the best way to succeed has been to treat everyone as a stranger you could do business with.

Seth cites Metcalfe’s law states that the value of a network increases with the square of the number of nodes on the network. The new social media platforms such as Twitter, Facebook, Blogsphere, and Internet is changing the circle of the gift system and he shows that there are three cicles of gifts, the first circle represents true gifts to family and friends. The second circle is for commerce, they pay for souvenir edition and the third circle is your tribe, followers, fans or friendlies.


There is no map


Seeing the future is hard because we are attached to the world and want stability and fear change. Seth gives plenty of examples of record industry and newspaper industry who have been too attached with their legacy model and failed to adjust in the new economy. In order to become linchpin, you need to draw a map and lead instead of being passive. You need to find a job that matches your passion.


Culture of connections

How to Make a Personal Connection with Customers
The industrialization removed human connection between different parties. The social media and Internet is changing that, now companies can connect directly with their customers and receive their input. Often, when companies negotiate with other companies, the key point of distinction is the perceived connection between the prospect and the organization. The salesman who relies only on the script would fail, instead you have to rely on honest signals and genuine gifts to make connections.


Seven attributes of Linchpin

Linchpins are geniuses, artists and givers of gifts, who extert emotional labor and make their own map. Here are seven abilities of the linchin:

  • Providing a unique interface between members of the organization
  • Delivering unique creativity
  • Managing a situation or organization of great complexity
  • Leading customers
  • Inspiring staff
  • Providing deep domain knowledge
  • Possessing a unique talent

Conclusion

We have been in declining economy for a while and many of the white collar and blue collar jobs lost in last few years won’t come back. I found a lot of Seth’s advice similar to agile movement in software development and My Job Went to India. I also wrote about Taylorism in my blog IT Sweatshops, where I deplored Taylorism based command and control structure in a lot of companies even the one that claim to adopt agile methodologies. Seth even says that you don’t need a resume as it hides the fact that you are linchpin. Instead have a project that an employer can see or blog that people can follow. I find this book offers very practical and timely advice for future market. In the job market, You need to differentiate yourself and have a trail of breadcrumbs of your previous work. Being an average is over, instead you have to be a linchpin and live without a map.

« Newer PostsOlder Posts »

Powered by WordPress