Shahzad Bhatti

July 2, 2007

Load and Functional Testing with Selenium and Grinder

Filed under: Computing — admin @ 4:06 pm

Problem Description

I recently had to make some architecture changes to our application and before making those changes, I decided to add a suite of functional tests to cover essential areas of the application. Like most real world dev. shops, testing is not integral part of the development process at Amazon and though, we had some unit tests but didn’t have any functional tests. I needed to get something quickly. Another thing was that our application uses a lot of AJAX, especially search functionality retrieves results using AJAX instead of showing everything when the form is submitted. So, I needed something that could test AJAX with real browser. I had heard of Selenium and a bit experience with it so I chose it for functional testing. Next, I needed to setup a load test environment and I chose Grinder due to some prior experience as well. The rest of the blog shows how to actually setup these two together as there were some glitches.

Selenium IDE

First, install Firefox plugin for Selenium IDE from I used it to record functional tests, which is pretty easy. Once the plugin is installed, select it from Tools->Selenium IDE and it will automatically start recording it. Point your firefox to your application and start capturing the use case that you are interested in. I suggest recording one use case at a time and when you are finished with the use case, switch over to the IDE and click “Stop Recording” red button. You can then choose to export the captured use case in a number of languages. I chose to export it to Java. For example, my simple test looked like:

 1 package; 
 3 import com.thoughtworks.selenium.*; 
 4 import java.util.regex.Pattern; 
 5 import java.util.Arrays; 
 6 import junit.framework.TestSuite; 
 9 public class BiwSearchTest extends SeleneseTestCase { 
10     private static final String url = System.getProperty("url", "http://shahbhat.desktop:8080"); 
11     static final long SLEEP_TIME = 60000; 
14     public void setUp() { 
15         selenium = new DefaultSelenium("localhost", 4444, "*firefox", url); 
16         selenium.start(); 
17     } 
20     public void tearDown() { 
21         selenium.stop(); 
22     } 
24     public void testProductSearch() throws Exception { 
26         selenium.waitForPageToLoad(String.valueOf(SLEEP_TIME)); 
27"link=Search by Products"); 
28"reportType", "label=Unfilled Demand"); 
29"GLProductGroup", "label=Apparel"); 
30"//a[@onclick='if(validateSelection()){submitSearch();} return false;']"); 
31         selenium.waitForPageToLoad(String.valueOf(SLEEP_TIME)); 
32         verifyFalse(selenium.isTextPresent("No results")); 
33         verifyTrue(selenium.isTextPresent("Loading: ")); 
34         selenium.getEval("this.browserbot.getCurrentWindow().initBody();"); 
35         String done = selenium.getValue("allDataFieldsComplete"); 
36         while ("true".equals(done) == false) { 
37             System.out.println("Waiting to load " + done); 
38             Thread.currentThread().sleep(5000); 
39             done = selenium.isElementPresent("allDataFieldsComplete") ? selenium.getValue("allDataFieldsComplete") : "not-present"; 
40         } 
41     } 
43     public static void main(String[] args) throws Exception { 
45     } 
47     public static TestSuite suite() { 
48         TestSuite suite = new TestSuite(BiwSearchTest.class); 
49         return suite; 
50     } 
51 } 

Selenium Remote Control

Next, I downloaded Selenium Remote Control from However, I found that that version does not work with Firefox 2.0 so I downloaded snapshot version from After unzipping it, I started remote control server with:

java -jar selenium-server.jar

I then compiled and ran my program
javac -d classes -classpath classes:selenium-remote-control-0.9.1-SNAPSHOT/java/selenium-java-client-driver.jar:selenium-remote-control-0.9.1-SNAPSHOT/server/selenium-server.jar:

java -classpath classes:selenium-remote-control-0.9.1-SNAPSHOT/java/selenium-java-client-driver.jar:selenium-remote-control-0.9.1-SNAPSHOT/server/selenium-server.jar:

As expected, it launched a firefox browser window (I closed all my firefox before running it), ran my test and then closed the browser.

Selenium Gotchas

I ran into a couple of Selenium gotaches. First, Selenium does not run your onLoad javascript and I had to explicitly call eval method to start the javascript. Second, some of the operations on Selenium don’t work. For example, one of the form showed results in another window and to get the handle of new window, I tried iterating through windows using Selenium’s getAllWindowsByIds/Names/Titles, but could not do it. I even tried getAttributeFromAllWindows, but that didn’t work either. Also, SeleniumTest class does not define constructor to take in name of a test, so you can’t just run a single test.

Setting up Grinder

Next I downloaded grinder 3.0 from I picked latest beta 33 release. I then created a properties file as follows:

grinder.statistics.delayReports = 1 

I then created a python script to kick off the Selenium test.

 1 from net.grinder.script.Grinder import grinder 
 2 from net.grinder.script import Test 
 3 from net.grinder.plugin.http import HTTPRequest 
 4 from java.lang import System 
 5 from java.lang import String 
 7 from junit.framework import TestSuite 
 8 from junit.framework import TestResult 
10 from import BiwAdpTest 
11 from import BiwAsinSearchTest 
12 from import BiwKeywordSearchTest 
13 from import BiwMediaSearchTest 
14 from import BiwProductSearchTest 
15 from import BiwSearchTest 
19 def createTestRunner(script): 
20     exec("x = %s.TestRunner()" % script) 
21     return x 
23 class TestRunner: 
24     def __init__(self): 
25         tid = grinder.threadID 
26         self.initialisationTime = System.currentTimeMillis() 
27         #if tid % 5 == 4: 
28         #    self.testRunner = createTestRunner(scripts[1]) 
30     def __call__(self): 
31         # Turn off automatic reporting for the current worker thread. 
32         # Having done this, the script can modify or set the statistics 
33         # before they are sent to the log and the console. 
34         grinder.statistics.delayReports = 1 
36         tid = grinder.threadID 
38         # Creates a Test Suite. 
39         if tid % 5 == 4: 
40             suite = TestSuite(BiwAdpTest().getClass()) 
41         elif tid % 5 == 3: 
42             suite = TestSuite(BiwAsinSearchTest().getClass()) 
43         elif tid % 5 == 2: 
44             suite = TestSuite(BiwKeywordSearchTest().getClass()) 
45         elif tid % 5 == 1: 
46             suite = TestSuite(BiwMediaSearchTest().getClass()) 
47         else: 
48             suite = TestSuite(BiwProductSearchTest().getClass()) 
50         # Returns the tests as an enumeration. 
51         tests = suite.tests(); 
53         # Iterate over the tests. 
54         testNumber = 0 
55         for test in tests: 
56             testNumber += 1 
57             testCase = Test(testNumber, test.getName()).wrap(suite) 
59             testResult = TestResult() 
60             testCase.runTest(test, testResult) 
62             if testResult.errorCount() > 0: 
63                 grinder.statistics.success = 0 
64             elif testResult.failureCount() > 0: 
65                 grinder.statistics.success = 0 
68     def __del__(self): 
69         elapsed = System.currentTimeMillis() - self.initialisationTime 

Grinder Gotchas

First I tried to call my test directly, but noticed that I wasn’t getting any data, so I had to wrap my test with Test to get the statistics.

Setting up Grinder Console/Agents

When setting up grinder, you launch test agents on multiple machine and then start a console on your local laptop or workstation. You kick of tests from console, once you start the tests all agents will start running your tests and will send back statistics to the console and you can view them on your console. For my test, I ran both agent and console on my laptop, but running agents on multiple servers will be similar.
I started console using
java -classpath grinder-3.0-beta33/lib/grinder-j2se5.jar;grinder-3.0-beta33/lib/grinder-xmlbeans.jar;grinder-3.0-beta33/lib/grinder.jar;grinder-3.0-beta33/lib/jsr173_1.0_api.jar;grinder-3.0-beta33/lib/jython.jar;grinder-3.0-beta33/lib/picocontainer-1.2-RC-1.jar;grinder-3.0-beta33/lib/xbean.jar net.grinder.Console

I then kicked of test agent using
java -classpath -classpath classes:selenium-remote-control-0.9.1-SNAPSHOT/java/selenium-java-client-driver.jar:selenium-remote-control-0.9.1-SNAPSHOT/server/selenium-server.jar:grinder-3.0-beta33/lib/grinder-j2se5.jar;grinder-3.0-beta33/lib/grinder-xmlbeans.jar;grinder-3.0-beta33/lib/grinder.jar;grinder-3.0-beta33/lib/jsr173_1.0_api.jar;grinder-3.0-beta33/lib/jython.jar;grinder-3.0-beta33/lib/picocontainer-1.2-RC-1.jar;grinder-3.0-beta33/lib/xbean.jar net.grinder.Grinder

By default the agent waits until you kick off the test from console so I switched over to console and click start and the console started collecting data. Now onto real work which is writing my own algorithm for load balancer because default load balancer is just too dumb. What I would like is an algorithm that takes user's network into account to find closest server (we have three data servers all over world), and actual health of the server.

Powered by WordPress