Shahzad Bhatti Welcome to my ramblings and rants!

Problem Description

I recently had to make some architecture changes to our application and before making those changes, I decided to add a suite of functional tests to cover essential areas of the application. Like most real world dev. shops, testing is not integral part of the development process at Amazon and though, we had some unit tests but didn’t have any functional tests. I needed to get something quickly. Another thing was that our application uses a lot of AJAX, especially search functionality retrieves results using AJAX instead of showing everything when the form is submitted. So, I needed something that could test AJAX with real browser. I had heard of Selenium and a bit experience with it so I chose it for functional testing. Next, I needed to setup a load test environment and I chose Grinder due to some prior experience as well. The rest of the blog shows how to actually setup these two together as there were some glitches.

Selenium IDE

First, install Firefox plugin for Selenium IDE from http://www.openqa.org/selenium-ide/. I used it to record functional tests, which is pretty easy. Once the plugin is installed, select it from Tools->Selenium IDE and it will automatically start recording it. Point your firefox to your application and start capturing the use case that you are interested in. I suggest recording one use case at a time and when you are finished with the use case, switch over to the IDE and click “Stop Recording” red button. You can then choose to export the captured use case in a number of languages. I chose to export it to Java. For example, my simple test looked like:

 1 package com.amazon.biw2.webapp.tests; 
 2 
 3 import com.thoughtworks.selenium.*; 
 4 import java.util.regex.Pattern; 
 5 import java.util.Arrays; 
 6 import junit.framework.TestSuite; 
 7 
 8 
 9 public class BiwSearchTest extends SeleneseTestCase { 
10     private static final String url = System.getProperty("url", "http://shahbhat.desktop:8080"); 
11     static final long SLEEP_TIME = 60000; 
12 
13 
14     public void setUp() { 
15         selenium = new DefaultSelenium("localhost", 4444, "*firefox", url); 
16         selenium.start(); 
17     } 
18 
19 
20     public void tearDown() { 
21         selenium.stop(); 
22     } 
23 
24     public void testProductSearch() throws Exception { 
25         selenium.open("/searchProductForm.html?TestMode=true"); 
26         selenium.waitForPageToLoad(String.valueOf(SLEEP_TIME)); 
27         selenium.click("link=Search by Products"); 
28         selenium.select("reportType", "label=Unfilled Demand"); 
29         selenium.select("GLProductGroup", "label=Apparel"); 
30         selenium.click("//a[@onclick='if(validateSelection()){submitSearch();} return false;']"); 
31         selenium.waitForPageToLoad(String.valueOf(SLEEP_TIME)); 
32         verifyFalse(selenium.isTextPresent("No results")); 
33         verifyTrue(selenium.isTextPresent("Loading: ")); 
34         selenium.getEval("this.browserbot.getCurrentWindow().initBody();"); 
35         String done = selenium.getValue("allDataFieldsComplete"); 
36         while ("true".equals(done) == false) { 
37             System.out.println("Waiting to load " + done); 
38             Thread.currentThread().sleep(5000); 
39             done = selenium.isElementPresent("allDataFieldsComplete") ? selenium.getValue("allDataFieldsComplete") : "not-present"; 
40         } 
41     } 
42 
43     public static void main(String[] args) throws Exception { 
44         junit.textui.TestRunner.run(BiwSearchTest.class); 
45     } 
46 
47     public static TestSuite suite() { 
48         TestSuite suite = new TestSuite(BiwSearchTest.class); 
49         return suite; 
50     } 
51 } 
52

Selenium Remote Control

Next, I downloaded Selenium Remote Control from http://www.openqa.org/selenium-rc/. However, I found that that version does not work with Firefox 2.0 so I downloaded snapshot version from http://release.openqa.org/selenium-remote-control/nightly/. After unzipping it, I started remote control server with:

java -jar selenium-server.jar

I then compiled and ran my program
javac -d classes -classpath classes:selenium-remote-control-0.9.1-SNAPSHOT/java/selenium-java-client-driver.jar:selenium-remote-control-0.9.1-SNAPSHOT/server/selenium-server.jar: BiwSearchTest.java


java -classpath classes:selenium-remote-control-0.9.1-SNAPSHOT/java/selenium-java-client-driver.jar:selenium-remote-control-0.9.1-SNAPSHOT/server/selenium-server.jar: com.amazon.biw2.webapp.tests.BiwSearchTest


As expected, it launched a firefox browser window (I closed all my firefox before running it), ran my test and then closed the browser.

Selenium Gotchas

I ran into a couple of Selenium gotaches. First, Selenium does not run your onLoad javascript and I had to explicitly call eval method to start the javascript. Second, some of the operations on Selenium don’t work. For example, one of the form showed results in another window and to get the handle of new window, I tried iterating through windows using Selenium’s getAllWindowsByIds/Names/Titles, but could not do it. I even tried getAttributeFromAllWindows, but that didn’t work either. Also, SeleniumTest class does not define constructor to take in name of a test, so you can’t just run a single test.

Setting up Grinder

Next I downloaded grinder 3.0 from http://grinder.sourceforge.net/download.html. I picked latest beta 33 release. I then created a properties file as follows:

grinder.processes=2 
grinder.threads=5 
grinder.runs=5 
grinder.script=biw_search.py 
grinder.logDirectory=logs 
grinder.numberOfOldLogs=0 
grinder.statistics.delayReports = 1 
grinder.consoleHost=localhost 
grinder.consolePort=6372 
grinder.processIncrementInterval=60000ms 
grinder.initialSleepTime=60000ms 
grinder.reportToConsole.interval=500ms

I then created a python script to kick off the Selenium test.

 1 from net.grinder.script.Grinder import grinder 
 2 from net.grinder.script import Test 
 3 from net.grinder.plugin.http import HTTPRequest 
 4 from java.lang import System 
 5 from java.lang import String 
 6 
 7 from junit.framework import TestSuite 
 8 from junit.framework import TestResult 
 9 
10 from com.amazon.biw2.webapp.tests import BiwAdpTest 
11 from com.amazon.biw2.webapp.tests import BiwAsinSearchTest 
12 from com.amazon.biw2.webapp.tests import BiwKeywordSearchTest 
13 from com.amazon.biw2.webapp.tests import BiwMediaSearchTest 
14 from com.amazon.biw2.webapp.tests import BiwProductSearchTest 
15 from com.amazon.biw2.webapp.tests import BiwSearchTest 
16 
17 
18 
19 def createTestRunner(script): 
20     exec("x = %s.TestRunner()" % script) 
21     return x 
22 
23 class TestRunner: 
24     def __init__(self): 
25         tid = grinder.threadID 
26         self.initialisationTime = System.currentTimeMillis() 
27         #if tid % 5 == 4: 
28         #    self.testRunner = createTestRunner(scripts[1]) 
29 
30     def __call__(self): 
31         # Turn off automatic reporting for the current worker thread. 
32         # Having done this, the script can modify or set the statistics 
33         # before they are sent to the log and the console. 
34         grinder.statistics.delayReports = 1 
35 
36         tid = grinder.threadID 
37 
38         # Creates a Test Suite. 
39         if tid % 5 == 4: 
40             suite = TestSuite(BiwAdpTest().getClass()) 
41         elif tid % 5 == 3: 
42             suite = TestSuite(BiwAsinSearchTest().getClass()) 
43         elif tid % 5 == 2: 
44             suite = TestSuite(BiwKeywordSearchTest().getClass()) 
45         elif tid % 5 == 1: 
46             suite = TestSuite(BiwMediaSearchTest().getClass()) 
47         else: 
48             suite = TestSuite(BiwProductSearchTest().getClass()) 
49 
50         # Returns the tests as an enumeration. 
51         tests = suite.tests(); 
52 
53         # Iterate over the tests. 
54         testNumber = 0 
55         for test in tests: 
56             testNumber += 1 
57             testCase = Test(testNumber, test.getName()).wrap(suite) 
58 
59             testResult = TestResult() 
60             testCase.runTest(test, testResult) 
61 
62             if testResult.errorCount() > 0: 
63                 grinder.statistics.success = 0 
64             elif testResult.failureCount() > 0: 
65                 grinder.statistics.success = 0 
66 
67 
68     def __del__(self): 
69         elapsed = System.currentTimeMillis() - self.initialisationTime 
70

Grinder Gotchas

First I tried to call my test directly, but noticed that I wasn’t getting any data, so I had to wrap my test with Test to get the statistics.

Setting up Grinder Console/Agents

When setting up grinder, you launch test agents on multiple machine and then start a console on your local laptop or workstation. You kick of tests from console, once you start the tests all agents will start running your tests and will send back statistics to the console and you can view them on your console. For my test, I ran both agent and console on my laptop, but running agents on multiple servers will be similar.
I started console using
java -classpath grinder-3.0-beta33/lib/grinder-j2se5.jar;grinder-3.0-beta33/lib/grinder-xmlbeans.jar;grinder-3.0-beta33/lib/grinder.jar;grinder-3.0-beta33/lib/jsr173_1.0_api.jar;grinder-3.0-beta33/lib/jython.jar;grinder-3.0-beta33/lib/picocontainer-1.2-RC-1.jar;grinder-3.0-beta33/lib/xbean.jar net.grinder.Console