Automated Testing Patterns and Smells

Wonderful tech talk by Gerard Meszaros who is a consultant specialising in agile development processes. In this particular presentation Gerard describes a number of common problems encountered when writing and running automated unit and functional tests. He describes these problems as “test smells”, and talks about their root causes. He also suggests possible solutions which he expresses as design patterns for testing. While many of the practices he talks about are directly actionable by developers or testers, it’s important to realise that many also require action from a supportive manager and/or system architect in order to be really achievable.

We use many flavours of xUnit test frameworks in our development group at Talis, and we generally follow a Test First development approach, I found this talk beneficial because many of the issues that Gerard talks about are problems we have encountered and I don’t doubt every development group out there, including ours, can benefit from the insight’s he provides.

The material he uses in his talk and many of the examples are from his book xUnit Test Patterns: Refactoring Test Code, which I’m certainly going to order.

Automated regression testing and CI with Selenium PHP

I’ve been doing some work this iteration on getting Selenium RC integrated into our build process so we can run a suite of automated functional regression tests against our application on each build. The application I’m working on is written in PHP, normally when you use Selenium IDE to record a test script it saves it as a HTML file.

For example a simple test script that goes to Google and verifies that the text “Search:” is present on the screen and the title of the page is “iGoogle” looks like this:

  2. <html>
  3. <head>
  4. <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  5. <title>New Test</title>
  6. </head>
  7. <body>
  8. <table cellpadding="1" cellspacing="1" border="1">
  9. <thead>
  10. <tr><td rowspan="1" colspan="3">New Test</td></tr>
  11. </thead><tbody>
  12. <tr>
  13.     <td>open</td>
  14.     <td>/ig?hl=en</td>
  15.     <td></td>
  16. </tr>
  17. <tr>
  18.     <td>verifyTextPresent</td>
  19.     <td>Search:</td>
  20.     <td></td>
  21. </tr>
  22. <tr>
  23.     <td>assertTitle</td>
  24.     <td>iGoogle</td>
  25.     <td></td>
  26. </tr>
  28. </tbody></table>
  29. </body>
  30. </html>

You can choose to export the script in several other languages, including PHP, in which case the test script it produces looks like this:

  2. <?php
  3. require_once ‘Testing/Selenium.php’;
  4. require_once ‘PHPUnit/Framework/TestCase.php’;
  6. class Example extends PHPUnit_Framework_TestCase
  7. {
  8.   function setUp()
  9.   {
  10.     $this->verificationErrors = array();
  11.     $this->selenium = new Testing_Selenium("*firefox", "http://localhost:4444/");
  12.     $result = $this->selenium->start();
  13.   }
  15.   function tearDown()
  16.   {
  17.     $this->selenium->stop();
  18.   }
  20.   function testMyTestCase()
  21.   {
  22.     $this->selenium->open("/ig?hl=en");
  23.     try
  24.     {
  25.       $this->assertTrue($this->selenium->isTextPresent("Search:"));
  26.     }
  27.     catch (PHPUnit_Framework_AssertionFailedError $e)
  28.     {
  29.       array_push($this->verificationErrors, $e->toString());
  30.     }
  31.     try
  32.     {  
  33.       $this->assertEquals("iGoogle", $this->selenium->getTitle());
  34.     }
  35.     catch (PHPUnit_Framework_AssertionFailedError $e)
  36.     {
  37.       array_push($this->verificationErrors, $e->toString());
  38.     }
  39.   }
  40. }
  41. ?>

The Export produces a valid PHPUnit test case that uses the Selenium PHP Client Driver(Selenium.php). Whilst the script is valid and will run you do need add a little more to it before the test will correctly report errors. As it stands all errors captured during the test are added to an array called verificationErrors, by catching the assertion Exceptions that are thrown when an assert fails, in other words if you ran this test as it is, and it did fail you wouldn’t know! To correct this we need to do two things. Firstly, each assert needs to have a message added to it which will printed out in the test report if the assert fails. Secondly we need to modify the tearDown method so that once a test has run, it checks the verificationErrors array, and if any failures have occurred, fails the test. After making these changes the PHP test script looks like this:

  2. <?php
  3. require_once ‘Testing/Selenium.php’;
  4. require_once ‘PHPUnit/Framework/TestCase.php’;
  6. class GoogleHomePageTest extends PHPUnit_Framework_TestCase
  7. {
  8.   function setUp()
  9.   {
  10.     $this->verificationErrors = array();
  11.     $this->selenium = new Testing_Selenium(
  12.                          "*firefox",
  13.                          "http://localhost:4444/"
  14.                       );
  15.     $result = $this->selenium->start();
  16.   }
  18.   function tearDown()
  19.   {
  20.     $this->selenium->stop();
  21.     if(count($this->verificationErrors) > 0 )
  22.     {
  23.        echo "VERIFICATION ERRORS:" . count($this->verificationErrors);
  24.        $this->fail(implode("\n",$this->verificationErrors));
  25.     }
  26.   }
  28.   function testGoogleHomePage()
  29.   {
  30.     $this->selenium->open("/ig?hl=en");
  31.     try
  32.     {
  33.       $this->assertTrue(
  34.                $this->selenium->isTextPresent("Search:"),
  35.                "The string Search: was not found"
  36.       );
  37.     }
  38.     catch (PHPUnit_Framework_AssertionFailedError $e)
  39.     {
  40.       array_push($this->verificationErrors, $e->toString());
  41.     }
  42.     try
  43.     {  
  44.       $this->assertEquals(
  45.                    "iGoogle",
  46.                    $this->selenium->getTitle(),
  47.                    "The page title did not match iGoogle."
  48.       );
  49.     }
  50.     catch (PHPUnit_Framework_AssertionFailedError $e)
  51.     {
  52.       array_push($this->verificationErrors, $e->toString());
  53.     }
  54.   }
  55. }
  56. ?>

Obviously, I have also given the PHP Class and test function slightly more meaningful names. Now you have a PHP Unit Test case that will use the Selenium PHP Client Driver with Selenium Remote Control to launch a browser, go to the specified URL, and test a couple of assertions. If any of those assertions fail, the tearDown method fails the test … pretty cool, right?

Well now it get’s better. Because the Selenium Client Driver has a published api which is pretty easy to follow, there’s no reason why you can’t just write test cases without using Selenium IDE … for those who want to you could even incorporate this into a TDD process. But for all this to hang together we need to be able to run a build, on a Continuous Integration server which checks out the code, runs unit tests and selenium regression tests against that code line, and only if all tests succeed passes the build.

We are currently using ANT, and CruiseControl to handle our CI/Automated build process. When running the automated suite of tests we need to ensure that the Selenium Remote Control server is also running which creates some complications. The Selenium Remote Control server takes several arguments which can also include the location of a test suite of html based selenium tests – which is really nice because the server will start, execute those tests and then end. Unfortunately you can’t invoke the server and pass it the location of a PHP based test suite. This means you need to find a way to start up the server, then run your tests, and once they are complete, shut the selenium server down.

He are the ANT targets that I have written to achieve this, if anyone can think of better ways of doing this I’d welcome any feedback or suggestions, to run this example you’d simply enter the command “ant selenium” :

  2. <target name="selenium" depends="clean, init" description="Run the Selenium tests">
  3.   <parallel>
  4.     <antcall target="StartRCServer" />
  5.     <antcall target="RunSeleniumTests" />
  6.   </parallel>
  7. </target>
  9. <target name="StartRCServer" description="Start the Selenium RC server">
  10.   <java jar="dependencies/SeleniumRC/lib/selenium-server.jar"
  11.         fork="true" failonerror="true">
  12.     <jvmarg value=""/>
  13.     <jvmarg value="-Dhttp.proxyPort=80"/>
  14.   </java>
  15. </target>
  17. <target name="RunSeleniumTests" description="RunAllSeleniumTests">    
  18.   <sleep milliseconds="2000" />
  19.   <echo message="======================================" />
  20.   <echo message="Running Selenium Regression Test Suite" />
  21.   <echo message="======================================" />
  22.   <exec executable="php"
  23.        failonerror="false"
  24.        dir="test/seleniumtests/regressiontests/"
  25.        resultproperty="regError">
  26.        <arg line="../../../dependencies/PHPUnit/TextUI/Command.php –log-xml ../../../doc/SeleniumTestReports/RegressionTests/TestReport.xml AllRegressionTests" />
  27.   </exec>
  28.   <get taskname="selenium-shutdown"
  29.     src="http://localhost:4444/selenium-server/driver/?cmd=shutDown"
  30.     dest="result.txt" ignoreerrors="true" />
  32.   <condition property="regressionTest.err">
  33.     <or>
  34.        <equals arg1="1" arg2="${regError}" />
  35.        <equals arg1="2" arg2="${regError}" />
  36.     </or>
  37.   </condition>
  39.   <fail if="regressionTest.err" message="ERROR: Selenium Regression Tests Failed" />               
  40. </target>

A couple of notes, the reason I have to use a conditional check at the end of the selenium target is because, if the exec task that runs the PHP tests was set to failonerror=true then the build would never reach the next line which shuts the Selenium RC server down. To ensure that always happens I have to set the exec to failonerror=false, but this means I have to check what the result was from the exec. Which if successful will return 0, if test failures exist will return 1, and if there were any errors (preventing a test to be exectuted ) will return 2. Hence the conditional check sets regressionTest.err if either of these latter two conditions are true.

Also in order to start up the server, which could take up to a second, but can’t be sure precisely how long. I have to use the Ant Parallel task, which calls the task to start the server and the task to run the tests at the same time. The task to run the tests has a 2 second sleep in it, which should be more than enough time to allow the server to start. This all kind of feels a little clunky, but at the moment it does work very well.

In a nutshell, thats how you integrate PHP based Automated Selenium Regression tests into a continuous build.

Testing how web pages look in IE without using a PC

At Talis were developing a number of web based applications built upon our platform. When building any web based application testing across different browsers is vitally important due to compatability issues across browsers – these issues arise from differing levels of compliance to standards, and because browser vendors don’t support common baselines. This makes testing not only important but also very frustrating at times.

Here’s some browser usage statistics I found over at w3schools which show’s how commonly used each of these browsers are, the most common at the moment is IE6 followed closely by FireFox:

2007 IE7 IE6 IE5 FireFox Mozilla Safari Opera
May 19.2% 38.1% 1.5% 33.7% 1.3% 1.5% 1.6%
April 19.1% 38.4% 1.7% 32.9% 1.3% 1.7% 1.6%
March 18.0% 38.7% 2.0% 31.8% 1.3% 1.7% 1.6%
February 16.4% 39.8% 2.5% 31.2% 1.4% 1.7% 1.5%
January 13.3% 42.3% 3.0% 31.0% 1.5% 1.7% 1.5%

Now, when developing a web based application it is possible for developers to have most of these browsers installed on a single machine. For example it is now possible for me as a developer to install Firefox, Safari, Netscape, Opera and a single version of IE on my Windows desktop.

The problem that Im often faced with though is that more often than not we have to test applications across multiple versions of IE. Out of the box it isn’t possible to have two different versions of IE installed on the same machine. This is largely due to the fact that IE is integrated so tightly into the OS so as soon as you update to IE7 it removes IE6.

There are some standalone installers out there that allow you to run multiple IE’s under windows. Here’s one such utility that some of my colleagues are using successfully. Unfortunatly I haven’t managed to get it to work for me, it crashes a lot on my machine, as is often the case when your forced to hack around with DLL’s. So I started looking for some alternatives.

I use VMWare a lot – the funky laptop I have has hardware support for virtualisation which means my Virtual Machines run fast enough for me to use them for day to day development work, I like the flexibility this approach gives me, for example I think I currently have a Red Hat, Ubuntu, Fedora Core 5, 6 and 7 VM’s built on my laptop and depending on what I want to do I just fire up the one I want to use. Since I do use linux quite a lot I was intrigued when I came across ies4Linux.

IEs4Linux is a simple way to have Microsoft Internet Explorer running on Linux. It currently installs IE6, IE5.5, IE5.0 as well as several service pack versions of these onto a single box. To install it under Fedora Core, as I did, just follow these simple instructions:

Open a terminal, login as a root and install the following dependencies:

  2.    yum -y install wine*
  3.    yum -y install cabextract

Now open a terminal, logging in with your normal user account and do the following:

  2.   wget
  3.   tar zxvf ies4linux-latest.tar.gz
  4.   cd ies4linux-*
  5.   ./ies4linux

The installer is interactive and will ask you which particular versions you wish to install. After that’s all done it will place icons on your desktop. A VM can see the host machine so it really doesn’t matter whether your developing your site under linux natively or if your doing the bulk of your development under Windows. You can still fire up IE on your VM and test how your web application looks and behaves. I’ve used this approach on a couple of projects and it seems to be working out quite well.

Additionally, sometimes you don’t actually want to see to test the behaviour of the web app, you simply want to verify that it renders consistently across different browsers. A number of web based services have started appearing that allow you to specify a URL and will capture an image of that page rendered inside a particular version of a browser, or even a set of browsers. NetRenderer is one such service, which I quite like because of its simplicity, that will capture images of a page in different flavours of IE and display them immediately. On the hand there is also BrowserShots which is a far more comprehensive service that will take images of web pages on different browsers across different OS’s, this service however does not return those images immediately but batches them up and you have to wait around half an hour which can be tedious – however the service is still being developed and we should be seeing real-time offerings at some point in the future.

The Role of Testing and QA in Agile Software Development

In our development group at Talis We’ve been thinking a lot about how to test more effectively in an agile environment. One of my colleagues, sent me a link to this excellent talk by Scott Ambler which examines the Role of Testing and QA in Agile Software Development.

Much of the talk is really an introduction to Agile Development which is beneficial to listen to because Scott dispels some of the myths around agile, and offers his own views on best practises using some examples. It does get a bit heated around the 45 minute mark when he’s discussing Database Refactoring, some of the people in the audience were struggling with the idea he was presenting which I felt was fairly simple. If you really want to skip all that try to forward to the 50 minute mark where he starts talking about sandboxes. What I will say is that if your having difficulty getting Agile accepted into your organisation then this might be a video you want to show your managers since it covers all the major issues and benefits.

Here’s some of the tips he has with regard to testing and improving quality:

  • Do Test Driven Development, the unit tests are the detailed design, they force developers to think about the design. Call it Just-in-time design.
  • Use Continuous Integration to build and run unit tests on each check-in to trunk.
  • Acceptance Tests are primary artefacts. Don’t bother with a requirements document simply maintain the acceptance test since the reality is that all testing teams will do is take that requirement and copy it into an acceptance test, so why introduce a traceability issue when you don’t need it.
  • Use Standards and Guidelines to help ensure teams are creating consistent artefacts.
  • Code Reviews and Inspections are not a best practise. They are used to compensate for people working alone, not sharing their work, not communicating, poor teamwork, and poor collaboration. Guru checks output is an anti-pattern. Working together, pairing, good communication, teamwork should negate the need for code reviews and inspections.
  • Short feedback loop is extremely important. The faster you can get testing results and feedback from stakeholders the better.
  • Testers need to be flexible, willing to pick up new skills, need to be able to work with others. They need to be generalising specialists. The trend that is emerging in agile or the emerging belief is that there is no need for traditional testers.

Scott is a passionate speaker and very convincing, some of the points he makes are quite controversial yet hard to ignore – especially his argument that traditional testers are becoming less necessary. I’m not sure I agree with all his views yet he has succeeded in forcing me to challenge my own views which I need to mull over and for that reason alone watching his talk has been invaluable.