Adam Christian

Following ‘automation’ and ‘continuous integration’ in the micro blogging world I have seen a major influx in people being super interested in functionally automating their web apps. I have seen a slew of things about Grid, and Selenium, and people hacking on Watir so I decided to show you from the ground up how incredibly easy it is to get automated test running setup using Windmill and Hudson. I am not going to walk you through every detail, this is much more high level but I do plan to start a ‘continuous integration’ page on getwindmill.com in the near future for those kinds of details.

The first step is to get a couple machines that you want use as slaves and a machine to run Hudson, our setup looks like this:

Each of the machines with a different OS has Windmill installed. To make them slaves you simply bring up the Hudson web page on the machine, and run the launcher.. now it’s a slave — crazy easy right?

Now to setup test runs for the machines, in Hudson you click: “New Job” on the left hand side and do something like the following:

Tie this job to the slave you want it to run on (we can’t have IE runs happening on MacOSX):

Tell this job to run 10 and 30 minutes after the hour:

The build steps to actually run the tests, the first kills any straggling processes (more details below):

On the Mac for the Safari job, I want to make sure there aren’t any instances of Safari left hanging, or windmill processes sitting around so we do:
ps -ax | grep windmill | awk '{ print $1 }' | xargs kill | true
ps -ax | grep Safari | awk '{ print $1 }' | xargs kill | true


Then we want to grab the latest test code from svn and launch the windmill test:
svn up /Users/adam/Documents/main_bt/windmill/
python /usr/local/bin/windmill safari http://www.facebook.com test=/Users/adam/Documents/main_bt/windmill/fb email=username@slide.com password=pass report=true exit
rm /Users/adam/Library/Cookies/Cookies.plist


I am telling windmill to run a test against facebook.com, with the test hierarchy in the windmill/fb directory in Safari, with the provided email and password, then to report it’s results and exit.

The only thing different on our windows test runs is the way we kill the processes:
Example:
taskkill /F /T /IM windmill.exe
taskkill /F /T /IM firefox.exe


You might be asking how do I use those variables, check it out in my setup module:

1
2
3
4
5
6

def setup_module(module):
    client = WindmillTestClient(__name__)
    client.type(text=functest.registry['email'], id=u'email')
    client.type(text=functest.registry['password'], id=u'pass')
    client.click(id=u'doquicklogin')
    client.waits.forPageLoad(timeout=u'100000')

You can also read a great entry about adding reporting to your tests on Mikeal Rogers blog, here.

And that last line removing Cookies.plist makes sure that the next test run starts without any cookies set to cause problems.

Have Hudson keep you updated on Jabber:

Grab the generated XML output so you can view the test results in Hudson:

Do this for each of the test runs you would like to have, and boom — continuous integration:

This is obviously a simple scenario, and you can do way, way more customization.. but this should get you off the ground. Happy testing!

A large part of the utility in a testing framework like Windmill is the ability to interoperate with a continuous integration environment. Much of the work that has gone into Windmill recently has been the result of continuous integration needs. There are many ways to do this with existing software packages out there that include Tinderbox,Buildbot and Cruise Control however we picked Hudson as a result of the super small learning overhead and amazing simplicity required to setup slaves on the network.

One of the requirements of course for parsing results is the need for JUnit compatible XML output from the Windmill test runs. I don’t claim to be a Python wizard, or a XML/Java wizard for that matter but it wasn’t that painful to hammer out a function to generate some minimal output to get the process off the ground.

I would love to get a wiki page up on Get Windmill to start documenting the many ways to use Windmill in a continuous integration environment. So let me know if you have a working setup and would like to contribute.

Example Reporting Excerpt from __init__.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

from functest import reports
from datetime import datetime
 
class JUnitReport(reports.FunctestReportInterface):
    def summary(self, test_list, totals_dict, stdout_capture):
 
        total_sec = 0
        for entry in test_list:
            time_delta = entry.endtime - entry.starttime
            total_sec += time_delta.seconds 
        out = '<?xml version="1.0" encoding="utf-8"?>\n'
        out += '<testsuite errors="'+str(totals_dict['fail'])+'" failures="'+str(totals_dict['fail'])+'" name="windmill.functional" tests="'+str(len(test_list))+'" time="'+str(total_sec)+'">\n'
 
        for entry in test_list:
            if entry.result is not True:    
                entry_time = entry.endtime - entry.starttime
                out += '<testcase classname="'+entry.__name__+'" name="'+entry.__name__+'" time="'+str(entry_time.seconds)+'.'+str(entry_time.microseconds)+'">\n'
                out += '<failure type="exceptions.AssertionError">\n'
                #out += str(stdout_capture)
                #until I can figure out how to get the traceback
                out += 'There was an error in '+ entry.__name__
                out += '\n</failure>\n'
                out += '</testcase\n>'
            else:
                entry_time = entry.endtime - entry.starttime
                out += '<testcase classname="'+entry.__name__+'" name="'+entry.__name__+'" time="'+str(entry_time.seconds)+'.'+str(entry_time.microseconds)+'"></testcase>\n' 
 
        out += '<system-out><![CDATA[]]></system-out>\n<system-err><![CDATA[]]></system-err>\n'
        out += '</testsuite>'
        f=open('continuous_test.log','w')
        f.write(out);
        f.close()
 
reports.register_reporter(JUnitReport())

Happy automating!