I use the ‘configuration matrix’ option to build a matrix of browsers to run the tests, this way I can have one job that represents a test run on multiple boxes and multiple browsers on each box. Drilling down allows me to see the results for each of these test runs within the job. (Configuration Matrix is awesome btw, except for one really annoying bug for, which there is a reasonable workaround)

Having many build resources, sometimes I want to move a job from one machine to another in order to equalize load on the different machines or to have a job running in ff2 and ie6 instead of ff3 and ie7. Thus the need for a uniform build step that will run the same way regardless of the machine or installed browsers, became necessary. Additionally the commands for running tests got bigger and less manageable all the time, so the time for a clean user interface finally came. This way I can automatically append arguments like ‘exit’ to keep the build step interface simple and clean.

The other simplification is that in the Hudson configuration page for the Windmill Test plug-in you can tell it to automatically call the contrib ‘clean_run.py’ script with the correct arguments (assuming clean_run.py can be found in your path).

As I am relatively new to the Java world, I struggled through the development process but thanks to some reasonable plug-in documentation, responses from the hudson dev mailing list and some IRC conversations with Kohsuke (thanks so much) I finally have something that sufficiently addresses my needs.

Screenshots

Project

The code is all available on GitHub and I welcome any improvements or input from the community as I know that continuous integration is a very important piece of the utility provided by Windmill and this pieces should be as easy as possible to setup.

The combination of this plug-in, and others like the amazon s3 plug-in or the hudson VMware plugin bring us that much closer to seamlessly creating test resources, running our functional web tests across all of our supported platforms and then throwing it away until it’s again needed (or using the CLOUD).I have also done some integration with virtual box that I have found to be very successful.

Download

• Hudson Windmill Plugin 0.4

Please log bugs and let me know what you think! If you are an avid Java and or Hudson plug-in developer and you are interested in contributing, please jump right in!

First off, I have to say that for those of you who don’t know, PyCon is a community organized event, and amazingly well done. I was impressed by the design of the conference, the way they had four talks going on at once and they tried to keep them in a similar interest track. Every talk I attended was at least “good”, and many were “great”. You could really feel a community vibe, and for a conference that had 800+ attendees in the middle of a major recession they had every right to be excited.

There were two major themes content wise that really impressed me, the first was an amazing amount of web framework focus. Django obviously being the twinkle in the eye of the community, but there were smaller communities for each of the other projects, Pylons-Turbo Gears, web2py and lots of tools built on top of them. One that struck me with some major promise is the Pinax Project. Their goal is to make it so that I don’t ever have to deal with building user registration and in site messaging… and all the other features expected for any site that has social network functionality.

The other major theme was a mini testing conference going on within PyCon, that I was very comfortable hanging around with. We had a hugely successful Birds Of a Feather, as well as a surprisingly active Open Space talk for Windmill.

The “Using Windmill” talk turns out to be pretty successful in every aspect that I really care about. I do wish that I had been able to get a little more sleep the night before, and I have to admit the size of the venue was a bit overwhelming. I now realize watching the footage that I used the word “UM” way too much, and the demo videos must have been hard to watch from the very back of the room. But barring those two things, I am quite happy (I shall learn and practice for the next round of shameless PR at OSCON 2009).

The “Functional Testing Tools in Python” panel was very successful, and a lot of fun. I always enjoy the friendly banter between the different project owners. Everyone has a different opinion on what they care about, focus on and feel they do the best. Obviously since the only two projects represented that focused on Web Testing were Windmill and Selenium, we got a lot of attention.

Watching that panel footage I definitely think that the introductions were too long, but I still think our Journey themed – mind blowing – Windmill demo video was a great intro. At the very least, the audience had a little entertainment before the geek droning began

Slide had an awesome presence this year, a fun booth, huge banners everywhere and 6 attendees. It was fun to see all of the great responses I received about Slide from people out there in Python land.

Here are links to the videos:

• Using Windmill: http://pycon.blip.tv/file/1947236/
• Functional Testing Tools in Python Panel: http://pycon.blip.tv/file/1947342/

And some pictures:

If you are interested in seeing the new and improved version of the “Using Windmill” talk, please make it out to OSCON 2009, “Scheduled for 16:30 on 22 Jul 2009.” in San Jose, CA.

We are waiting to hear back from both Open Source Bridge and the AJAX Experience as to whether we will be participating in those conferences (fingers crossed)!

The pain began when I started in on moving our build slaves over to GIT. Of course two of the three are running Windows because we have to run the Windmill tests against IE. In an attempt to keep things simple, I wanted to avoid installing cygwin on the machines, so I tried msysGIT. Oh WAIT I have to get ssh to work before I can actually use GIT to pull down the repo. 

After trying and failing with openSSH, I finally realized that the Windows puTTY Package was the best way to go about this. There was much frustration involved with this process because it requires that you take the ssh key you generated on your mac (and had added on the server) and convert it to a puTTY ppk. Fortunately this turns out not to be that panful using puTTYgen. The next piece to the puzzle was to get pageant to load this key automatically when the machine boots. I went through some rigamarole trying to create a shortcut in the “Startup” items and appending a string to it’s path to get it to load the correct key. This didn’t work, I’m sure it was a combination of Windows being terribly un user friendly and my brain expecting things to “Just Work TM”. Finally I just created a short cut for the actual key and stuck it in the “Startup” folder, and it works! (You still have to enter your pass phrase on boot) but its better than the alternatives. I actually found a post on a puTTY forum where someone was asking to automate this piece too and the response was basically, “No, never, die. Doing that defeats the purpose of SSH”. 

One step that I forgot to mention is that to get GIT to use the right SSH key you have to set the environment variable: GIT_SSH=path/to/putty/plink.exe

Next I went and started my git clone, which kicked off perfectly. That wasn’t so bad! 46% later at 1.99GB of the retrieval process I received two fatal errors that looked similar to file system errors. I then did some Googling to find that msysGIT only supports repository’s up to 2GB. The answer on the forums for this problem was, “have a smaller repo”. This is a problem, so I head back to the GIT homepage, where I find that the only real lasting option is to use cygwin GIT. 

After I get cygwin GIT all installed I finally get the repo fully downloaded I check to see if I can use the msysGIT to pull changes (which should never be 2gb worth), and it absolutely freaks out. So that’s not an option, and neither is adding cygwin/bin to your path because that complains about ‘exec ssh’ not existing.

Your probably wondering, why don’t you just use cygwin to do your GIT stuff? Well the issue is that we are using hudson to queue up jobs on the machines using the cmd environment to pull changes and run the tests in the repo. And it became very clear that the best way to handle this was to run the slave agent from cygwin, that way the jobs are actually running in cygwin land and this turned out to work — awesome!

To finish describing my pain, the last problem was that since I am running the tests from c:\main, (you have to be inside a git repo to pull and I want to keep these jobs simple) I am generating the log file in the wrong place, because the job looks in it’s home directory to find the jUnit Compatible XML results file. Fortunately there are some environment variables that I can use to get the files back where I want; mv *.log %WORKSPACE%

Now that life is back to normal and I’m getting used to living in the new GIT world, I can get back to important things like making Windmill more awesome and increasing test coverage.

Cheers!

The first step is to get a couple machines that you want use as slaves and a machine to run Hudson, our setup looks like this:

Each of the machines with a different OS has Windmill installed. To make them slaves you simply bring up the Hudson web page on the machine, and run the launcher.. now it’s a slave — crazy easy right?

Now to setup test runs for the machines, in Hudson you click: “New Job” on the left hand side and do something like the following:

Tie this job to the slave you want it to run on (we can’t have IE runs happening on MacOSX):

Tell this job to run 10 and 30 minutes after the hour:

The build steps to actually run the tests, the first kills any straggling processes (more details below):

On the Mac for the Safari job, I want to make sure there aren’t any instances of Safari left hanging, or windmill processes sitting around so we do:
ps -ax | grep windmill | awk '{ print $1 }' | xargs kill | true
ps -ax | grep Safari | awk '{ print $1 }' | xargs kill | true


Then we want to grab the latest test code from svn and launch the windmill test:
svn up /Users/adam/Documents/main_bt/windmill/
python /usr/local/bin/windmill safari http://www.facebook.com test=/Users/adam/Documents/main_bt/windmill/fb email=username@slide.com password=pass report=true exit
rm /Users/adam/Library/Cookies/Cookies.plist


I am telling windmill to run a test against facebook.com, with the test hierarchy in the windmill/fb directory in Safari, with the provided email and password, then to report it’s results and exit.

The only thing different on our windows test runs is the way we kill the processes:
Example:
taskkill /F /T /IM windmill.exe
taskkill /F /T /IM firefox.exe


You might be asking how do I use those variables, check it out in my setup module:

1
2
3
4
5
6

def setup_module(module):
    client = WindmillTestClient(__name__)
    client.type(text=functest.registry['email'], id=u'email')
    client.type(text=functest.registry['password'], id=u'pass')
    client.click(id=u'doquicklogin')
    client.waits.forPageLoad(timeout=u'100000')

You can also read a great entry about adding reporting to your tests on Mikeal Rogers blog, here.

And that last line removing Cookies.plist makes sure that the next test run starts without any cookies set to cause problems.

Have Hudson keep you updated on Jabber:

Grab the generated XML output so you can view the test results in Hudson:

Do this for each of the test runs you would like to have, and boom — continuous integration:

This is obviously a simple scenario, and you can do way, way more customization.. but this should get you off the ground. Happy testing!

One of the requirements of course for parsing results is the need for JUnit compatible XML output from the Windmill test runs. I don’t claim to be a Python wizard, or a XML/Java wizard for that matter but it wasn’t that painful to hammer out a function to generate some minimal output to get the process off the ground.

I would love to get a wiki page up on Get Windmill to start documenting the many ways to use Windmill in a continuous integration environment. So let me know if you have a working setup and would like to contribute.

Example Reporting Excerpt from __init__.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

from functest import reports
from datetime import datetime
 
class JUnitReport(reports.FunctestReportInterface):
    def summary(self, test_list, totals_dict, stdout_capture):
 
        total_sec = 0
        for entry in test_list:
            time_delta = entry.endtime - entry.starttime
            total_sec += time_delta.seconds 
        out = '<?xml version="1.0" encoding="utf-8"?>\n'
        out += '<testsuite errors="'+str(totals_dict['fail'])+'" failures="'+str(totals_dict['fail'])+'" name="windmill.functional" tests="'+str(len(test_list))+'" time="'+str(total_sec)+'">\n'
 
        for entry in test_list:
            if entry.result is not True:    
                entry_time = entry.endtime - entry.starttime
                out += '<testcase classname="'+entry.__name__+'" name="'+entry.__name__+'" time="'+str(entry_time.seconds)+'.'+str(entry_time.microseconds)+'">\n'
                out += '<failure type="exceptions.AssertionError">\n'
                #out += str(stdout_capture)
                #until I can figure out how to get the traceback
                out += 'There was an error in '+ entry.__name__
                out += '\n</failure>\n'
                out += '</testcase\n>'
            else:
                entry_time = entry.endtime - entry.starttime
                out += '<testcase classname="'+entry.__name__+'" name="'+entry.__name__+'" time="'+str(entry_time.seconds)+'.'+str(entry_time.microseconds)+'"></testcase>\n' 
 
        out += '<system-out><![CDATA[]]></system-out>\n<system-err><![CDATA[]]></system-err>\n'
        out += '</testsuite>'
        f=open('continuous_test.log','w')
        f.write(out);
        f.close()
 
reports.register_reporter(JUnitReport())

Happy automating!

The schedule for the con can be found here; http://wiki.mozilla.org/Summit2008/Sessions/Schedule

I do have to say that Mozilla is doing a great job at taking care of all their contributors and employees this week, and I feel like a lot of progress is being made on many levels. It’s very useful to see details as to how people are going about this, but it’s even more useful to see things on a bit higher level. It can be easy to get lost in the forest of Open Source, especially the way that Mozilla builds on things organically. 

The sessions are ranging from “The History of the Browser” to Mobile, QA, UE/UI, Thunderbird, planning etc. As Moz has people all over the world I am getting to hear some very interesting accents and interpretations of things. At the dinner last night I was sitting across from people from Germany, Macedonia and Israel. Amazing how the Moz projects can cross the globe and still keep communication open with each of it’s appendages. The keynote from Mitchell Baker and John Lilly gave me a very solid impression of what they feel the company values, where it should be going and most importantly their extreme satisfaction with where it currently is.

Tomorrow I will be doing a session with Mikeal Rogers and Clint Talbert to demo and explain the GristMill XUL testing framework. My piece of this project is called Mozmill, which started out as a straight across port of the Windmill source with the addition of XUL support and has turned into a much more advanced tool that will fill some needs that have existed for a very long time.

I found it interesting to find out that the VP of Engineering Mike Schroepfer announced his departure last week during Oscon. Their have been mixed reactions about this, but many speculations were proven correct today when he announced he was going to facebook, http://blog.mozilla.com/schrep/. The idea of an IPO is can be extremely appealing, apparently they made him an offer he couldn’t refuse.

This morning we were alerted that tons, literally tons of rock had fallen on the highway 99 which turns out is the most convenient of two possible road trips to get back to Vancouver. We are leaving Friday morning, and the new word is that we will all be loaded onto shuttles to embark on the 7 hour trip back to the airport. This will make for an extremely long day and I am not looking forward to it, but the trip was well worth it and I look forward to getting home and picking up where I left off at Slide.

At this point in my career my focus is on Web Development, Test Automation (specifically for the web & browsers), and social networking. Obviously on a moment by moment basis your interests are pulled in varying directions, but that sums up the bulk of my attention.

If you are interested in the full schedule grid, it can be seen here: Oscon 08 Schedule Grid.

Wednesday

• 8:45 AM: Welcome
• 9:30 AM: Keynote
• 10:45 AM: “An Introduction to Ruby Web Frameworks” (It’s going to be tough to convince me to move away from Django)“Changing Education… Open Content, Open Hardware, Open Curricula” looks more interesting today.
• 11:35 AM: This one is tough, either “Web Graphics and Animations without Flash”, “Beautiful Concurrency with Erlang”, or “Beyond REST? Building Data Services with XMPP PubSub”, “CouchDB from 10,000 ft” apparently thats the thing see, or “What Has Ruby Done for You Lately?
• 12:20 PM: Really important, LUNCH!
• 1:45 PM: Probably “Thunderbird 3″, maybe “The Open-Source Identity Revolution”
• 2:35 PM: “Caching and Performance Lessons from Facebook”, never know when this one might come in handy working for Slide inc.
• 4:30 PM: “Open Source Community Antipatterns”, I’m really looking forward to hearing Ted Leung explain how to NOT run an Open Source Project…
• 5:30 PM: Probably “Give your Site a Boost with memcached”, or “Shell Scripting Craftmanship”

As some of you may have heard, today I resigned from my position at Rearden Commerce. Leaving a company is never a fun thing, because you know how you feel when you hear that someone else is leaving.. and you can see it in people’s eyes. I have reminded myself multiple times today that I am still going to be 30 mins away, most of my communication with those people has been via email and IM and there is no reason for me not to stay in touch.

Why did I resign?

That’s a very good question. Let me preface this by saying that I really don’t have anything about Rearden that I can point at and say ‘this thing’ is why I left. Rearden is a great company, they were professional through out my entire experience there. They employ many very talented and driven engineers, and they have a great product. My gut feeling after spending some time there, is that they will do very well. The management team is very skilled and they know their market and niche extremely well. Every day I went to work I heard about a new major deal or a small company Rearden had acquired to contribute to their march toward owning the ‘Personal Assistant’ space.

When I first arrived there I struggled with two things, and ultimately wound up being my demise as an employee. I have an extreme passion for Open Source, being part of that community, and giving my time to contribute. So you are probably thinking, ‘Why didn’t I just do that on the side?’ — well the answer is that I did do it on the side and the results were slow and my sleep schedule paid the price. Rearden has a very business/enterprise specific niche at the moment, and building and deploying new features to those customers is a priority (as it should be), but I couldn’t stop my Open Source envy. 

Secondly, a overwhelming majority of their user base is using IE6. As a web developer — the last thing I do when building anything in client side JavaScript is to test it in IE6. I basically hold my nose, load the page and pray that things ‘mostly work’. Now I’m not going to claim that I can ever get away from doing this, but building really cutting edge features based on new technology becomes significantly less probable when you are catering to this crowd. I know that Rearden has some really cool future plans, and is publicly talking about bringing the application to the consumer market — but I’m impatient and I just simply didn’t want to wait.

What’s Next?

I am going to jump right into a gig with Slide Inc. as a Web Developer. However, before I get to any Web Development tasks I am going to be addressing a pretty serious need they have in their QA department. Slide currently has many applications that are used directly on their site, slide.com and on social networks (primarily facebook.com and myspace.com) and right now they have essentially no functional automation.

At OSAF I saw what a major difference automated testing can make, and the reason I am so excited about this is because I was a QA Engineer at one point manually testing a pretty complex web application (Cosmo) and I have seen how much a difference test automation can make in the release cycle, the development cycle, QA test cycles and simply the daily lives of your poor QA teams.

How am I going to accomplish this task you might ask? Thats the best part — I have fixed about 10 bugs in Windmill in the last week, and will be putting whatever effort is required into getting Windmill to a state where we can functionally automate all of Slides application testing. This looks to be a serious win for Slide, and a serious win for Windmill. 

At some point in the future, when I feel that this project is to the point where it can be maintained and built on by the Slide QA teams I will move on to Web Development tasks. At that point a smaller amount of time will still be allocated to maintaining Windmill, adding new features that Slide and the community need and working towards the next evolution of Windmill. That is quite a ways off in the future, so I will address all that when the time comes.

The rest of my ‘free’ development time, will be consumed by a project that I am involved in with the Mozilla Corporation. This project lives in the QA realm as well, and could probably be classified as a distant cousin to Windmill. More details about that will be announced the week of OSCON, so keep your eyes pealed.

Change can be extremely tough, but it is also very exciting. I want to thank all of my former peers at Rearden for a good experience, and I wish them all the absolute best.