Adam Christian

Working up to the Windmill 1.0 Beta 1 Release, I finally had the opportunity to put some time into making the IDE (that a lot of you live in when in test writing mode) a little bit nicer to look at.

The IDE has been growing organically since 0.1 and there was a lot of functionality hacked into it that wasn’t in the original game plan, so I did what I could to improve the beauty of the CSS/Layout as well as the whole mess of code behind it.

Launching
If you have used our latest release, or are running trunk you know that we have significantly improved the load times for the Windmill IDE. By compressing the JavaScript when the service is instantiated we can simply hand the IDE window one file that contains the vast majority of the required code.

The reason that this makes such a huge performance difference is because we are loading the source via the local windmill proxy and the data size size had very little impact, the overhead was in the browser two connection limit. When you have to pull down ~30 files two at a time it takes its toll and made the IDE feel very sluggish and more like a web page loading than an IDE.

In the process of figuring out exactly what was slowing down the launch time we added some more informative messages and output so you don’t sit there staring at a twirling circle graphic wondering if anything is happening. And to make the experience even more fun, I couldn’t help but implement a progress bar.

General Layout
I removed the toolbar at the bottom of the screen, which I felt it was an irritation for test editing (especially with the drag and droppable actions). It is now in a drop down menu at the top right of the screen, with the rest of the UI access to IDE functions.

Settings and Firebug Lite Improvements
The settings dialog has continued to improve by implementing more useful defaults, adding new options, removing deprecated options and simply making it just look better. Thanks jQuery UI!

Firebug Lite has been a very popular feature since we first announced it, which has led to a handful of bug fixes over the last month. The most major of these was that the initial Windmill implementation of Firebug Lite required you to have Internet access as it was using resources that were hosted elsewhere.

These have since been copied to our source tree and are made available by the Windmill server so you can happily introspect your Web Apps JavaScript while writing tests on your Intranet.

Output and Performance
Instead of writing all the raw windmill output to the output and performance tabs there is now an array called windmill.errorArr, where all terrible errors and warnings about technical details are pushed in the case you are interested to see all that data. However, it’s more likely that you aren’t and scrolling through all that output data becomes tedious.

This is why we have implemented output in blocks with the background color representing pass/failure with green/red (white for performance). These blocks are expandable, clicking them will reveal all output (or performance information) we know about the action that was executed. This should give you a faster general overview of your results and allow you to quickly see the details you care about.

Other Worthwhile Mentions
We moved our XPath implementation from Ajax-Slt to JS-XPath, which has proven to be more accurate when it comes to resolving XPath generated in Firefox (or using Firebug) against non XPath native browsers such as IE.

Many bugs and improvements have been made to the DOM Explorer, which should now feel a lot more like the Firebug DOM inspector, but should work in any browser.

We have also put a lot of effort into improving the communication between the JavaScript Controller and the Python Service so that when a test fails you get as much detailed information in the service as you do in the IDE.

Timing and MozMill
The timing has lined up nicely as we are working on both a 1.0 release for Windmill and MozMill. MozMill is geared towards automated testing of all applications on the Mozilla Platform and functions in the trusted space providing lots of very useful flexibility.

You can currently try out MozMill 1.0rc1 as a Firefox Add-on, and keep your eyes pealed as some exciting new MozMill feature work is around the corner.

Participate
We are always trying to make life easier for the test writer, so please log your bugs and feel free to come chat with us in #windmill on FreeNode.

Following ‘automation’ and ‘continuous integration’ in the micro blogging world I have seen a major influx in people being super interested in functionally automating their web apps. I have seen a slew of things about Grid, and Selenium, and people hacking on Watir so I decided to show you from the ground up how incredibly easy it is to get automated test running setup using Windmill and Hudson. I am not going to walk you through every detail, this is much more high level but I do plan to start a ‘continuous integration’ page on getwindmill.com in the near future for those kinds of details.

The first step is to get a couple machines that you want use as slaves and a machine to run Hudson, our setup looks like this:

Each of the machines with a different OS has Windmill installed. To make them slaves you simply bring up the Hudson web page on the machine, and run the launcher.. now it’s a slave — crazy easy right?

Now to setup test runs for the machines, in Hudson you click: “New Job” on the left hand side and do something like the following:

Tie this job to the slave you want it to run on (we can’t have IE runs happening on MacOSX):

Tell this job to run 10 and 30 minutes after the hour:

The build steps to actually run the tests, the first kills any straggling processes (more details below):

On the Mac for the Safari job, I want to make sure there aren’t any instances of Safari left hanging, or windmill processes sitting around so we do:
ps -ax | grep windmill | awk '{ print $1 }' | xargs kill | true
ps -ax | grep Safari | awk '{ print $1 }' | xargs kill | true


Then we want to grab the latest test code from svn and launch the windmill test:
svn up /Users/adam/Documents/main_bt/windmill/
python /usr/local/bin/windmill safari http://www.facebook.com test=/Users/adam/Documents/main_bt/windmill/fb email=username@slide.com password=pass report=true exit
rm /Users/adam/Library/Cookies/Cookies.plist


I am telling windmill to run a test against facebook.com, with the test hierarchy in the windmill/fb directory in Safari, with the provided email and password, then to report it’s results and exit.

The only thing different on our windows test runs is the way we kill the processes:
Example:
taskkill /F /T /IM windmill.exe
taskkill /F /T /IM firefox.exe


You might be asking how do I use those variables, check it out in my setup module:

1
2
3
4
5
6

def setup_module(module):
    client = WindmillTestClient(__name__)
    client.type(text=functest.registry['email'], id=u'email')
    client.type(text=functest.registry['password'], id=u'pass')
    client.click(id=u'doquicklogin')
    client.waits.forPageLoad(timeout=u'100000')

You can also read a great entry about adding reporting to your tests on Mikeal Rogers blog, here.

And that last line removing Cookies.plist makes sure that the next test run starts without any cookies set to cause problems.

Have Hudson keep you updated on Jabber:

Grab the generated XML output so you can view the test results in Hudson:

Do this for each of the test runs you would like to have, and boom — continuous integration:

This is obviously a simple scenario, and you can do way, way more customization.. but this should get you off the ground. Happy testing!

Project Status

I have spent nearly every day since July 7th working to bring the Windmill Project up to a level where it can be used reliably in a production environment. Our mission starts with “Windmill is a web testing framework intended for complete automation of user interface testing”, of course this refers to the web including everything and anything inside the browser window. This turns out to be a very large task, one that only an Open Source labor of love could possibly attempt to accomplish.

Windmill has slowly evolved as a project with user contributions, a moderately active IRC channel, and enough users to keep me from forgetting what a useful and powerful tool it is. When I was offered the opportunity to work on the project I quickly saw how much needed to be done in order to get to where we needed to be. We still aren’t quite there, and like most Open Source projects we might not ever get to the envisioned perfection, however recently we hit a very important milestone. The project is now fully hosted and run by the committers, and in many ways “Grown Up”, thanks to a lot of good advise and hard work. The milestone we have reached, is that Windmill is ready for YOU to use. This week we pushed 0.8.2, which is a release that has addressed all of the major issues that we know about and have discovered with heavy usage over the past months. Our hopes are that you will go install Windmill 0.8.2 and things will just WORK. If not, I can’t wait to get your issues in trac and see what we can do to fix them.

Priorities

The main things we care about when it comes to our web testing tools:

• Low barrier to entry, low learning curve, and ease of use
• Thorough documentation, community and project support
• Support for the big 3 platforms; Windows, MacOSX and Linux
• Support for the big 4 browsers; Firefox, IE, Safari and Opera
• Easy integration with continuous integration tools
• Reliability; developers aren’t going to pay attention if the failures aren’t real
• A really nice looking logo, and a web site that is easy on the eyes..

There are always more features to implement, but Windmill hasn’t needed new features for a very long time. What Windmill needed was some serious QA, some code cleanup and a whole mess of bug fixes. If you look through the Trac Timeline you will see the massive amounts of all of the above that have happened and I am proud as hell when I launch the application today and see all that it can do.

What can Windmill do?

Windmill offers the ability to build, write, record and run tests as well as aid in debugging and development. In addition, the framework provides the ability to create and maintain hierarchies of smart and thorough tests that will ensure the quality of your web applications over time. Not only can we save you hours creating and maintaing tests, but we can also help you see your web application as a growing feature rich product, instead of a QA nightmare.

Many tools out there provide ways to write tests, some even provide recorders and DOM explorers, but none that I have ever seen provide this rich functionality cross platform and cross browser, which is really what is required in order to build a thorough test repository that represents all your possible users.

The current set of major features can be found at the Windmill Features Page as well as more details about what is currently available. One of the more exciting new features is the full integration with Firebug Lite. Web developers rely on the existence of Firebug in order to quickly build and debug web applications, and Firebug Lite is the next best thing. It’s hard to even describe how useful it has been to instantly access the JavaScript Console and DOM inspector in IE to debug a failing test. As the Open Source community grows, and tools are improved and brought to light, I think it’s very important to do everything we can to utilize these tools and use them to enhance the Windmill Framework.

Keeping it Open

The Open Source aspect of Windmill has turned out to be it’s greatest asset. The project is almost entirely written in JavaScript and Python, which instantly gives us many advantages over the competition. The JavaScript community is constantly evolving and is most certainly the futures technology platform. Python has a very strong community as well and has given us immense amounts of functionality and flexibility right out of the box.

One of the most exciting things to me personally about this particular project is the immense potential user base out there, and the large impact the Windmill Tools can have on the daily work flow of it’s users. Windmill was obviously inspired with the hopes of minimizing the need for manual testing of rich web applications, and has grown to be much more than that.

The future of the work to be done on Windmill will primarily be driven by the needs of it’s users, the changes and development of the industry and the success of it reaching the goal, to make web automation better.

Moving Forward

Concluding this major push of work, testing, documentation and moving of infrastructure; we now need to see how the community feels. There are lots of choices out there for web automation and we have made many differentiating choices along the way. It is now time to get the word out and take in some real feedback.

Thanks you all for input, contributions, patience and valuable feedback. Those of you who spent many hours on Freenode in #windmill with us debugging and hunting down those spastic blockers are troopers and we really appreciate it.

A large part of the utility in a testing framework like Windmill is the ability to interoperate with a continuous integration environment. Much of the work that has gone into Windmill recently has been the result of continuous integration needs. There are many ways to do this with existing software packages out there that include Tinderbox,Buildbot and Cruise Control however we picked Hudson as a result of the super small learning overhead and amazing simplicity required to setup slaves on the network.

One of the requirements of course for parsing results is the need for JUnit compatible XML output from the Windmill test runs. I don’t claim to be a Python wizard, or a XML/Java wizard for that matter but it wasn’t that painful to hammer out a function to generate some minimal output to get the process off the ground.

I would love to get a wiki page up on Get Windmill to start documenting the many ways to use Windmill in a continuous integration environment. So let me know if you have a working setup and would like to contribute.

Example Reporting Excerpt from __init__.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

from functest import reports
from datetime import datetime
 
class JUnitReport(reports.FunctestReportInterface):
    def summary(self, test_list, totals_dict, stdout_capture):
 
        total_sec = 0
        for entry in test_list:
            time_delta = entry.endtime - entry.starttime
            total_sec += time_delta.seconds 
        out = '<?xml version="1.0" encoding="utf-8"?>\n'
        out += '<testsuite errors="'+str(totals_dict['fail'])+'" failures="'+str(totals_dict['fail'])+'" name="windmill.functional" tests="'+str(len(test_list))+'" time="'+str(total_sec)+'">\n'
 
        for entry in test_list:
            if entry.result is not True:    
                entry_time = entry.endtime - entry.starttime
                out += '<testcase classname="'+entry.__name__+'" name="'+entry.__name__+'" time="'+str(entry_time.seconds)+'.'+str(entry_time.microseconds)+'">\n'
                out += '<failure type="exceptions.AssertionError">\n'
                #out += str(stdout_capture)
                #until I can figure out how to get the traceback
                out += 'There was an error in '+ entry.__name__
                out += '\n</failure>\n'
                out += '</testcase\n>'
            else:
                entry_time = entry.endtime - entry.starttime
                out += '<testcase classname="'+entry.__name__+'" name="'+entry.__name__+'" time="'+str(entry_time.seconds)+'.'+str(entry_time.microseconds)+'"></testcase>\n' 
 
        out += '<system-out><![CDATA[]]></system-out>\n<system-err><![CDATA[]]></system-err>\n'
        out += '</testsuite>'
        f=open('continuous_test.log','w')
        f.write(out);
        f.close()
 
reports.register_reporter(JUnitReport())

Happy automating!