WPF in Visual Studio 2010 – Part 6 : Automated UI Testing

This is the sixth article in the WPF in Visual Studio 2010 series. This week, guest author Phil Price gives us his view of what it took to test the new WPF UI. - Paul Harrington

Background

This post covers an overview of techniques that we used to create and maintain automated user interface regression tests for Visual Studio. Regression tests are a type of software test that, collectively, aim to be an oracle of expected functionality for the target application, run often against new builds of product – they aim to uncover regressions in behavior introduced in a new build.

Pre-Visual Studio 2010 UI Automation

Throughout Visual Studio 2005 and Visual Studio 2008 we built a sizable UI automation framework that abstracted the different facets of the user interface into atomic classes. A concrete example of this would be a class named “ToolWindow” which would have methods such as “ClickClose” which would click the “X” button on the tool window to close it, and “DragTo(x, y)” which would drag a tool window to the specified screen coordinate. Using this framework a vast suite of regression tests was built to exercise the functionality within Visual Studio. Prior to Visual Studio 2010, the UI automation framework would largely use HWND search techniques to find a particular HWND of interest. A trivial example would be to find the status bar and get its current caption*:

// Find visual studio main window
IntPtr visualStudioHwnd = NativeMethods.FindWindowEx(
    IntPtr.Zero, IntPtr.Zero, "wndclass_desked_gsk", null);

// Find the Status Bar
IntPtr statusBar = NativeMethods.FindWindowEx(
    visualStudioHwnd, IntPtr.Zero, "VsStatusBar", null);

// Wraps SendMessage of WM_GETTEXTLENGTH and WM_GETTEXT
string statusBarText = NativeMethods.GetWindowText(statusBar);

* Please note that these examples are designed to illustrate the APIs we use to achieve UI automation. A test case would not do this directly, but go through a framework.

For some controls HWNDs wouldn’t expose enough information. In this case, one could retrieve the MSAA IAccessible from the HWND and party on that. IAccessible is the same interface that screen readers and other assistive technology may use to gather information about an application’s user interface and capabilities. It just so happens that it provides great information for consumption from UI tests too. An example of this would be to click each of the visible top level menu items.

IntPtr dock = NativeMethods.FindWindowEx(
    visualStudioHwnd, IntPtr.Zero, null, "MsoDockTop");
IntPtr mainMenu = NativeMethods.FindWindowEx(
    dock, IntPtr.Zero, null, "Menu Bar");

// Get the IAccessible. 
// Wraps ole32!AccessibleFromWindow 
IAccessible menuBarAccessible = NativeMethods.GetAccessibleFromWindow(
    mainMenu, (uint)NativeMethods.OBJID.CLIENT);

// Wraps ole32!AccessibleChildren
object[] children = NativeMethods.GetAccessibleChildren(menuBarAccessible);

foreach (IAccessible child in children
                .Where(child => child is IAccessible).Cast<IAccessible>())
{
    // Skip invisible children (hidden)
    if ((uint)child.accState == ((uint)child.accState | NativeMethods.STATE_SYSTEM_INVISIBLE))
    {
        continue;
    }

    //// Click each item
    int top, left, width, height = 0;
    child.accLocation(out left, out top, out width, out height, 0);
    NativeMethods.SetCursorPos(left + width / 2, top + height / 2);
    NativeMethods.SendInput(
        NativeMethods.MOUSEINPUT.FromFlags(NativeMethods.MOUSEEVENTF_LEFTDOWN));
    NativeMethods.SendInput(
        NativeMethods.MOUSEINPUT.FromFlags(NativeMethods.MOUSEEVENTF_LEFTUP));
    Thread.Sleep(100);
}

By the end of the Visual Studio 2008 product cycle, HWND searching, various User32 calls and IAccessible were the techniques used to execute our UI automation. It had been successful and we’d shipped two products using it. But, as you’ve probably already guessed, that was about to change.

Intermezzo: Numbers

Before we move on to UI testing for Visual Studio 2010, I’d like to share a few numbers that represent the scale of automated UI regression testing for Visual Studio 2008.

• Visual Studio 2008 had ~65,000 automated UI regression tests at the time of shipping.
• The core UI automation framework (non-Visual Studio specifics) was around ~41,000 lines of code (excluding comments, 175 files)
• The core Visual Studio parts (think tool windows, editors, projects etc.) of the UI automation framework was around ~147,000 lines of code (excluding comments, 763 files)
• On top of the core Visual Studio parts was built an additional layer of product features (Data tools, Debugger, Deployment, DLinq, Test Tools, Office Tools, VSSDK, Smart Devices, Team Foundation, VC, Web Forms, Win Forms and Class Designer) totaling an additional ~400,000 lines of code (excluding comments, 1840 files)

Visual Studio 2010 UI Automation

With the introduction of the WPF shell the previously used technique of zipping around the HWND tree became more difficult, along with calls to APIs like SendMessage and GetWindowLong which were used quite extensively. There are still HWNDs within the WPF shell for some content – but many HWNDs were removed in place of WPF UI elements, and the structure up to the HWND you may care about (for example Solution Explorer) changed. We had built up assumptions about the structure of the HWND tree within the UI automation framework for Visual Studio. These assumptions no longer held true.

So what to do? It’s clear that relying on a HWND structure isn’t going to cut it, but we’d like to continue to use the majority of our old regression tests from the previous release of Visual Studio because it would be very expensive to re-write them all.

Enter Microsoft UI Automation, aka UIA. UIA is, to quote MSDN, “the new accessibility framework for Microsoft Windows, available on all operating systems that support Windows Presentation Foundation (WPF).” UIA and WPF were developed hand in hand (since the first release of WPF). That is to say that the support for WPF UI elements through UIA clients is better than when using MSAA. To enable us to keep our automated UI regression tests, we had to re-write large parts, especially in the navigational portions of the framework that dealt with finding elements, to use UIA instead.

Taking the previous examples, the UIA equivalent would look something like this:

Status Bar example:

var mainWindow = AutomationElement.FromHandle(visualStudioHwnd);

var statusBar = mainWindow.FindFirst(
    TreeScope.Children,
    new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.StatusBar));

// For VS2010 Text for status bar is now within the first child
var firstStatusBarCell = statusBar.FindFirst(TreeScope.Children, Condition.TrueCondition);
string statusBarText = firstStatusBarCell.Current.Name;

Menu item example:

// Get the main menu bar
var menuBar = mainWindow.FindFirst(TreeScope.Children,
    new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.MenuBar));

// Enumerate each child menu item
foreach (AutomationElement child in menuBar.FindAll(
                    TreeScope.Children,
                    new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.MenuItem)))
{
    // Click it 
    Point clickable = child.GetClickablePoint();
    NativeMethods.SetCursorPos((int)clickable.X, (int)clickable.Y);
    NativeMethods.SendInput(
        NativeMethods.MOUSEINPUT.FromFlags(NativeMethods.MOUSEEVENTF_LEFTDOWN));
    NativeMethods.SendInput(
        NativeMethods.MOUSEINPUT.FromFlags(NativeMethods.MOUSEEVENTF_LEFTUP));
    Thread.Sleep(100);
}

It is also worth noting that UIA supports a concept called patterns, which are, to quote MSDN, “a way to categorize and expose a control's functionality independent of the control type or the appearance of the control.” IAccessible supported “DoDefaultAction”, whereas UIA supports many different types of actions. Taking the example above, instead of a mouse click one could get the “ExpandCollapsePattern” and call the “Expand” method to expand each menu item.

foreach (AutomationElement child in ...)
{
    object pattern;
    if (child.TryGetCurrentPattern(ExpandCollapsePattern.Pattern, out pattern))
    {
        ExpandCollapsePattern expandPattern = (ExpandCollapsePattern)pattern;
        expandPattern.Expand();
    }
}

Intermezzo #2: Updated numbers

• Visual Studio 2010 has around ~70,000 automated UI regression tests. This includes tests for new features, new tests for old features.
• The core UI automation framework increased by ~10,000 lines of code to ~51,000 lines of code. Much of this work was shimming the framework onto the same underlying technology used by the Coded UI Test feature that is shipping with Visual Studio 2010 (which supports UIA).
• The core Visual Studio framework increased by ~21,000 lines of code to ~168,000 lines of code. Along with new features, this includes backwardly compatible automation support for the features that changed to use WPF for Visual Studio 2010. Most notably the windowing system, menu/toolbar support and the editor.
• The other Visual Studio parts of the framework increased by ~20,000 lines of code to ~420,000 lines of code – this most mostly new feature development.

Lessons from using UIA

In the process of migrating to the WPF shell we learned many lessons. I’d like to share a few key ones in regards to automated UI testing.

• A “few” tactical re-writes (most notably Tool Windows, Menus/Toolbars and Editor) can go a long way. Having a framework that abstracted our UI was a life saver for shipping 2010. Small teams could iterate quickly on getting their automation support up to par using existing regressions tests as a meter of success.
• There is not really such a thing as an “invisible” or “background” elements in UIA – when an element is gone, it will no longer be in the UIA tree. This is a good thing for assistive technologies (less pruning of the tree required), but can be troublesome for UI Automation frameworks. Historically we would tend to cache UI elements so we could get back to them again faster. When needed, one could call a mix of ShowWindow and accSelect with SELFLAG_TAKEFOCUS. This no longer works. If, say, a tool window is tabbed and not in the foreground; our cached element for it will because invalid, and if it’s requested we need to a) bring it into the foreground and b) find the element from scratch again.
• HWND searching via various user32 API calls is fast. Using IAccessible is reasonably fast. UIA is not as fast. Net effect: running our tests became more expensive as they take more machine time.
• The inverse of the above is true in terms of fidelity; UIA gives the most attractive set of automation interfaces for test case and framework authors.
• UIA has the capability to proxy MSAA and UIA elements and UIA as MSAA elements. However, when the proxy is created some information may be lost in translation. Generally it’s pretty good but there are edge cases.
• The majority of work in UI tests and UI test frameworks is spent finding the thing you want. You need a good story for this; after all a mouse click is always just a mouse click - it doesn’t change. The same is true of sending keyboard input.
• The Win32 HWND class name of WPF’s windows contains the name of AppDomain it was created in and a unique Guid. In other words “wndclass_desked_gsk” became, for example “HwndWrapper[DefaultDomain;; c36a0d13-8c63-499f-b47c-5acecbdcac92]”. Furthermore, the guid changes on every launch. This makes the main window difficult to search for. It turned out using the DTE’s MainWindow.HWnd property was the best way to get the main window handle for Visual Studio.

Tools

If you feel inspired to create automated UI regression tests for your product you may find the following set of tools useful.

• Coded UI Test – New to Visual Studio 2010, a test type that enables record and playback of automation UI tests.
• White – UI automation framework
• UIA Verify – Verifies UIA pattern implementation for an application
• UI Spy – Allows you to observe the UIA tree
• Active Accessibility 2.0 SDK Tools – Contains various tools, one of which, AccExplorer, allows you to observe the IAccessible tree
• Spy++ (installs with Visual Studio) – allows you to observe the HWND tree

Phil Price, Software Design Engineer in Test, Visual Studio Platform

Bio: Phil has been an SDET at Microsoft for 4 years. Previously working on the project system for Visual Studio 2008 and more recently on the WPF shell window management features.

Thanks

A big thank you to Phil for his expert insights. Next time, Part 7 will conclude the WPF in Visual Studio 2010 series with a short round-up of WPF 4 topics that we didn’t have time to cover elsewhere - Paul Harrington

Previous posts in the series:

• Part 1 : Introduction
• Part 2 : Performance
• Part 3 : Focus and Activation
• Part 4 : Direct Hosting of WPF content
• Part 5 : Window Management