How to Break an Automated Popup Test
There are a set of automated tests used to check XUL menus and popups. These tests check to ensure that popups open and close in response to particular events, that the popups appear in the right location and that highlighting and navigation between menu items and submenus work correctly.
However, these tests tend to be very finicky in terms of when they decide to work properly. Usually they do, but they have a habit of breaking down when even a seemingly unrelated change in made. Compounding the problem is that frequently only one platform or even a specific machine will fail to run the tests properly, making the real issue difficult to diagnose. There are several cases where I’ve just guessed at a fix and got lucky, and several tests which have just been disabled since I can’t reproduce the problem locally. For others, the response when the tests fail is: “Look, Enn broke the tests again”.
Once a popup test fails, other tests will often fail as well. In this case, the usual cause is that a popup or a window is still open because the previous test failed, causing later tests, especially those that fire events, to be unable to work properly.
There are several reasons why the popup tests are so picky:
- Popups cannot opened unless the window is focused. This prevents pages in background tabs, for instance, from displaying popups. However, it also means that the window must have the focus in order for a popup test to succeed. One trap here is that a new window is not focused until after the load event, which means that a popup opened during a load event may not open. Documents loaded in an existing window will be able to open popups within the load event as the existing window is already focused. This can cause problems if you’re not aware of it and don’t check the test in both circumstances, for instance if you just run one test individually versus running through the test suite.
- Tooltips are opened when the mouse stops moving for a few moments and disappear when the mouse is moved again. The tooltip tests synthesize mouse movement events. This means however that while running a tooltip test, a spurious movement of the real mouse can cause additional events to fire, causing the test to fail as the tooltip either opens or closes at the wrong time. In fact, just having the mouse inside the window where the test is running can do this as well. The solution: move the mouse pointer somewhere else.
- Some tests check for keyboard shortcuts on menus. Because the tests run inside a browser window, this can interfere with the shortcuts used by the browser itself, and vice versa. For instance, testing a particular key combination might open a browser menu, or activate a browser command, causing later tests to fail. Tricky to diagnose, as the test that fails might occur somewhat later than the key test. This issue isn’t unique to the menu tests though; any test which emulates the keyboard may have this issue. The keys tested in the current tests are carefully chosen to not conflict in this way, but only in English builds. If you are testing on another language, keyboard shortcuts may be different and you may have mysterious test failures.
- The tests normally run in a little frame. Sometimes this frame isn’t large enough. This can be an issue if you are using a larger font for example. This can cause elements that the test uses, especially those tests that emulate mouse clicks, to be hidden. Since you can’t really click on an element that is scrolled offscreen, or because the coordinates may be different, the test can fail.
- Some tests open a separate window to run, either because they need a top level chrome window, or because they need more space. Having a low screen resolution can cause these windows to open smaller than desired. As well, a lower resolution can cause popups to appear in different locations or smaller than expected, preventing tests from working properly.
- The theme you are using can affect how the test runs. For instance, the addition of a one pixel border to an element can affect calculations, causing coordinates to be off by one.
- Theme issues can also come up due to differences between platforms. One such case I discovered recently was due to a seemingly harmless rule in the default Windows theme:
menubar > menu[_moz-menuactive="true"][open="true"] { border-width: 3px 1px 1px 3px; }This rule causes the border of a menu to change when the open attribute is set to true, which occurs while the popup is being opened. This has the side effect of causing a layout to occur to handle the new border, which in turn causes the popup to be made visible and the popupshown event to be fired. This is fine, but other platforms don’t have this style rule, so the layout to open the popup doesn’t occur until later, so the popupshown event which was being tested for fires at a different point.
These are some of the problems to be aware of when running the popup related tests, and many of these apply to other types of tests as well. If you are having problems with the tests, say on your own machine, (the popup tests are in the toolkit directory and are usually identified with a word like ‘popup’, ‘menu’ or ‘tooltip’ in their name), it may be worth checking if any of these problems apply.