Parallelism of automation tests
- Processes, standards and quality
- Technologies
- Others
Automation tests are a very important aspect of software development, which, I guess, doesn’t have to be additionally explained. Usually, requirements concern a few environments in various combinations browser-system. Also while using the most hated, previous versions of Internet Explorer. Due to that N tests for M differently configured environments should be performed. As a result, we would have a great number of similar, if not the same, actions performed to verify the accuracy of operations. And as it is commonly known, it is possible however highly impractical!
Quick Theory
In general, automation tests that use UI can take a lot of time. The expected result should present a credible feedback on a tested application. The execution time is linked to a several factors concerning, amongst other, aspects:
- preparation of environment,
- initialization of tests (e.g. switching on browser),
- or the very operations/actions that are a part of performing tests.
Issue
Acquiring feedback on a tested application from automation tests can take a lot of time, especially if there is a great number of them. As a consequence tests are usually started at night or even less often.
Solution
There are plenty of different solutions. For example, you can try to choose a certain set of tests, which will be run with every build. On the other hand, you can try to select tests that will check areas of change. There are many different approaches, however each of them has its pros and cons and it is impossible to describe them all as every single approach could be presented in a separate article. Anyway, it is problematic to get answers to questions concerning all required environments, and frequently, we simply replace it with a blind belief that everything is ok. The approach I’m going to present below assumes paralleling tests.
Description:
Selenium Grid 2 – a tool developed by Google providing an appropriate architecture for parallel tests, also responsible for maintaining test on the machine side.
Diagram: Selenium Grid Architecture – modified for the purpose of this article.
As it can be seen in the diagram Selenium Grid, both in the 1st and 2nd version, provides architecture composed of two basic elements – hub and nodes attached to it. Hub provides interface indispensable to communicate with nodes. Nodes are machines dedicated to tests (not only computers, but also smart phones or tablets – all devices, that can be used to start Selenium to run tests) and hub clients, at the same time. They can be controlled in one of the two ways, using Selenium RC or RemoteWebDriver. Communication between basic units of architecture is performed via HTTP with the use of jsons that control commands. Hub stays in contact with the nodes through continuous polling, and in the worst case, the node is marked as disconnected or unavailable. Above mentioned Selenium commands, more precisely their set marked in the diagram as „selenese”, tells Selenium what it should do e.g. search for a button or check text on a given site.
Selenium WebDriver / Selenium 2.0 -relatively well known framework to automate functional tests based on web browser. It was created to replace Selenium RC, which can be used to accomplish the presented approach as well.
MbUnit, Gallio – Gallio Icarus is an open platform for automation that supports parallel testing (essential to perform those tests), which acted as a test runner. It was expanded with the use of MbUnit, a test framework, similar to xUnit or NUnit, by far less popular but rather powerful one.
Example
Below you can find a basic, working configuration. Undoubtedly, for the purpose of the project it will be essential to apply additional settings that define ports, browsers, instances or paths to drivers.
Used tools:
- Selenium Server – version 2.41.0 – including Selenium Grid 2
- Gallio – installer depending on the system (x86/x64), version used was GallioBundle-3.4.14.0-Setup-x64.msi
- MbUnit
- Visual Studio 2013
Installation of a tool and configuration of the project:
- Installing Gallio in a freely selected location.
- Creating new testing walkthrough in Visual Studio.
- Installing trough nuget in a walkthrough:
- Selenium WebDriver
- Selenium WebDriver Support Classes
- Selenium.WebDriver.ChromeDriver (if we plan to run it in Chrome) or Selenium.WebDriver.IeDriver (if we plan to run it in Internet Explorer)
- Adding to walkthrough reference to Mbunit.DLL file and gallio.dll located in Gallio/bin
Configuration of architecture:
The next steps are to configure the environment and implementing it to run Selenium Grid 2. Using the console, it is necessary to implement the following instructions where a * .jar file server is located. Substituting with own values:
- server version,
- IP,
- port (where hub works),
java -jar selenium-server-standalone-2.41.0.jar -host 10.57.160.34 -port 4445 -role hub
This will result in starting hub on a port 4445 (not providing port will start hub on a default 4444 one), available at the IP 10.57.160.34
Next, nodes should be added. After switching to a machine that is supposed to become a node responsible for the tests, the driver supporting the browser should be copied (for cases where Chrome or IE are used). As in the previous case, it is necessary to run the console in a place where a * .jar file is located and follow the instruction to run the host. Substituting parameters with your own values:
- server version,
- path to copied driver,
- IP host,
- port (where host should work)
- hub address
java -jar selenium-server-standalone-2.41.0.jar -Dwebdriver.chrome.driver="D:\\chromedriver.exe" -host 10.57.100.2 -port 5556 -role webdriver -hub http://10.57.160.34:4445/grid/register -browser browserName=chrome,platform=WINDOWS
If other hosts are started on the same physical machine, then they should have different ports. Undefined host port causes it to run on the default 5555 port. Next, the above instructions outlined and defined the role of the test framework (for SeleniumRC there is a parameter node after the „role”). Then the address is defined as a hub to which the node must be registered and the nodes (a machine performing the tests) abilities are defined. The last step is optional, as Selenium Grid itself can determine that it has the ability after registering the node. Those characteristics as well as a number of registered hosts can be check on http://{hub_IP}/grid/console.
Adding the second host is performed similarly, except for the port and a browser which we use.
There are plenty of possible host configurations concerning optional parameters. Depending on defined requirements it is possible to determine browsers, number of instances, version and platform. In order to get more details it is necessary to use the following documentation [link: https://code.google.com/p/selenium/wiki/Grid2].
Tests
Getting back to the project of a test walkthrough in Visual Studio, it is necessary to fill Assembly.cs with three additional lines
using MbUnit.Framework; [assembly:DegreeOfParallelism(2)] [assembly:Parallelizable(TestScope.All)]
where DegreeOfParallelism represents a quantity of test which will be initiated simultaneously by MbUnit – the value should be greater than 0, by default it was assumed that it should be equal to the number of CPUs. In the event that someone tried very hard to set 0 for this attribute, after compiling and loading the dll file in the Test Runner (Gallio Icarus), tests won’t be visible. Theoretically, there is no upper limit (except of the value int) which would restrict the degree of parallelism. However, the very structure of the tests, their order and the number of available nodes in the Selenium Grid will significantly affect the value adapting it to the current conditions.
Exemplary test should look like that:
using System; using MbUnit.Framework; using OpenQA.Selenium.Remote; using OpenQA.Selenium; namespace SampleProject { [TestFixture] [Parallelizable] public class UnitTest1 { private IWebDriver driver; private string baseURL; [SetUp] public void Init() { DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities = DesiredCapabilities.Chrome(); capabilities.SetCapability(CapabilityType.BrowserName, "chrome"); capabilities.SetCapability(CapabilityType.Platform, new Platform(PlatformType.Windows)); driver = new RemoteWebDriver(new Uri("http://10.57.192.122:4445/wd/hub"), capabilities); baseURL = "http://google.pl"; } [TearDown] public void Cleanup() { driver.Close(); driver.Quit(); } [Test] public void TestMethod1() { driver.Navigate().GoToUrl(baseURL + "/"); Thread.Sleep(5000); } } }
This test doesn’t have any assertion, therefore it doesn’t check anything, it simply starts browser, goes over to the site defined by baseUrl and waits 5 seconds.
Next, another test class with tests should be created analogically to the first one. As a result, a walkthrough should contain two test classes. The following step is to compile the project, and then open in Gallio Icarus a generated in this way dll file named after the project. This file is probably located in bin/Debug.
Once the file is loaded correctly in Gallio Icarus tests belonging to two different test classes should be visible on the left hand side, which was distinguished with the help of the hierarchy tree. Pressing the start button will initiate tests.
It is important to remember that the parallelism of tests is carried out on the test classes and not the very tests, therefore the class is marked with the [Parallelizable] attribute.
If all worked out, we can observe on nodes how the test proceeded, and more execution control instructions that appeared in open consoles. Depending on the result it is presented as green or red, as in other test runners.
Summary
A set of Selenium Grid 2, WebDriver, Gallio and Mbunit may very well prove themselves as a distributed automated testing environment and alternative for Jenkins Slave. Once properly configured environment enables to obtain information on the quality of the product throughout the SDLC. Thus, solving the problem of time testing and the diversity of environment in which those tests were run.
Sources:
- Selenium Grid 2: http://code.google.com/p/selenium/wiki/Grid2
- Diagram Selenium Grid: http://grid.selenium.googlecode.com/git-history/22ed3ff910401af083bf06a4d13514f4c6a623ca/src/main/webapp/how_it_works.html
- Selenium Grid 2 download: http://selenium-release.storage.googleapis.com/index.html
- Distributed testing with Selenium Grid: http://www.packtpub.com/sites/default/files/downloads/Distributed_Testing_with_Selenium_Grid.pdf
- More information about WebDriver: http://docs.seleniumhq.org/docs/03_webdriver.jsp
- Gallio and MbUnit download: http://code.google.com/p/mb-unit/downloads/list
- Selenese: http://www.seleniumhq.org/docs/02_selenium_ide.jsp#selenium-commands-selenese