We have selected three popular automation tools which can automate web applications in different browsers and in different operating systems. These are based on selenium webdriver API.
We have analysed these automation tools in different categories to identify the advantages and disadvantages of each so that the reader of this document can choose a tool that is best suitable for their needs.
We have compared the three automation tools against different categories which we think are essential for automating a web/mobile application. Below comparison table shows which tool is good at which category.
|Architecture||Protractor is a wrapper around WebdriverJS (selenium-webdriver node module).||WebdriverIO is custom implementation of W3C webdriver JSON wire protocol.||Night Watch is custom implementation of W3C webdriver JSON wire protocol.|
|Supported browsers||Edge, Chrome, Firefox, Safari, Opera, HtmlUnit, PhantomJs||Edge, Chrome, Firefox, Safari, Opera, HtmlUnit, PhantomJs||Edge, Chrome, Firefox, Safari, Opera, HtmlUnit, PhantomJs|
|Angular specific locator strategies||Supported||Partially supported by adding ‘webdriverjs-angular’ node module. Only ‘WaitForAngular’ has been implemented.||Not supported. But still we should be able to automate angular js application as selenium does.|
|Inbuilt test runner||Available||Available||Available|
|Supported testing frameworks||Jasmine, Mocha, Cucumber||Jasmine, Mocha, Cucumber||Mocha, inbuilt framework|
|Reporting||Junit Xml Reporter,
Protractor Html screenshot reporter, Jasmine spec reporter
|Allure reporting, Jasmine Spec reporter, Junit xml reporter, Cucumber reporting||Junit xml reporter,
|Mobile Support (Appium)||Partial support (Supports mobile browsers but does not support native apps)||Android, IOS||Android, IOS|
|Page Object Pattern||Supported||Supported||Supported|
|Retrying flaky tests||Possible||Possible||Possible|
|TypeScript support||Inbuilt support||Possible when using Codecept.js||Not yet supported|
|Additional Features||Angular application support||CSS regression testing using ‘webdrivercss’ npm package.||–|
- Protractor is the only automation tool which has inbuilt support for AngularJS element identification strategies. Better to go with protractor if the application is developed using AngularJS.
- It has support for different unit test frameworks like (Jasmine, Mocha, Cucumber etc..)
- Good reporting mechanism (html, xml etc..)
- Compatible with Continuous integration
- TypeScript support is available which will help Angular 2 developers to write unit tests very easily.
- It does not have support to automate mobile Apps.
- It is implemented as a wrapper to the WebdriverJs. So there is one more layer added in between Selenium server and the Protractor. If there is an issue with WebdriverJs the Protractor team should wait for the WebDriverJs team to fix that issue.
- WebDriverIO is a custom implementation of W3C webdriver API. This gives it an edge to have full control over implementation rather than depending on WebDriverJS implementation.
- It is fully extensible and is written to be as flexible and framework agnostic as possible. It can be applied in any context and serves not only the purpose of testing.
- It has a command line interface ‘wdio’ which makes test configuration as easy and simple as possible so that a non-programmer can configure the setup.
- It has support for most BDD and TDD test frameworks.
- It has good support and enthusiastic developer community and end users which gives it an edge over NightwatchJS.
- It can be used with ‘webdrivercss’ to compare css stylings of an element in the webpage.
- Since it is a custom implementation, it is also a disadvantage as it deviates from generic syntax which may confuse selenium developers coming from other languages.
- It can be used for automating AngularJS apps but it is not as customized as Protractor.
- Similar to WebDriverIO, it is also a custom implementation of W3C webdriver API.
- As all other JS libraries, it is extensible easily – one can write custom commands, assertions extending it.
- It has its own testing framework and assertions mechanism which solves the programmer’s headache of choosing a testing framework.
- It does not have much of choices for unit test frameworks as it has an own testing framework and also support Mocha.
- Slightly lesser support than WebDriverIO and Protractor.