Over the past several years, I’ve noticed the term ‘visual regression testing’ cropping up in web development conversations more and more… and for good reason. It promises to provide tremendous value for those who are tired of manually checking their sites for style issues.
With this growing popularity, it seems like every month there’s a new tool released, each promising to provide awesome visual regression testing automation. Unfortunately, I’ve found most to be more dream than reality, the truth being that these types of tests are deceptively complicated.
There are, however, a few that stand out from the crowd. While each has its benefits and drawbacks, I believe every tool in this list is worth a look.
A quick chart looking at the major features of five of the most popular visual regression tools out there [click the icon to enlarge]
Wraith has been around for several years. It was one of the first tools on the scene, and it continues to see enhancements in functionality, with over 200 pull requests closed in its GitHub repo. It’s the go-to tool for anyone developing in Ruby.
From your project directory, runwraith setup. Then runwraith capture configs/config.yaml. This will run a set of tests on two different versions of the BBC homepage. After completing your test run, check out your screenshot gallery by opening the newly createdshots/gallery.htmlpage in your browser of choice.
You can also customise your tests. In theconfigs/config.yamlfile you can update the sites and pages to test, along with setting screen widths and diff mode.
- YAML configuration file is simple to work with
- You can compare two different URLs
- Installation can be tricky if you’re not familiar with Ruby
- Running interactions specific to individual pages can be difficult
Another veteran option, PhantomCSS has been a popular choice among frontend developers. Its familiarity and functionality make PhantomCSS a great choice for frontend folks looking to stretch their legs in terms of website testing. Built on top of PhantomJS/CasperJS, PhantomCSS adds to the fantastic functionality provided by those two tools.
Then run your file via the CapserJS CLI:casperjs test myfile.js. Your script will run in the background and your images will be saved to thescreenshotsfolder.
- Many tutorials and presentations on it
- Built on top of CasperJS, allowing integration of page actions into tests
- Tests limited to PhantomJS
- No interface for reviewing/managing screenshots
I find Gemini appealing because it packages traditional Selenium testing in an interface that isn’t too complicated. As with PhantomCSS, you can define custom actions around your screengrabs. Unlike PhantomCSS, Gemini goes several steps further by providing ‘test suites’ that can help organise your code.
Create a.gemini.ymlfile with the following contents (replacing the URL to your Selenium grid server):
Then create a test file, and put it into thegeminifolder in the root of your project. The file can be as simple as:
Next, create your baseline images by runninggemini update. You should see one test passed. You can run a regression test viagemini test, which will compare the new images with the ones stored inside thegemini/screensdirectory.
- Selenium integration allows you to test on a wide variety of browsers and devices
- Well-documented site with various examples
- Provides a framework for sorting tests into suites
- No direct access to Selenium, limiting the type of actions you can take on a page
- Running the same test on multiple resolutions requires advanced configuration
First, a caveat: the future of WebdriverCSS is uncertain, as there are efforts to replace it with a more up-to-date module. That said, I think it’s still worth mentioning, as the same basic structure will carry on to the next implementation.
Like Gemini, WebdriverCSS hooks into Selenium for its functionality. It also sits on top of another tool: WebdriverIO. Because it is part of the WebdriverIO ecosystem, it benefits from everything that framework has to provide. This includes hundreds of commands you can send to the browser, before and after taking screenshots.
WebdriverCSS also integrates well with test frameworks like Mocha, Jasmine and Cucumber.js. Add in connectivity with cloud Selenium services like Sauce Labs and Browserstack and you’ve got a full-featured functional test suite with visual regression testing as the cherry on the top.
Run the test by typing this command into your command line:$ node test.js. Validate the images that were created by checking thewebdrivercssfolder. If you run your test again and the screenshots change, you’ll notice adifffolder will be added and the respective diff images will be inside of it.
- Since it uses WebdriverIO, you can take advantage of all the features that framework provides
- Quick configuration of different screen resolutions
- Enables you to hide/mask certain areas of the screenshot
- Future uncertain as this specific plugin is no longer officially maintained
- Requires you to learn how to use WebdriverIO
Unlike the other tools on this list, Spectre doesn’t run tests. Instead, it focuses on providing image comparison capabilities, along with an admin interface for managing screenshots.
In normal use, you’d pair Spectre up with a tool like WebdriverIO or PhantomJS. The latter would do the work of capturing the screenshots, while Spectre would manage storing and processing of images. I’m a fan of this, as the separation of concerns makes it easier as a community to have a testing tool of our choice, but also build out a common admin interface.
On your Spectre website, open up the path/tests/new/. From there you can customise the test information and upload an initial screenshot. After submitting your test, go to the main page of your site where you’ll see a ‘Test Project’ created, along with a notification of 1 passed. You can click on any of the links on that page for further details on the test.
Next, go back to the/tests/newpage and upload a new image. When completed, return to the main page and you’ll see that Spectre ran the comparison and is now reporting a failing test.
- Clean, friendly interface for managing screenshots
- Doesn’t try to do too many things, focusing efforts on a specific area of the testing landscape
- Requires knowledge of Ruby server setup
- Requires you to know how to send data/screenshots via a REST API
There are a few tools that I feel are worth mentioning, even if they didn’t get a featured spot in the list above. They are:
For a comprehensive list of tools and articles on the subject, check outvisualregressiontesting.com.
‘That list is great, but can’t you just tell me which tool to use?’ I hear you ask. Well … no, because it greatly depends on what you’re testing. If you’re checking a fairly simple site, Wraith or Gemini are great options. But if you need to mimic user actions, then WebdriverCSS or PhantomCSS would be a better fit. If you’re unsure, don’t worry. Just pick one and try it out. No matter which tool you choose, you’ll learn valuable lessons about frontend testing and just how powerful (albeit complicated) it can be.