If you are new to the concept of allowlist-testing and just want to try it without too many strings attached, what are your basic tool options?
At retest, we are big fans of allowlist-testing. We honestly think allowlist-testing is the future of testing for testing any interface. For those unfamiliar with the term: Denylist-testing is where typical assertion-based testing denies specified changes and ignores all else. Allowlist-testing guards against all changes, except for the changes you specified as irrelevant, i.e. allowlisted. Other names for that technique are whitelist/blacklist testing (now dismissed for being racist), difference testing, snapshot testing, Golden Master testing, approval testing or characterization testing.
We wanted to come up with an executable demo of a tool with Allowlist testing capabilities. It should allow you to focus on the concept and to play around with it to get some first impression, with the least amount of overhead possible. This demo tool can be used in workshops, conferences, tweeted or otherwise easily shared.
For that, there are 3 simple requirements:
- The tool should be hassle-free and simple to install and play around with, to produce results with as little overhead and as little prerequisites in terms of knowledge and technology as possible.
- The tool should demo the many advantages of allowlist-testing.
- It should be vendor-agnostic, i.e. the tool should ideally be open source and to the least extent commercial.
This list of requirements is short and relatively straight forward, and there are many implementations for allowlist-testing. Yet in my opinion, candidates fulfilling those criteria are hard to come by.
Hassle-free and simple to try Allowlist-testing tools
The tool should be hassle-free and simple to try disqualifies quite a lot of interesting candidates. Most open-source frameworks are test frameworks that are used during development by developers. That means that you need to set up a project environment with some build mechanism for test execution.
The same applies to another great example of an allowlist-testing tool, Approval Tests. Approval Tests is available as a library in almost any platform (Java, C++, NodeJS, Python, etc.). But it mainly focuses on technical interfaces and works great with XML and other technical formats. The barriers to using it are definitely a lot lower than Jest. You only need a test that can be executed in any of the supported platforms, e.g. in Java. It is open-source and free to use, so Approval Tests fulfills both criteria.
Another great tool is TextTest that runs on Python and can be combined with a number of other platforms — Java Swing, SWT, and Tkinter. To set it up, you need to install Python and some more, depending on the platform of your choice. However, it is very much geared towards domain-language-written tests. For a more visual test of e.g. a GUI (custom or Web), it needs some more tooling to drive the GUI. For Python and Java GUIs there is StoryText, which is specially designed to work with TextTest.
recheck-web is an open-source and comes with a Chrome extension that can be used to easily try it. The Chrome extension is simple to install and simple to remove. In order for the Chrome extension to run without any additional setup costs, it sends the data to retest.org. To guard your sensible test data, you need to create a free account before trying it. The detailed results are in a proprietary format, that you need an open-source CLI or a free GUI to open. The GUI comes in a self-contained ZIP file.
Demo the Advantages of Allowlist-testing
Although pixel comparison is a way of Golden Master testing, a tool must implement some mechanism to allowlist changes (i.e. not notifying the user when they occur), in order to also count as a allowlist-testing tool. This is important, so the tool can be used in test automation on a regular basis, without reporting (too many) false positives. Many tools fail at this, such as
In the long term, mere pixel-comparison is of limited value. A good tool should both allow to mass-accept a change, as allowlist-testing often creates redundancies, and should provide convenient ignore options.
Be Open source and not Commercial
The remainder of the tools are commercial, none offer an open-source solution to their testing tool:
So, what do you choose when you want to demonstrate allowlist-testing to a bunch of testers, that have diverse backgrounds (i.e. are used to work on different platforms)? Do you happen to know any other tool option we forgot?