If you are new to the concept of allowlist-testing and just want to try it without too many strings attached, what are your basic tool options?
We wanted to come up with an executable demo of a tool with Allowlist testing capabilities. It should allow you to focus on the concept and to play around with it to get some first impression, with the least amount of overhead possible. This demo tool can be used in workshops, conferences, tweeted or otherwise easily shared.
For that, there are 3 simple requirements:
- The tool should be hassle-free and simple to install and play around with, to produce results with as little overhead and as little prerequisites in terms of knowledge and technology as possible.
- The tool should demo the many advantages of allowlist-testing.
- It should be vendor-agnostic, i.e. the tool should ideally be open source and to the least extent commercial.
This list of requirements is short and relatively straight forward, and there are many implementations for allowlist-testing. Yet in my opinion, candidates fulfilling those criteria are hard to come by.
Hassle-free and simple to try Allowlist-testing tools
The tool should be hassle-free and simple to try disqualifies quite a lot of interesting candidates. Most open-source frameworks are test frameworks that are used during development by developers. That means that you need to set up a project environment with some build mechanism for test execution.
Jest
Being used in over 1.000.000 public repos on GitHub, Jest is a very popular example of allowlist-testing—if not the most popular example. However, it only works with JavaScript in a Node, React or Angular project environment. And setting up yarn or node.js together in a project just to execute a few tests for a short demo is definitely neither simple nor hassle-free.
Approval Tests
The same applies to another great example of an allowlist-testing tool, Approval Tests. Approval Tests is available as a library in almost any platform (Java, C++, NodeJS, Python, etc.). But it mainly focuses on technical interfaces and works great with XML and other technical formats. The barriers to using it are definitely a lot lower than Jest. You only need a test that can be executed in any of the supported platforms, e.g. in Java. It is open-source and free to use, so Approval Tests fulfills both criteria.
TextTest
Another great tool is TextTest that runs on Python and can be combined with a number of other platforms — Java Swing, SWT, and Tkinter. To set it up, you need to install Python and some more, depending on the platform of your choice. However, it is very much geared towards domain-language-written tests. For a more visual test of e.g. a GUI (custom or Web), it needs some more tooling to drive the GUI. For Python and Java GUIs there is StoryText, which is specially designed to work with TextTest.
recheck-web
recheck-web is an open-source and comes with a Chrome extension that can be used to easily try it. The Chrome extension is simple to install and simple to remove. In order for the Chrome extension to run without any additional setup costs, it sends the data to retest.org. To guard your sensible test data, you need to create a free account before trying it. The detailed results are in a proprietary format, that you need an open-source CLI or a free GUI to open. The GUI comes in a self-contained ZIP file.
Demo the Advantages of Allowlist-testing
Although pixel comparison is a way of Golden Master testing, a tool must implement some mechanism to allowlist changes (i.e. not notifying the user when they occur), in order to also count as an allowlist-testing tool. This is important, so the tool can be used in test automation on a regular basis, without reporting (too many) false positives. Many tools fail at this, such as
In the long term, mere pixel-comparison is of limited value. A good tool should both allow to mass-accept a change, as allowlist-testing often creates redundancies, and should provide convenient ignore options.
Be Open source and not Commercial
The remainder of the tools are commercial, none offer an open-source solution to their testing tool:
Summary
So, what do you choose when you want to demonstrate allowlist-testing to a bunch of testers, that have diverse backgrounds (i.e. are used to work on different platforms)? Do you happen to know any other tool option we forgot?