NOTE: This tutorial is a little premature, as the software hasn’t been written yet. The tutorial is based on an earlier prototype.

Introduction

Subplot is a tool for automated testing of acceptance criteria. Acceptance tests define if a program (or other system) is accepable, from a user’s or client’s or employer’s point of view. Basically, in a commercial consulting setting, acceptance tests determine if the client is willing to pay the bill for the software having been developed.

Acceptance tests differ from other integration tests by looking at the software only from the user’s point of view, not the developers’. Because of this, Subplot produces two things: a standalone document aimed at the user to describe the accepance tests, and a report from executing the automated tests.

The document is meant to be a vehicle of communication between the various stakeholders in a project, for specifying in detail what the acceptance criteria are. The document both explains the criteria, and how they’re verified, and lets the verification to be done automatically.

Subplot generates the document, and a test program to execute the tests. The generated test program produces the test report.

ERROR: dot exited with 1: "Error: <stdin>: syntax error in line 1 near \'md\'\n"

Example

Here is a quick example of using Subplot. Let’s imagine you are in need of a new implementation of the echo(1) command line utility, and you want to commission the implementation. You need to specify acceptance tests for it. After negotations with the developers, and their project manager, and your own sales and test teams, you end up with the following requirements:

  • If echo is run without arguments, it should write a newline character, and exit with a zero exit code.
  • If echo is run with arguments, it should write each argument, separated by a space character, and then a newline character, and finally exit with a zero exit code.

These are specified for Subplot in a Markdown document. A minimal one for echo might look like this:

Acceptance tests in Subplot are expresses as scenarios, which roughly correspond to use cases and acceptance requirements. A scenario has an overall structure of given/when/then:

  • given some initial context (setup)
  • when some action is taken (the thing to test)
  • then the end result looks in a specific way (verify what happened)

These are embedded in markdown using fenced code blocks marked as containing subplot-scenario text. Subplot extracts these and generates a test program to execute them.

The scenario steps need to be formulated in a way that all stakeholders in the project understand. This is crucial. The scenario steps, and the Markdown document in general, is typically written by the software architect or analyst of the developing organization, in collaboration with the client, and approved by the client.

Subplot is not magic artificial intelligence. Each scenario step needs to be implemented by programmers so that computers also understand them. Each step corresponds to a function, written in a programming language such as Python or Rust, and a YAML file binds the step to the function, using a pattern to capture relevant parts of the step. The pattern can be simple or a full PCRE regular expression. A binding file might look like this:

This means that for a step saying ‘then standard output contains “foo”’, the Python function stdout_is_text is called and given the string foo as an argument.

The developers on the project, in collaboration with the testing team, need to supply the bindings and the Python functions. Subplot produces the code that extracts the interesting parts of scenario steps, and calls your functions in the right order, plus any other scaffolding needed.

Installing Subplot

To install Debian for Debian unstable, add the following to your APT sources lists:

deb http://ci-prod-controller.vm.liw.fi/debian unstable-ci main

Then run the following commands as root:

To run Subplot from git, you need to clone the git repository and need Python 3, and the Markdown parser installed, as well as document typesetting tools, from Debian 10 (buster) or later:

Note that subplot contains sample versions of the files for the echo acceptance tests (echo.md, echo.yaml, echo.py, and echo-prelude.py) so that you don’t need to type them in yourself.

Producing a document

To produce a PDF document of the echo requirements, for you to review, the following commands are needed (prefix the commands with “./” if running from a git checkout):

echo.md is the markdown input file. echo.yaml is the bindings, and its name is inferred by docgen. echo.pdf and echo.html are the formatted documents. The input files are included in the subplot git repository.

Running the tests

To generate the test program, and running it to produce the test report, the following commands are needed:

The output of the last command is the test report:

OK: No arguments
OK: Hello, world

echo.yaml and echo.md are again the bindings and markdown files. Also, echo.py has the Python functions, needed at runtime, though not referenced above. The names of echo.py and echo.yaml are inferred by ftt-codegen.

Exercise

Your mission, should you choose to accept it, is to write a Subplot to write an acceptance test suite for simple uses of the cp(1) command. You can model it after the echo(1) one, and you get to specify the actual acceptance criteria yourself. For the exercise, it’s enough for you to check that a regular file can be copied to another name in the same directory and that the end result is the same. You should try to get docgen produce a PDF of your subplot. You don’t need to run the tests.

Finally

It would great if you could give us, Lars and Daniel, feedback on Subplot and this tutorial.

Questions for before you go through the tutorial (if we are interviewing you):

  • What kinds of testing do you already do?
  • Who are your usual stakeholders?
  • What sorts of things would you like to test better?

Questions for after you go through the tutorial:

  • What do you think of Subplot as a concept?
  • What’s good about Subplot?
  • What’s less good about Subplot?
  • Did you have any problems following the tutorial?
  • Do you see yourself using Subplot in the future?
  • Do you see Subplot as filling a gap?