bbt README

---

• Overview
  • What does the behavior description look like?
  • Partial parsing
  • One more example
• Installation
  • Stable version
  • Latest version
    • building from sources
    • AppImage (Linux only)
• First use
• Why should I use bbt?
  • Write once
  • Write a real documentation, not just a ".feature" Gherkin file
  • Be proficient in no time
  • No-fuss no-wait run
  • Ready to publish output
• Status of the project
• Help and comments
• Further reading

Overview

bbt is a simple tool to black box check the behavior of an executable through Command Line Interface (CLI).
Hence the name: bbt stands for Black Box Tester.

bbt targets both specification of the behavior and end-to-end test automation for the very common case of apps taking some input and producing some output.
It enable developers to write and execute comprehensive test scenarios in just a few minutes.

The outstanding feature of btt is that it directly uses your documentation in plain english.
There is no script nor other file to write.

bbt doesn't care about the type of document: call it Acceptance test, feature or behavior description, test scenario, README file or user guide, that's the same.
bbt make no difference between runing a scenario in a test file and checking an example given in a README file, as long as it recognizes a behavior description in it.

What does the behavior description look like?

Here is a minimal example:

### Scenario: I want to know gcc version

- When I run `gcc --version`
- Then the output contains `14.2.0`

The behavior is described in almost natural English, using the BDD / Gherkin usual pattern Given / When / Then.

It's in Markdown ¹, so that the text above render as:

---

Scenario: I want to know gcc version

• When I run gcc --version
• Then the output contains 14.2.0

---

bbt being about documentation and simplicity, Markdown is a perfect fit.

Let's consider a slightly more complete example:

(Markdown source here)

We have:

1. A Feature and a Scenario header (followed by the feature/scenario name)

   bbt processes only headers starting with Gherkin keywords:

   • # Features
   • # Background
   • # Scenario or # Example

   In this example, the Overview Header is ignored.
   Note also that the header's level doesn't matter (#### Scenario, is equal to # Scenario for bbt), so that you're free to structure the file as you want.

2. Steps

   Within Scenarios, bbt reads Steps, that is lines starting with:

   • - Given
   • - When
   • - Then
   • - And
   • - But

   Those lines contains the things to check or do.
   Note that the only possible list marker for Steps is -, so that other list markers like '*' or '+' may be used for comments and will be ignored by bbt.

3. Step arguments

   Within or after step lines, Step's argument may be:

   • strings for file name, command to run, etc. (for example here config.ini)
   • or multiline text for expected output, file content, etc. (for example here the config.ini file content).

   As per MDG, strings uses Markdown code span (that is a string between backticks), and multiline text uses fenced code blocks (that is a text between two ``` lines).

Everything else in the file is ignored. bbt staying out of the way, you're free to use markdown almost without constraints to draft nice documentations.

Partial parsing

A distinctive feature of bbt is that it seems to directly understand those almost normal English sentences like:

- When I run `sut --quiet input.txt`
- Then there is no output

This is achieved thanks to a partial parser. It means that there is no rigid grammar, because bbt takes into account only some keywords to recognize the skeleton of the sentence.

When you write:

• Then I should get version 15.0.0 (Fix #2398 and #2402)

bbt only sees two keywords and one parameter:

• then get version 15.0.0

And this is what gives the ability to write steps in almost natural language.

One more example

This example shows how simple it is to run a gcc sanity test, that compiles and runs the ubiquitous Hello Word.

Installation

Stable version

is available on Windows, Linux and Darwin thanks to the Alire package manager:

1. Install

2. Install bbt :

   alr install bbt

   The exe should be in ~/.alire/bin.

   Alternatively, you may choose another installation directory with:

   alr install --prefix=/path/to/installation bbt  

   Ensure that the installation directory is in your PATH.

Latest version

building from sources

git clone https://github.com/LionelDraghi/bbt  
cd bbt  
alr build 

AppImage (Linux only)

Download the AppImage here, and:

chmod +x bbt-0.1.0-x86_64.AppImage
ln -s bbt-0.1.0-x86_64.AppImage bbt

(Thanks to @mgrojo and Alr2AppImage).

First use

Start your own scenario file with one of the examples here: https://github.com/LionelDraghi/bbt/tree/main/docs/examples

Then run

bbt create_template

This will create a bbt_template.md file, which is both a template and a quick reference card.
Note : bbt recognize it's own template and doesn't run it, so that you can keep it with your scenarios.

A good example to start simple is given by Simon in it's ada_caser project : there is just a scenarios file called tests.md.

Why should I use bbt?

Write once

Specification is the only source of truth. This is bbt most interesting feature, there is nothing else: no intermediate representation, no glue code, no scripting language, no duplication of the original source at all.

With two main consequences:

1. writing a test is a matter of minutes,
2. there is no more place for a discrepancy between documentation and tests.

Alternative tools exist, some are mentioned in my quick overview of some comparable tools.
But as far as I know, bbt is the only one to provide such a direct "run the doc" approach.

Write a real documentation, not just a ".feature" Gherkin file

bbt effectiveness does not come at the cost of limiting documentation readability or expressiveness:

• First, the vast majority of the file is just plain markdown : use it, structure it the way you like, give as much context as you want, and use all Markdown cool extensions (for example graphics with Mermaid);
• Second, even the part that is interpreted by bbt, the steps, is written in readable English thanks to the partial parsing.

Nice consequence, bbt scenarios may be written by non coders people.

Be proficient in no time

bbt Steps uses a limited English subset, with a vocabulary dedicated to test with no-surprise keywords like run, output, contains, etc.

Although simple, you don't have to learn this subset by heart, you may :

• ask for a template scenario by running bbt create_template (short form bbt ct), or
• ask for the complete grammar with bbt list_grammar (short form bbt lg).

No-fuss no-wait run

To run a scenario : bbt my_scenario.md
To run all the md files in the tests tree bbt -r tests
To run only a selection bbt --select "Sanity check" tests

bbt has no dependencies on external lib or tools (diff, for example), and is tested on Linux, Darwin and Windows.

Ready to publish output

bbt output is in Markdown format. You can adjust the detail level with the usual "-q" and "-v" options.

The output cross-references the executed scenario files: if a test fails, just click on the link and you are in the scenario.
You can push it on GitHub without further processing.

To see what it looks like, consider bbt own tests.

Test results are generated when running bbt, by just using the -o option (--output).

Status of the project

bbt is in an early stage, meaning that interface and behavior are subject to changes.
Feel free to make features suggestions in bbt discussions.

The code has grown fast in 2024, and is far from being clean.
Nevertheless, bbt is working, and has as a serious test base.

A very conclusive test on the effectiveness of bbt as being conducted on the day 4 of Advent of Code 2024's challenges.
Tests where easy and fast to setup, allowing to stay most of the time focus on coding.

In real life, the acc project has largely migrated to BBT, resulting in a drastically reduced number of files and a significant gain in maintenability and readability of the tests.
Other people are using it too.

btt compile on Linux, Windows and Mac OS, and the test suite is run on the three platforms.
On MacOS, it may be useful to set the environment variable GNAT_FILE_NAME_CASE_SENSITIVE to 1, cf. discussion here to avoid small glitches on file names.

Help and comments

Comments are welcome here

Further reading

• User Guide: concepts, command, features...
• Developer Guide: design overview, issues, fixme...
• References: syntax, grammar, and more details on non obvious behavior
• Project status: changelog, tests, TDL...
• Command line help

Footnotes

1. More precisely, bbt comply (mostly) with Markdown with Gherkin (MDG), a convention to embed Gherkin scenarios in GitHub Flavored Markdown files. ↩