Thoughts on Integration tests

[andrewdavidmackenzie]

This is more of a discussion than an issue. If you active GitHub discussions sometime, I can move it there.

I was thinking about how to test my own app, and then when trying to help with #78 I had the same question:

"How to do integration tests of this library (or an app that uses it) when it depends on hardware and radio comms.

One idea would be a full radio simulator/mock! That seems a lot of work for just this, and maybe not even feasible. Maybe something simple over network, IPC or files could be hacked together though. But since that would not be testing crate communications with real firmware in real devices, its value would be limited.

The idea I had was to do something similar to what I did in my pigg (RPi GPIO Gui) project, so here's a short description of it.

PIGG Hardware Tests Example

Project Description

The project consists of two main parts:

• An Iced GUI app that talks to a device that has GPIO over the network
• A device with GPIO that receives commands over the network and outputs IO states back. Various implementations exist in the project
  • A host simulation - this can run on Linux, Windows, MacOS, RPi etc
  • A RPi "host" that can run on linux on an RPi (Pi 3, 4, 5, Pi Zero, Pi Zero 2 etc) and talks to real GPIO HW
  • An embassy embedded firmware that runs on Pi Pico, Pico 2, Pico W, Pico 2 W

I have the usual unit tests in code, and a set of rust integration tests in /tests that are able to start processes on the host machine and test the UI aspects (to some degree) and the interaction between GUI and "Device".

Both of those (via "cargo test") can be run on any supported host. I run them on the different OS in a GitHub Actions matrix.

What's Missing

But still no testing of interaction with real HW, use of the library used to talk to real hardware (rppal) or of the firmware version on Pi Picos...

What I did

I bought a cheap, powerful, efficient Mac Mini as a server that is "always on". €700 I think it was and is a great little machine! I have it connected to second input on my desktop monitors and wireless keyboard/mouse for when I need to interact with it.

I happen to power it off solar panels and energy station to mitigate my guilt on the environment :-)

Test Hardware Setup

I have attached to it's multiple USB ports (for power and data):

• Raspberry Pi Zero
• Pi Pico
• Pi Pico 2 (this also has a breadboard with switches and leds and stuff for manual testing)

So, whenever the server is on, they are powered and attached.

Hardware Tests

I wrote a series of tests in rust a separate crate (hw_tests) that I can invoke manually, or via Makefile.

This test crate is a member of the workspace, but not a default member of the workspace (so not run when "cargo test" or similar is run on the workspace).

These tests use mDNS discovery and USB discovery to find the available devices, and run a series of tests against them, both using the GUI full app (and command line options to specify what to connect to) and also "library" methods provided by the projects to test directly a specific device. The devices are black boxes and I test the network protocol and commands work on all the devices discovered (should be 3).

Discovery is an app feature, which is why I use it (to test it). The device IDs or address expected could be hard coded also.

These tests are not executed when using "cargo test" or my Makefile targets, as other contributors may not have them and they would fail.

Custom GH Actions Runner

It is very easy to setup and configure a custom runner in GitHub actions.

I specified one with the "pigg" label, and I installed and configured such a runner on my Mac Mini.

I have a specific "hw_tests" job in my GitHub Actions CI (that is run on PRs, on Master after merges and daily) that must be run using this runner.

As long as the Mac Mini is powered on, when my CI workflow runs:

• GH Actions will only give that job to a runner with the "pigg" label
• The Custom runner sitting waiting on my Mac Mini will pick it up and run it
• Since the hardware devices are attached to that Mac Mini, and powered, the tests should find them, run and pass

As a bonus, if I need a hardware reset, I can do that remotely with my Energy Station, that can turn AC power off and on again, restarting the Mac Mini and devices.

Since the Customer GH Runner is installed as a system service, it will come up and be ready to take and run jobs

That all sounds complicated, but it's not so bad. The GH customer runner stuff is easy to install, use and configure and has proven reliable.

As long as I have the server powered and the expected hardware devices connected to it, it all runs pretty smoothly.

What about Meshtastic

I am not sure if overall for firmware and other client testing Meshtastic has some customer hardware setup for automatic testing.

If so, maybe that could be leveraged for automated testing of the rust crate against real radio devices and a real mesh?

It would require writing some rust tests that would use the crate as a library to configure, send and receive messages on various radios etc - with some establish conventions for channels and node names etc.

It might be challenging to open this up to automated test of collaborators PRs, but should be possible. Maybe with some manual intervention from maintainers to add labels/commands to PRs to trigger the testing.

If no such existing setup exists, one could be done using:

• One or more dedicated servers (mac or linux or windows)
• GitHub actions custom runners on said machines
• Having one (better two) radios available for the machine to connect to (not used by others), and the tests could take it from there...configuring them (channels, users etc) to a known state and then sending and validating received messages of different types on default or custom channels or directly to the other node.
• With a description of the expected HW connected, even collaborators could reproduce that on their own machines and run the tests manually prior to sending PRs.
• Frequencies, Meshes, Device names could be config options allowing others to setup for their geography or other specifics.

Comments welcome!

[lukipuki]

We should first have a good set of tests that can be easily run on any hardware, even x86 architecture. You can mock/fake the responses of the actual hardware.

Rust's trait system together with #[tokio::test] allows this , but given the low man power for the development, it has never been done. The famous "Test pyramid" (see e.g. https://martinfowler.com/bliki/TestPyramid.html) recommends unit tests. End to end tests like yours are expensive to run and often give false negatives.