Iterations on the Developer Experience for writing specs

[solsson]

Feature branch goal:

• Make specs driven development more attractive than ad-hoc manual testing. If we succeed, backend PRs will include specs that provide knowledge transfer and can be used for automated regression testing.

[solsson]

Next experiment after #26:

• Focus on a CI/CD environment where we want the asserted backends to keep running
• But the specs should exit 0 after X consecutive passes of all specs
• An initial wait of Y seconds before starting specs could be useful to reduce test noise
• A configurable interval between reruns would also be useful to reduce noise

Unlike in dev environments where we aim for one-liners, during CI/CD it's trivial to maintain a set of environment variables for the container.

The reason exit is so important is that with the current generation of kubernetes-assert we're having issues with specs that stay around for very long and thus produce lots of test data and other types of noise/costs.

[solsson]

From the evaluation internally we've observed that:

• It's a clear improvement that specs stop running upon success
  • Specs that produce test data no longer fills our database :)
  • Logs get easier to follow
• Actually it's also common that both dev and CI/CD results in failed specs that keep running, at least until the next working day. There is no improvement for that scenario.
• We tricked ourselves, based on ystack example specs that specs and app should have separate skaffold yamls
  • But they should have the same so that developers:
    • Get one stream of logs to follow
    • Don't need two terminals
    • Are always encouraged to be test driven during skaffold dev

[solsson]

An initial wait of Y seconds before starting specs could be useful to reduce test noise

Relatively low priority.

A configurable interval between reruns would also be useful to reduce noise

Relatively low priority. 10 s is a very good default.

But the specs should exit 0 after X consecutive passes of all specs
... it's also common that both dev and CI/CD results in failed specs that keep running

Can we find criterias and runtime changes so that spec containers always exit?

• If all specs have passed X times we exit 0
  • Should lead to a Completed status in kubectl get pods.
• Evaluate a max time for the specs container, after which it exits non-zero if there are still test failures
  • Can this be cobined with restart policy so that the pod doesn't crash loop but instead we get an Error status.
  • How do we prevent exit when someone is actually developing?
    • Max time countdown should be reset on actual file watch change
    • We can define "actually developing" as editing the specs at least once within the max time interval.

We tricked ourselves, based on ystack example specs that specs and app should have separate skaffold yamls

The examples we run should dev both backend ans specs in the same skaffold dev command.

If we succeed with the above, the y-assert hack in ystack could be replaced with something that basically does kubectl get pods with some selector on specs' labels and checks the pod status.