approach to evaluate dev deploys fitness to advance to production

[elrayle]

Description

Decisions to advance dev deploys to production are evaluated by current integration tests and ad hoc testing on the dev deploys. This issue proposes a systematic way to evaluate the fitness to advance to production.

Proposal

• randomly select 50 coordinates per package manager impacted by the changes (from last 12 months only)
• get current production summary definition for each
• use dev deploys to force generate harvest results and force generate a summary definition
• compare production and dev summary definitions for diffs
• evaluate diffs to determine if they block advancement to production

Factors blocking advancement

• diff is a significant change requiring further evaluation to determine acceptability
• dev summary definition is known to be wrong

Related Work

• existing integration tests using fixtures

[elrayle]

NOTES:

• 50 coordinates is a starting place. Not sure where the sweet spot will be for testing enough coordinates without testing so many the tests take forever to complete.
• Would be nice to have an option to limit the tests to a specific package manager if the changes only impact a single package manager.