Skip to content

Improve ci runs#5831

Merged
nickva merged 4 commits intomainfrom
improve-ci-runs
Dec 23, 2025
Merged

Improve ci runs#5831
nickva merged 4 commits intomainfrom
improve-ci-runs

Conversation

@nickva
Copy link
Contributor

@nickva nickva commented Dec 22, 2025

A few improvements to the test suite to help it run better in parallel.

The main improvement is to separate the per-app makefile targets early, during templating phase, so they never share a common app structure during runtime. That lets us avoid reconfiguring the data directory at runtime, which effectively is an almost full restart internally since we're tearing down all the couch servers. So we're saving an extra app shutdown + restart cycle per app with the new scheme and improving the isolation.

The new scheme uses the tmp/$app namespace for each app. After the tests run any unsuccessful tmp/$app directories will be left behind so they can be inspected. This is especially useful for looking at the failed couch.log entries. There is a separate catlogs target to concatenate failed app couch.log files though maybe keeping them separately may be even easier to inspect. Since if app a failed, we'd just rather look at app a's logs and not app c's logs.

Since setup_eunit rebar template now takes an app argument, opted to remove some older eunit targets keeping only the new ones supporting parallel execution. Otherwise, would have to add a special case statement in the plugin for no-app vs one app cases.

Previously, make eunit apps=a,b,c did not run apps a, b and c as separate goals in parallel execution. They were passed in as the a,b,c goal. To fix that parse the apps into individual space separated goals, so they can
be sped up by parallel execution as well.

To keep the apps as isolated from each other as possible make build_eunit_config also create separate etc folders, use separate BUILDDIR macros and each apps only uses its own fixtures. Also use 0-valued ports theeunit template to avoid any chance of port collision and resulting flakiness.

Rename SUBDIRS to EUNIT_SUBDIRS since they used for eunit tests currently only. Also, exclude docs and fauxton apps from the target list, since they are not proper erlang apps and don't have eunit tests.

Newer Erlangs have file:del_dir_r/1 so use that in cleanup_dirs to save a few lines of code.

There are a few other separate commits to update application with a few timeouts I noticed when running with higher concurrency. Another is make the replicator use its own fixture for attachments and not dip into couch app's fixtures folder since those could be setup up or torn down separately and we would rather not add intra-target dependencies during parallel runs.

Let's try running with failfast=false for a bit in Jenkins since it might make it easier to debug failures. Once things settle, or if we see a lot of backlog from failing (doomed run) we can switch back.

@nickva nickva force-pushed the improve-ci-runs branch 7 times, most recently from efd8910 to 3bb5cae Compare December 23, 2025 04:15
The main improvement is to separate the per-app makefile targets early, during
templating phase, so they never share a common app structure during runtime.
That lets us avoid reconfiguring the data directory at runtime, which
effectively is an almost full restart internally since we're tearing down all
the couch servers. So we're saving an extra app shutdown + restart cycle per
app with the new scheme and improving the isolation.

The new scheme uses the `tmp/$app` namespace for each app. After the tests run
any unsuccessful `tmp/$app` directories will be left behind so they can be
inspected. This is especially useful for looking at the failed couch.log
entries. There is a separate catlogs target to concatenate failed app couch.log
files though maybe keeping them separately may be even easier to inspect. Since
if app a failed, we'd just rather look at app a's logs and not app c's logs.

Since setup_eunit rebar template now takes an app argument, opted to remove
some older eunit targets keeping only the new ones supporting parallel
execution. Otherwise, would have to add a special case statement in the plugin
for no-app vs one app cases.

Previously, `make eunit apps=a,b,c` did not run apps `a`, `b` and `c` as
separate goals in parallel execution. They were passed in as the `a,b,c` goal.
To fix that parse the apps into individual space separated goals, so they can
be sped up by parallel execution as well.

To keep the apps as isolated from each other as possible make
build_eunit_config also create separate etc folders, use separate BUILDDIR
macros and each apps only uses its own fixtures. Also use 0-valued ports the
eunit template to avoid any chance of port collision and resulting flakiness.

Rename SUBDIRS to EUNIT_SUBDIRS since they used for eunit tests currently only.
Also, exclude docs and fauxton apps from the target list, since they are not
proper erlang apps and don't have eunit tests.

Newer Erlangs have `file:del_dir_r/1` so use that in cleanup_dirs to save a few
lines of code.
@nickva nickva merged commit 7c4401d into main Dec 23, 2025
49 checks passed
@nickva nickva deleted the improve-ci-runs branch December 23, 2025 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants