Skip to content

Fetch the exoplanet and mass-radius data through fwl-io - #782

Merged
timlichtenberg merged 4 commits into
mainfrom
tl/fwl-io-observe-datasets
Jul 25, 2026
Merged

Fetch the exoplanet and mass-radius data through fwl-io#782
timlichtenberg merged 4 commits into
mainfrom
tl/fwl-io-observe-datasets

Conversation

@timlichtenberg

Copy link
Copy Markdown
Member

Description

Starts moving PROTEUS data provisioning onto fwl-io, the ecosystem data package that MORS already consumes. Relates to #605.

Today a dataset is fetched as a whole Zenodo record with no pinned version, and its files are verified only when a checksum manifest happens to be available. fwl-io pins a dataset to a Zenodo version DOI, verifies every file against a registry committed next to the manifest, and places it in a directory named for the record, so a re-pinned deposit lands beside its predecessor instead of overwriting it.

This is the first of several steps. It moves two datasets, the DACE exoplanet catalogue and the Zeng-2019 mass-radius relations, and leaves everything else on the existing downloader. I picked those two because they are read by one plotting module and carry no physics, so the manifest, the registries, the entry point and the packaging all get exercised before anything a simulation depends on moves.

What is in it:

  • src/proteus/data/: a manifest declaring both datasets with their version DOIs, the two committed registries generated by fwl-io sync, and the helpers that resolve and fetch a declared dataset.
  • The manifest is exposed through the fwl_io.manifests entry point, so fwl-io fetch proteus works without importing PROTEUS.
  • The population diagram resolves its data directory from the pin instead of joining a path, and both datasets keep their warn-and-skip contract when the data is missing or cannot be fetched.
  • The two records are removed from the legacy source map, so each dataset is pinned in exactly one place.
  • src/proteus/data and tests/data are re-included in git: the run-output rule matches a data/ directory at any depth, which would otherwise have left the manifest and its registries untracked.

Two things worth flagging for review:

  • The retry claim. The retry and backoff in fwl-io are not in a release yet, only on main, so against the floor this PR declares (fwl-io>=26.7.22) these two datasets get one attempt at one mirror, where the old path retried three times and could fall back to OSF. That is why the docs here do not claim retry. It is acceptable for two decorative overlays whose absence skips a plot, and it argues for cutting an fwl-io release before anything a run depends on migrates.
  • Users re-download once. The data moves to a new location, so the first run after this fetches both datasets again (about 1 MB). The old copies are not read and can be deleted; the docs say so.

Validation of changes

Tested on macOS with Python 3.12.

  • Fetched both datasets for real into a scratch FWL_DATA: they land in observe/exoplanet_reference/r15727878 and observe/mass_radius/zeng_2019/r15727899, and every file verifies against the committed registry.
  • Built a wheel and inspected it: the manifest and both registries are inside, and the entry point is registered. This is now an integration test rather than a one-off check, because declaring package data does not prove the artifact carries it.
  • Confirmed a failed fetch cannot stop a run: a download failure and an offline tree are both reported and the run continues, and the interior tables that are fetched after the overlays still arrive.
  • Full unit suite passes (2714 passed, 11 skipped), ruff check clean, bash tools/validate_test_structure.sh clean, and the test-quality lint improved (single-assert count 11 to 7).
  • New tests cover the manifest contents and derived locations, the committed checksums, entry-point discovery, the versioned path (pinned literally, with the bare path ruled out), the caller-supplied data root reaching the fetcher, an fwl-io too old to read the manifest being named as the stale side, the one-pin invariant against the legacy map, and the plot skipping when a dataset cannot be resolved at all.

Checklist

  • I have followed the contributing guidelines
  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • My changes generate no new warnings or errors
  • I have checked that the tests still pass on my computer
  • I have updated the docs, as appropriate
  • I have added tests for these changes, as appropriate
  • I have checked that all dependencies have been updated, as required

PROTEUS provisions its reference data with a downloader that fetches whole Zenodo records and verifies them loosely, so a dataset has no pinned version and its files are checked only when a checksum manifest happens to be available. fwl-io already solves this for the ecosystem, and MORS consumes it; this starts PROTEUS on the same path.

Declare the exoplanet catalogue and the Zeng-2019 mass-radius relations in a manifest PROTEUS ships, each pinned to a Zenodo version DOI with a committed registry of file checksums, and expose the manifest through the fwl_io.manifests entry point so fwl-io can fetch them without importing PROTEUS. Both datasets now land in a version directory named for their record, and the population diagram resolves that directory from the pin rather than joining a path by hand.

The two records are removed from the legacy source map so each dataset is pinned in exactly one place, and the retry and checksum behaviour comes from fwl-io rather than from the local downloader. These two datasets are decorative overlays for a run, so they keep their warn-and-skip contract when absent.

Also re-include src/proteus/data and tests/data in git: the run-output rule matches a data/ directory at any depth, which would have left the manifest and its registries untracked.

I picked these two datasets first because they are read by one plotting module and carry no physics, so the manifest, the registries, the entry point, and the packaging all get exercised before anything a simulation depends on moves.
fwl-io reports an unreachable mirror, an offline tree, or an unresolvable data root by raising, and those errors are not the kind the start-of-run data check catches, so a blip while fetching the exoplanet catalogue would abort the simulation. The reference overlays are also fetched before the interior tables a run actually needs, so the abort took those down with it. Both overlays now report a failure and carry on, which is what the downloader they replace did.

Cover the failure paths that had none: a fetch that fails and one that is offline, the interior data still arriving afterwards, and the population plot skipping when a dataset cannot be resolved at all rather than merely being absent.

Describe the two provisioning mechanisms per dataset in the reference table, and say which one the Zenodo API token applies to. Drop the claim that a transient failure is retried: the released fwl-io the floor admits does not do that yet, so for these two datasets a failed fetch means a skipped plot until it does. Point contributors at the manifest for datasets fwl-io provisions, and note that the old copies of both datasets are no longer read.

Also assert that the caller's data root reaches the fetcher, pin the shipped values against a real built wheel rather than against the packaging declaration alone, and name the unversioned-directory branch as the schema tripwire it is.
@timlichtenberg
timlichtenberg requested a review from a team as a code owner July 24, 2026 21:45
@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.05882% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.71%. Comparing base (e2fbd35) to head (9444a6a).

Files with missing lines Patch % Lines
src/proteus/data/__init__.py 94.59% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #782      +/-   ##
==========================================
+ Coverage   92.68%   92.71%   +0.03%     
==========================================
  Files         110      111       +1     
  Lines       15833    15888      +55     
  Branches     2850     2850              
==========================================
+ Hits        14675    14731      +56     
+ Misses       1158     1157       -1     
Flag Coverage Δ
unit-tests 84.01% <97.05%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The page cited Zeng et al. (2016) for curves whose Zenodo record names Zeng et al. (2019) and links the PNAS DOI, which is also where the dataset key `zeng_2019` comes from. Points at the 2019 paper and at the author's table page the record was built from, so the citation, the record and the key all agree.
The floor this pinned had neither a request timeout nor retry, so an unreachable Zenodo did not fail the fetch, it hung it: pooch handed requests no timeout, and a stalled socket blocks the start of a run with no diagnostic and no bound. The decorative-dataset contract only covers a fetch that raises, so nothing caught this. 26.7.25 carries an explicit connect and read timeout on every request and retries a transient failure with backoff, so the worst case is a bounded wait followed by the warn-and-continue path.

The data page now states that behaviour, and states that these two datasets declare Zenodo alone, so an unreachable Zenodo means a failed fetch rather than a fallback to a second mirror. That was true the moment they moved off the legacy downloader, which tried OSF second, and it should be written down rather than discovered.

Also fixes a real disagreement between the two sides of this path. PROTEUS resolved FWL_DATA without expanding a leading '~' while fwl-io expands it, so a home-relative FWL_DATA sent the fetch to a directory literally named '~' below the working directory while the readers looked in the home directory. A test now compares the two resolvers directly instead of asserting either one alone.
@timlichtenberg
timlichtenberg merged commit 8b10dea into main Jul 25, 2026
22 of 23 checks passed
@timlichtenberg
timlichtenberg deleted the tl/fwl-io-observe-datasets branch July 25, 2026 18:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant