Fetch the exoplanet and mass-radius data through fwl-io - #782
Merged
Conversation
PROTEUS provisions its reference data with a downloader that fetches whole Zenodo records and verifies them loosely, so a dataset has no pinned version and its files are checked only when a checksum manifest happens to be available. fwl-io already solves this for the ecosystem, and MORS consumes it; this starts PROTEUS on the same path. Declare the exoplanet catalogue and the Zeng-2019 mass-radius relations in a manifest PROTEUS ships, each pinned to a Zenodo version DOI with a committed registry of file checksums, and expose the manifest through the fwl_io.manifests entry point so fwl-io can fetch them without importing PROTEUS. Both datasets now land in a version directory named for their record, and the population diagram resolves that directory from the pin rather than joining a path by hand. The two records are removed from the legacy source map so each dataset is pinned in exactly one place, and the retry and checksum behaviour comes from fwl-io rather than from the local downloader. These two datasets are decorative overlays for a run, so they keep their warn-and-skip contract when absent. Also re-include src/proteus/data and tests/data in git: the run-output rule matches a data/ directory at any depth, which would have left the manifest and its registries untracked. I picked these two datasets first because they are read by one plotting module and carry no physics, so the manifest, the registries, the entry point, and the packaging all get exercised before anything a simulation depends on moves.
fwl-io reports an unreachable mirror, an offline tree, or an unresolvable data root by raising, and those errors are not the kind the start-of-run data check catches, so a blip while fetching the exoplanet catalogue would abort the simulation. The reference overlays are also fetched before the interior tables a run actually needs, so the abort took those down with it. Both overlays now report a failure and carry on, which is what the downloader they replace did. Cover the failure paths that had none: a fetch that fails and one that is offline, the interior data still arriving afterwards, and the population plot skipping when a dataset cannot be resolved at all rather than merely being absent. Describe the two provisioning mechanisms per dataset in the reference table, and say which one the Zenodo API token applies to. Drop the claim that a transient failure is retried: the released fwl-io the floor admits does not do that yet, so for these two datasets a failed fetch means a skipped plot until it does. Point contributors at the manifest for datasets fwl-io provisions, and note that the old copies of both datasets are no longer read. Also assert that the caller's data root reaches the fetcher, pin the shipped values against a real built wheel rather than against the packaging declaration alone, and name the unversioned-directory branch as the schema tripwire it is.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #782 +/- ##
==========================================
+ Coverage 92.68% 92.71% +0.03%
==========================================
Files 110 111 +1
Lines 15833 15888 +55
Branches 2850 2850
==========================================
+ Hits 14675 14731 +56
+ Misses 1158 1157 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The page cited Zeng et al. (2016) for curves whose Zenodo record names Zeng et al. (2019) and links the PNAS DOI, which is also where the dataset key `zeng_2019` comes from. Points at the 2019 paper and at the author's table page the record was built from, so the citation, the record and the key all agree.
The floor this pinned had neither a request timeout nor retry, so an unreachable Zenodo did not fail the fetch, it hung it: pooch handed requests no timeout, and a stalled socket blocks the start of a run with no diagnostic and no bound. The decorative-dataset contract only covers a fetch that raises, so nothing caught this. 26.7.25 carries an explicit connect and read timeout on every request and retries a transient failure with backoff, so the worst case is a bounded wait followed by the warn-and-continue path. The data page now states that behaviour, and states that these two datasets declare Zenodo alone, so an unreachable Zenodo means a failed fetch rather than a fallback to a second mirror. That was true the moment they moved off the legacy downloader, which tried OSF second, and it should be written down rather than discovered. Also fixes a real disagreement between the two sides of this path. PROTEUS resolved FWL_DATA without expanding a leading '~' while fwl-io expands it, so a home-relative FWL_DATA sent the fetch to a directory literally named '~' below the working directory while the readers looked in the home directory. A test now compares the two resolvers directly instead of asserting either one alone.
This was referenced Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Starts moving PROTEUS data provisioning onto fwl-io, the ecosystem data package that MORS already consumes. Relates to #605.
Today a dataset is fetched as a whole Zenodo record with no pinned version, and its files are verified only when a checksum manifest happens to be available. fwl-io pins a dataset to a Zenodo version DOI, verifies every file against a registry committed next to the manifest, and places it in a directory named for the record, so a re-pinned deposit lands beside its predecessor instead of overwriting it.
This is the first of several steps. It moves two datasets, the DACE exoplanet catalogue and the Zeng-2019 mass-radius relations, and leaves everything else on the existing downloader. I picked those two because they are read by one plotting module and carry no physics, so the manifest, the registries, the entry point and the packaging all get exercised before anything a simulation depends on moves.
What is in it:
src/proteus/data/: a manifest declaring both datasets with their version DOIs, the two committed registries generated byfwl-io sync, and the helpers that resolve and fetch a declared dataset.fwl_io.manifestsentry point, sofwl-io fetch proteusworks without importing PROTEUS.src/proteus/dataandtests/dataare re-included in git: the run-output rule matches adata/directory at any depth, which would otherwise have left the manifest and its registries untracked.Two things worth flagging for review:
main, so against the floor this PR declares (fwl-io>=26.7.22) these two datasets get one attempt at one mirror, where the old path retried three times and could fall back to OSF. That is why the docs here do not claim retry. It is acceptable for two decorative overlays whose absence skips a plot, and it argues for cutting an fwl-io release before anything a run depends on migrates.Validation of changes
Tested on macOS with Python 3.12.
FWL_DATA: they land inobserve/exoplanet_reference/r15727878andobserve/mass_radius/zeng_2019/r15727899, and every file verifies against the committed registry.ruff checkclean,bash tools/validate_test_structure.shclean, and the test-quality lint improved (single-assert count 11 to 7).Checklist