db/cache: bulk gauge cache + drop data_db/info_db shims - #7
Merged
Conversation
Replace the per-gauge loop (one update_latest_gauge call per gauge x data_type) with a single window-function CTE that rebuilds the entire latest_gauge_observation cache in ~3 statements. Old path issued ~5x4xN queries for N gauges; new path is O(1) regardless of N. Verified by test_bulk_matches_per_gauge_loop (row-for-row equivalence) and test_bulk_uses_constant_query_count (<=6 statements). Tiebreaker on identical observed_at is source_id DESC, matching the deterministic ordering of the per-gauge implementation.
After T3-20 split data_db.py / info_db.py into observations / cache / gauges / sources / reaches, the old paths remained as re-export facades to avoid touching callers. With every importer now migrated to the canonical modules, the shims (and their drift-guard test) are dead weight. - Delete src/kayak/db/data_db.py, src/kayak/db/info_db.py - Delete tests/test_db/test_module_split.py (drift guard) - Rename tests/test_data_db.py -> tests/test_db/test_observations.py - Rename tests/test_info_db.py -> tests/test_db/test_queries.py - Repoint imports in 8 src + 5 test files to canonical modules No functional change.
mousebrains
added a commit
that referenced
this pull request
May 10, 2026
store_observation() now drops NaN/Inf at INFO. Government feeds occasionally publish sentinel values (-999999) that arithmetic turns into Inf/NaN; without this they poison the latest-observation cache and the merge_sources median. _build_csv() routes string columns through _csv_safe() which prefixes "'" if the value would be interpreted as a formula by Excel/Sheets/ Numbers. Numeric values are emitted via format strings and don't pass through, so legitimate negative deltas remain numbers. Closes pre-launch must-fix #6 and #7. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mousebrains
added a commit
that referenced
this pull request
May 23, 2026
Adds a Responses section to docs/REVIEW_gradient_profile.md mapping each finding to the action taken in commit 1ea9a4b plus the verification results. Findings #1, #4, #6 fixed; #3 reworded as readability-only (the "bug" was already correctly handled by the 1m close-enough tolerance, verified by identical sample counts on re-resample); #2/#5/#8/#9 deferred with rationale; #7/#10 confirmed clean. Verification checks #1 and #2 both pass after fixing a stale 0045 docstring and dropping the wall-clock timestamp from the 0046 generator (now byte-stable across regens). Format: one entry per finding, so the next live-host review pass can read top-to-bottom and see exactly what changed vs. what was tabled. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
mousebrains
added a commit
that referenced
this pull request
Jun 6, 2026
) First slice of the dataset-separation plan's S4a (testability boundary), editable-install only. - tests/fixtures/dataset/: redistribution-safe complete-projection fixture (2 states, 3 gauges across USGS-OGC/URL/calc, 3 real NHD reaches), built + provenance-recorded by tests/fixtures/build_dataset_fixture.py. - kayak.dataset.layout: the shared dataset descriptor (files, columns, types, id-bearing tables, sidecars), derived from the ORM; consumed by the validator, export_metadata, and the fixture builder. - levels validate-dataset <dir>: one authoritative gate — readability, complete projection, typed/length/range/canonical-id values, duplicate PKs, id-counter invariants, reach-name/cross-set integrity, strict gradient-sidecar contract, and a materialized check-reaches run; all crash-safe. - export_metadata quantizes Numeric(9,6) coordinates to 6 dp; the live dataset was normalized to match (kayak_data #6/#7). - Gradient chart made reservoir/lake-aware: the validator is decoupled from reach.length, the renderer (svg_plot.php) and hover/elevation (gradient- profile.js) treat leading/trailing gaps as zero gradient / no-data consistently across all four chart layers. - main.py __main__ guard so `python -m kayak.cli.main` runs the CLI. Deferred: repoint the METADATA_DIR-reading tests at the fixture; S4a-2 (wheel / importlib.resources). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
update_all_latest_gaugesis now O(1) statements. Replaces the per-gauge loop (~5 SQLs x 4 data_types x N gauges) with a single window-function CTE that rebuilds the entirelatest_gauge_observationcache. Verified row-for-row against the legacy per-gauge implementation (test_bulk_matches_per_gauge_loop) and that statement count stays <=6 regardless of gauge count (test_bulk_uses_constant_query_count).data_db.py/info_db.pyre-export shims removed. After T3-20 split them into per-entity modules (observations / cache / gauges / sources / reaches), the old paths lingered as facades to avoid touching callers. Every importer is now repointed at the canonical module; the shims and their drift-guard test (test_module_split.py) are gone. The two test files that exercised the shims have been renamed undertests/test_db/to match their actual scope.Test plan
pytest -q-> 579 passed, 1 deselectedruff check src/ tests/clean;ruff format --checkcleanmypy src/-> Success: no issues found in 48 source filestest_bulk_matches_per_gauge_loopproves bulk SQL produces identical rows to the per-gauge loop across 3 gauges x 2 data typestest_bulk_uses_constant_query_countasserts <=6 statements (down from ~60+ on the old path)🤖 Generated with Claude Code