Montana USGS gauges + state-scoped gauges page - #10
Merged
Conversation
Adds docs/PLAN_montana_gauges.md describing a state-scoped gauges page fed from HUC4 1701 ∩ state=MT USGS continuous sites (7-day active cutoff). Three independent landings: extend scripts/fetch_usgs_sites.py for discovery; generate an idempotent 0036_montana_usgs_gauges.sql migration from a hand-reviewed candidate CSV; parameterize _write_gauges_page so it can emit gauges.<state-lower>.html (no new flag needed — the existing is_all_page=False mechanism already suppresses the redundant state filter). USGS-only for this pass: zero sources.yaml changes; fetch-usgs-ogc auto- discovers from gauge.usgs_id. Load delta is +3 OGC calls/hour at the likely MT count, +6 at the worst plausible case (135+ new sites pushing the batch count from 2 to 3). NWPS / NWRFC and reaches deferred. Iterated through five self-review passes against the live codebase before landing: corrected the site-count baseline, caught that all_state_names() returns reach-states only (so the obvious "MT in states" guard would be a bug), pinned the existing is_all_page filter- bar mechanism to a regression test, and added an unparseable-row fallback so USGS station names that don't split cleanly still produce a non-empty sort_name slug. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends scripts/fetch_usgs_sites.py to fetch USGS active stream gauge metadata for Montana alongside the existing OR/WA/ID/NV/CA cluster. The geographic post-filter (lat≥40, lon<-111) is skipped for MT because HUC4 1701 — the Pacific drainage of Montana — extends east of -111° into Glacier NP and the Bob Marshall. The downstream `huc_cd LIKE '1701%'` filter applied by the candidate-pull query is the actual boundary. Phase 1 of docs/PLAN_montana_gauges.md. Next steps (manual): run the script to refresh Gauge-metadata-cache/gauges.db, then pull the candidate CSV via the SQL in the plan's Reproduce block. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Parameterizes _write_gauges_page with an optional `state` abbreviation
kwarg ("MT", "OR", …). When set, rows are filtered to that state,
filename becomes gauges.<state_full>.html, the page title gains a
state suffix, and is_all_page=False threads through to suppress the
redundant single-pill state filter row (existing _build_filter_bar
mechanism, pinned by test_build_filter_bar_omits_state_on_single_state_page).
Function now returns bool — True when a page was written, False when
the state filter produced no rows. This is the signal deploy.py uses
to gate the sitemap entry, since all_state_names() returns reach-states
only and Montana has no reaches yet (so "MT in states" would always
be False and the obvious guard would be a bug).
_emit_sitemap gains an extra_urls kwarg for caller-supplied entries
that aren't part of the default set.
Phase 3 of docs/PLAN_montana_gauges.md. The state filter matches against
the FULL state name on each row (`_apply_gauge_metadata` stores
`_ABBR_TO_STATE[g.state]` in row["state"]) — the kwarg is the postal
abbreviation for API ergonomics, resolved via _ABBR_TO_STATE internally.
Tests:
- test_state_scoped_page_filters_rows pins the row-filter behavior +
the "/gauges.montana.html" canonical + the state-filter suppression.
- test_state_scoped_page_returns_false_when_empty pins the no-rows
short-circuit (no file written, returns False).
- test_all_states_page_unchanged_when_state_unset pins the default
(gauges.html) path through.
- test_build_filters extended with the gauges-bar is_all_page=False
regression mirroring the existing all-pages variant.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 62 USGS continuous gauges in the Pacific drainage portion of Montana (HUC4 1701 ∩ state=MT, 7-day-active) to source/gauge/ gauge_source. fetch-usgs-ogc auto-discovers them via gauge.usgs_id; no sources.yaml change. Idempotent INSERT OR IGNORE / WHERE NOT EXISTS following migration 0027's pattern. scripts/generate_mt_migration.py regenerates the SQL from data/discover/montana_candidates.csv (the discovery output, gitignored via the new data/discover/ rule). 16 of the 62 are flagged with -- REVIEW: comments — Anaconda/Butte Superfund-area monitoring sites (Silver Bow Cr, Warm Springs Cr, Mill Cr, Willow Cr, Lost Cr, Blacktail Cr), the Libby Wetland Site, Lake Como Spillway, the Rock Cr canal diversion, and Mission Cr above reservoir. Trim these by editing the SQL or regenerating after trimming the CSV. Verification on an in-memory sandbox: - 62 source rows (agency='USGS', fetch_url_id=NULL) - 62 gauge rows (state='MT', usgs_id populated) - 62 gauge_source links (no orphan inserts) - zero collisions with existing gauge.name (no OR/WA/ID overlap) Plan-doc fix: state_cd in Gauge-metadata-cache stores FIPS codes (30 = MT), not abbreviations — the reproduce-block SQL was wrong. Phase 2 of docs/PLAN_montana_gauges.md. The gauges.montana.html page materializes after the first fetch-usgs-ogc run populates observations (state-scoped builder from commit fff0bf8 has the internal row-count guard). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`ruff format --check` in CI flagged three files whose function-call wrapping was tighter than ruff prefers at the project's 100-char line limit. Reformatting only — no behavior change. 25/25 tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reuses _write_gauges_page(state=...) — already parameterized by the MT commit — to also emit gauges.oregon.html, gauges.washington.html, and gauges.idaho.html. The all-pages gauges.html call and the row-count guard for empty states are unchanged. Adding more states (CA / NV) is a one-line extension to the tuple in deploy.py when ready. Discoverability via the existing per-state placeholder pages: when a gauge page is successfully written for a state, the corresponding Oregon.html / Washington.html / Idaho.html gets a leading "Live <state> gauge readings (table)" anchor above its external-resource list. The placeholder builder's signature grows a `gauge_state_pages: set[str] | None` kwarg that deploy.py populates from the write-bool returns. Sitemap follows the MT pattern: each successfully-written page is appended to extra_urls; pages that didn't materialize (row-count guard returned False) are silently absent. Tests parameterized across MT / OR / WA / ID for both the row-filter and empty-state-returns-False cases. Three new placeholder-cross-link tests pin the live-data anchor's presence, absence, and back-compat default. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Self-review of the OR/WA/ID extension surfaced the gauges.<state>.html slug rule duplicated across three call sites: gauges.py picks the filename, deploy.py builds the sitemap URL, shell.py emits the cross- link href. A future multi-word state (New Mexico, Rhode Island) would risk drift if one site updated without the others. Extract a tiny ``_state_slug`` helper in ``_shared.py`` and route all three call sites through it. Adds a parameterized direct test covering single-word and multi-word inputs so the contract is pinned. Iteration found no other issues — the suspected JS smoke-test interaction with Oregon.html turned out to be orthogonal (the placeholder builder doesn't emit a filter-bar; the smoke test only runs against a no-reaches setup so Oregon.html isn't generated at all). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The nav-bar Gauge Picker link on state-scoped gauges pages now carries
?state=<full-name> (URL-encoded). gauge_picker.php parses it and pre-
checks only that state's pill — users arriving from gauges.montana.html
land in a Montana-focused picker view and can check other state pills
to expand. The all-pages /gauges.html link stays paramless (active_state
is "" there), preserving the all-checked default.
Falls back to "all checked" when the param is missing, empty, or names
a state with no current observations. Server-side default; no JS change
required (gauge_picker.js's checkedStates() already reads .checked from
the DOM).
shell.py uses urllib.parse.quote for percent-encoding so multi-word
states ("New Mexico" → ?state=New%20Mexico) work cleanly.
Tests (PHPUnit, 4 cases, 13 assertions):
- missing param → all pills checked
- ?state=Montana → only Montana checked, Oregon not
- ?state=Nowhere (invalid) → all pills checked (graceful fallback)
- ?state= (empty) → all pills checked
PHPStan level 8 clean; php-cs-fixer clean; 40/40 Python build tests
still pass; 143/143 PHP tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hand-picked from entries circled by Pat on https://levels-legacy.wkcc.org/?P=Montana.html (2026-05-19). Replaces the auto-discovered HUC4 1701 candidate set (62 sites) that the earlier draft of docs/PLAN_montana_gauges.md envisioned — 10 of the 13 sites are east of the Continental Divide, so the HUC4 1701 boundary no longer applies. mt.list is a tab-separated <row#>, <usgs_site_no>, <human-readable label>. Tooling reads column 2 only; columns 1 and 3 are for human review. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two major changes from the prior draft: 1. Site scope narrowed from HUC4 1701 (62 auto-discovered sites) to the hand-picked list of 13 USGS gauges in montana/mt.list. The HUC4 1701 boundary that justified the discovery sweep no longer applies — 10 of the 13 are east of the Continental Divide (Missouri basin). 2. The Phase-3 gauges.<state>.html builder (merged for OR/WA/ID in fff0bf8 / 7a42220 / db82e3b / 17f9efe / df6581e) is being reverted in favor of the existing fragment-filter mechanism — filters.js already honors #st=<state> on gauges.html and index.html, and the State filter pill is data-driven so Montana auto-appears once Phase 2 lands. State landing pages (Oregon.html, the new Montana.html) become canonical entry points via top-of-page cross-link anchors to the filtered all-states pages and the two pickers. The picker.php HTML entry gets a matching ?state=<full-name> parser that currently only exists on gauge_picker.php (added in df6581e). Nav-bar reach- picker link picks up symmetric pre-fill. _state_slug helper becomes dead code and gets deleted alongside the gauges.<state>.html removal. The plan additionally documents the 16 AW reaches (11 cache-recorded + 5 paddler-curated proxy) that pair with the 13 gauges, including put-in/take-out coordinates and runnable flow ranges from Gauge-metadata-cache. The proxy associations (Big Hole, Madison ×2, Missouri/Great Falls, Sun SF Wilderness) are blocked on MT-boater verification before the reach-import follow-up PR. Plan iterated through five self-review passes; final pass found no new actionable issues. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 of the Montana-gauges plan: switch the generator script from data/discover/montana_candidates.csv (the auto-discovered HUC4 1701 sweep) to montana/mt.list (13 hand-picked sites curated from the legacy site's circled entries). Drop the _REVIEW_HINTS heuristic — the curated list has already been hand-screened. Per-site metadata comes from Gauge-metadata-cache/gauges.db::usgs_site. Migration shrinks from 62 to 13 sites and now spans both Pacific (HUC4 1701) and Missouri (HUC4 1002/1003) drainages — 10 of 13 are east of the Continental Divide. Sanity-checked end-to-end on /Users/pat/tpw/DB/kayak.db sandbox: 13 source rows, 13 gauge rows, 13 gauge_source links land cleanly. DB updates themselves will happen on the live server. See docs/PLAN_montana_gauges.md § Phase 2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…g pages Phase 3 of the Montana-gauges plan. The state-scoped gauges page builder (added in fff0bf8 / 7a42220 for OR/WA/ID) is reverted because the existing fragment-filter machinery (filters.js #st=<state>) already serves the same view. State landing pages (Oregon.html, the new Montana.html, etc.) become canonical entry points via top-of-page cross-link anchors to the filtered all-states views and the pre-filtered pickers. Code: - _shared.py: add "Montana" to _NAV_STATES; delete the _state_slug helper (no consumers remain after the revert). - deploy.py: drop the state-scoped gauges-page loop + extra_sitemap_urls. Switch the landing-page loop and sitemap loop from `states` (reach- states from all_state_names) to sorted(_NAV_STATES) so Montana (gauges-only) still gets a landing page + sitemap entry. - gauges.py: drop the `state` kwarg + filter branch on _write_gauges_page; function returns to its pre-Phase-3 signature. - shell.py: _build_placeholder_page emits 3 or 4 cross-link anchors — "Reaches in {state}" (suppressed when state has no reaches), "Live {state} gauges", "Reach picker — {state}" (suppressed for no-reach states), "Gauge picker — {state}". urllib.parse.quote-encoded. _build_nav iterates sorted(_NAV_STATES) (not the reach-states list), and the Reach Picker link now carries ?state=<active_state> for symmetric pre-fill with the gauge picker. Adds _STATE_LINKS["Montana"] (AW, USGS, NWRFC, MBRFC, USBR, Windy) + a Montana Windy URL. - php/picker.php: add the ?state=<full-name> HTML-entry parser mirroring gauge_picker.php's pattern (df6581e). $primary_state falls back to 'Oregon' when the param is missing or invalid. Tests: - Delete tests/test_build_gauges_state_filter.py (pinned the reverted builder). - Add tests/test_placeholder_state_links.py: 16 cases covering the cross-link anchors on every _NAV_STATES landing page, Montana reach-anchor suppression, nav-bar picker pre-fill (both reach and gauge), and the new MT nav button. - Add tests/php/PickerIntegrationTest.php: parallel of GaugePickerIntegrationTest covering picker.php's new ?state= parser. - tests/test_cli/test_build.py: update test_non_nav_states_excluded (Montana is now in _NAV_STATES; use Wyoming/Utah instead). End-to-end build verified on a sandbox DB: Montana.html lands with only the gauge-side cross-links, Oregon.html has all four anchors, sitemap.xml includes every nav state's landing page, no gauges.<state>.html artifacts generated. 831 pytest + 146 phpunit tests pass; ruff + mypy + phpstan clean. See docs/PLAN_montana_gauges.md § Phase 3. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI failed on the previous push: the /Oregon.html smoke test asserted the page contained an `#filter-bar` element, but Oregon.html is now a curated-bookmarks landing page (after the Phase-3 revert) — no filter bar, no levels table. The page exists and 200s; the assertion shape just no longer matches. Two changes: 1. /Oregon.html test: drop the `#filter-bar` assertion. Replace it with a check for the new top-of-page "Live Oregon gauges" cross-link anchor (`href="https://github.com/gauges.html#st=Oregon"`) — that pins the landing-page builder and the fragment-filter wiring at the same time. Status + no-JS-errors assertions stay. Comment rewritten to describe what Oregon.html actually is. 2. New /gauges.html#st=Oregon test: covers the original test's intent (filter-bar UI renders + filters.js binds event handlers on a state-context page) but against the page that actually has the filter bar. Reads the #st= fragment via filters.js, so it pins both the State-pill rendering and the fragment-filter integration the new architecture relies on. All 8 smoke tests pass locally. The 4 .jpeg screenshots in montana/ remain untracked per the earlier instruction. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 21, 2026
mousebrains
added a commit
that referenced
this pull request
May 23, 2026
Adds a Responses section to docs/REVIEW_gradient_profile.md mapping each finding to the action taken in commit 1ea9a4b plus the verification results. Findings #1, #4, #6 fixed; #3 reworded as readability-only (the "bug" was already correctly handled by the 1m close-enough tolerance, verified by identical sample counts on re-resample); #2/#5/#8/#9 deferred with rationale; #7/#10 confirmed clean. Verification checks #1 and #2 both pass after fixing a stale 0045 docstring and dropping the wall-clock timestamp from the 0046 generator (now byte-stable across regens). Format: one entry per finding, so the next live-host review pass can read top-to-bottom and see exactly what changed vs. what was tabled. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mousebrains
added a commit
that referenced
this pull request
May 29, 2026
The gradient_profile renderer pops a trailing insignificant bar when
the previous bar is significant, so a single bridge/dam DEM artifact
doesn't make the chart "trail off." But it was unconditional on the
trailing bar's width, so any reach ending in slack water (reservoir,
lakeshore, navigation pool) had its flat low-gradient tail absorbed
into the previous narrow significant bar via the first/last edge-
stretch logic — which made the chart imply a drop bigger than the
whole reach's elevation_lost.
Concrete: reach 419 (Canyon Creek into Merwin Reservoir):
bars 1-10 covered 0..2.6 mi, summing ~266 ft of drop (matches the
reach's elevation_lost of 262 ft)
bar 11 covered 2.6..4.2 mi at 0.1 ft/mi (the flat reservoir,
flagged insignificant by build_profile)
prior bar (#10) was 0.2 mi wide at 151 ft/mi
Old renderer popped #11; the first/last edge-stretch logic then
drew bar #10's right edge all the way to take-out (4.375 mi),
visually rendering as 151 ft/mi over ~1.97 mi — i.e. ~285 ft
of drop in a reach with only 262 ft total.
The pop is still wanted for short artifacts (bridges, dams, road
embankments — typically a few hundred metres). Gate on w_mi < 0.5
mi: the algorithm uses max_window_mi=5, so any insignificant tail
above 0.5 mi is genuinely a long flat segment that should render
as such. The underlying gradient_profile JSON keeps both bars
regardless of the renderer choice.
Confirmed on the four current candidates (411, 412, 416, 417, 418, 419);
419 is the visible-on-chart fix; 412 (NF Lewis into Swift Reservoir)
benefits the same way once its trace runs onto Swift's slack water.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mousebrains
added a commit
that referenced
this pull request
Jun 1, 2026
Montana coverage shipped in release 1.1.0 (#10, #13) and the maintainer confirms it's finished — archiving the one item deferred from #109's consistency sweep. - Add an ARCHIVED-COMPLETE banner: the third-revision plan's leftovers were overtaken by events (the state-scoped gauges pages were kept, not reverted; the Phase-2 0036 regeneration is moot under the metadata-single-source redesign — data migrations retired in Phase 6). git mv -> docs/done/, indexed in docs/done/README.md (the test_doc_plans_filed guard requires it). - Fix montana's own `../docs/one-offs/mt.list` links (off by one level after the move into docs/done/) -> `../one-offs/mt.list`; target verified. - test_doc_plans_filed.py: drop the now-stale montana-as-in-progress docstring example (use the landed-but-kept PLAN_production_discipline instead). Frozen historical mentions of the old path (docs/one-offs/ scripts + archived round-5 docs) are intentionally left as point-in-time provenance. Full gate green: ruff + format, pytest 1020 passed. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
0036. Fetched by existingfetch-usgs-ogc(zerosources.yamlchanges).gauges.<state>.htmlstate-scoped variant ofgauges.html. Parameterized_write_gauges_page(state=...)writesgauges.montana.htmlwhen MT gauges have recent obs; returns False (no file written) when empty.is_all_page=Falsemechanism).scripts/fetch_usgs_sites.pynow covers MT (geographic post-filter skipped for MT since HUC 1701 extends east of -111°), andscripts/generate_mt_migration.pyregenerates the migration fromdata/discover/montana_candidates.csv(gitignored).Scope decisions (locked in
docs/PLAN_montana_gauges.md)Load impact
Current OGC budget: 150 sites → 1 batch × 3 params = 3 base calls/hour. Adding ≤135 MT sites pushes us to 2 batches × 3 = +3 calls/hour. NWIS quota is generous (hundreds/hour). Build-time cost: single-digit seconds added. Maintenance overhead is the real cost —
scripts/audit_gauges.pywill fire a "data started" alert per MT gauge on first run (expected, settles after).Pre-merge review needed
16 of the 62 candidates carry
-- REVIEW:comments in the migration — Anaconda/Butte Superfund-area monitoring (Silver Bow Cr, Warm Springs Cr, Mill Cr, Willow Cr, Lost Cr, Blacktail Cr), the Libby Wetland Site, Lake Como Spillway, Rock Cr canal diversion, Mission Cr above reservoir. Decide which to drop; either delete the inserts by hand or trimdata/discover/montana_candidates.csvand re-runscripts/generate_mt_migration.py.Test plan
pytest tests/test_build_gauges_state_filter.py tests/test_build_filters.py— 25/25 pass (new state-scoped builder tests + existing filter-bar regressions).gauges.montana.html(2 had no in-window obs).kayak.dbto confirm the orphan-check is unaffected (sources havefetch_url_id IS NULL— not orphan-checked by design).-- REVIEW:-flagged rows and decide which to trim before merging.kayak-pipeline.timerrun populates obs; secondlevels buildwritesgauges.montana.htmlto the live nginx docroot.Commits
ce6f2d3docs: plan doc (five self-review passes documented in the file)0d1d4b1Phase 1: discovery script Montana supportfff0bf8Phase 3: state-scoped page builder + tests2e18e14Phase 2: migration 0036 + generator🤖 Generated with Claude Code