Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ pip install -e ".[dev]" # Install in editable mode with dev deps (p
levels --help # CLI entry point (registered in pyproject.toml)
levels init-db # Create tables and stamp migrations (schema only)
levels migrate # Apply any pending src/kayak/data/db/migrations/*.sql files
levels pipeline # fetch → fetch-usgs-ogc → calc-rating → update-gauge-cache → calculator → build → orphan-check → check-reaches
levels pipeline # fetch → fetch-usgs-ogc → fetch-licor → calc-rating → update-gauge-cache → calculator → build → orphan-check → check-reaches
levels build # Generate static HTML/CSV/text to $OUTPUT_DIR (required)

# Less-common subcommands (see `levels <cmd> --help` for details)
Expand Down Expand Up @@ -180,6 +180,7 @@ Runs these steps in order:

1. **fetch** — reads the active `fetch_url` rows from the DB (synced from the dataset CSVs by `levels sync-metadata`; no longer the engine `sources.yaml` — S1), fetches URLs, dispatches to registered parsers, stores `Observation` rows. A station a feed emits with no `source` row is dropped (known siblings still saved) and flagged per the URL's `unknown_station_policy` (default `reject` → non-zero fetch exit)
2. **fetch-usgs-ogc** — fetches USGS data via the OGC API for gauges linked to a USGS source
2a. **fetch-licor** — POSTs the LI-COR public-dashboard timeseries API for `fetch_url` rows whose parser declares a POST transport (`licor`), storing flow/gauge/temperature; soft step (a LI-COR outage logs + leaves the gauge stale, never blocks the run). The default GET `fetch` skips these rows.
3. **calc-rating** — interpolates missing flow from gage height (or vice versa) using `Rating`/`RatingData` tables
4. **update-gauge-cache** — recomputes gauge-level latest observation values
5. **calculator** — evaluates `CalcExpression` formulas referencing `LatestObservation` values
Expand Down
15 changes: 15 additions & 0 deletions src/kayak/cli/fetch.py
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,21 @@ def _prepare_work_items(
url = args.url_prefix + fu.url
parser_name = args.parser_type or fu.parser

# Non-GET parsers (e.g. licor, transport="POST") ride a dedicated step,
# not the shared async GET client — skip them here so `levels fetch`
# doesn't GET a POST-only endpoint every run. An unknown parser_name
# (parser_cls is None) is left to the existing error path below.
parser_cls = get_parser_class(parser_name) if parser_name else None
transport = getattr(parser_cls, "transport", "GET")
if parser_cls is not None and transport != "GET":
logger.debug(
"Skipping %s (parser %r transport=%s, handled by its own step)",
fu.url,
parser_name,
transport,
)
continue

if args.show_name:
print(f"Processing {url} parser={parser_name}")
else:
Expand Down
Loading
Loading