analytics: bound reverse-DNS to stop kayak-status.service timeouts - #16
Merged
Conversation
kayak-status.service hit its 120s TimeoutStartSec on 2026-05-23: `levels status` rendered in ~2.5s CPU but the process could not exit because reverse-DNS worker threads were stuck in socket.gethostbyaddr (a blocking C call that ignores timeouts) on black-holed PTR queries, and concurrent.futures' atexit hook joins those threads at interpreter exit. Duration had been creeping (31s -> 83s -> >120s) as the unresolvable-IP set grew, since black-holed IPs were never cached and got re-probed nightly. - Resolve PTRs on daemon threads under a wall-clock budget, so stuck lookups are abandoned at the deadline instead of pinning the process. - Drop the rDNS budget 180s -> 45s and raise TimeoutStartSec 120s -> 300s so the budget sits well under the timeout (it was previously above it). - Negative-cache black-holed IPs with exponential backoff (7d, 14d, 28d, 56d, 112d; capped ~26wk); a confirmed result resets the backoff. Cache entry TTL raised to ~26wk to match. - Only penalize IPs a worker actually started; IPs left unpulled when the pool saturates are retried clean next run. Adds tests/test_analytics_rdns.py (10 tests): budget bound, backoff escalation/cap, cache round-trip, legacy/TTL handling, queue saturation, cross-call dedup. Full suite 934 passed; ruff + mypy clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two review follow-ups to the previous commit's rDNS fix: - Set the wall-clock deadline BEFORE starting workers in _resolve_parallel, so the budget actually covers from "now" rather than "once every worker has been kicked off". At 128 threads the drift is a few ms, but the docstring promises wall clock — the math should match. - Clarify what "abandoned at the deadline" means in the docstring. The Python Thread object stops being joined, but the OS thread keeps blocking inside libc's gethostbyaddr until the interpreter exits and the runtime kills it. Safe for the one-shot status render; would leak threads if this resolver were imported into a long-lived process. The earlier wording could have been read as "cancelled". No behavior change. 10 rdns tests still pass; ruff/format clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mousebrains
added a commit
that referenced
this pull request
Jun 9, 2026
…ssion (S2-E2) (#153) * feat(regression): repoint the build + generators at DATASET_DIR/regression (S2-E2) The dataset now carries the validated regression reports (kayak_data #15/#16), but the build still rendered them from the engine's docs/regression/. E2 repoints the read at the dataset so the served reports come from where they're owned + validated. - web/build/deploy.py: _deploy_regression_artifacts reads DATASET_DIR/"regression" (was BASE_DIR/"docs"/"regression"); DATASET_DIR added to the kayak.config import; docstrings updated. The missing-dir early-return ("none configured") is unchanged, so a dataset without regression/ serves no reports rather than failing. - scripts/regression/gauge_pair_linear.py + gauge_lead_lag.py: --out now defaults to $DATASET_DIR/regression/<name>.md when DATASET_DIR is set in the env, via a new _resolve_out() helper using stdlib os.environ only — the "no kayak imports / runnable without the venv" standalone contract is preserved. The generated calc_expression.note points at regression/<name>.md (dataset-relative). README + docstring examples updated to the DATASET_DIR flow. - scripts/wheel-smoke.sh: build now runs with DATASET_DIR=<fixture> (HOME→$WORK and SUDO_USER cleared so config ignores a dev ~/.config/kayak/.env; no-op in CI) and asserts the fixture report renders + is sanitized under /static/regression/ — exercising the full E2 path end-to-end in the packaged wheel. - tests: the 4 test_deploy_regression_* monkeypatch build_mod.DATASET_DIR and seed base/regression/; add _resolve_out unit tests for both generators. - doc sweep: models.py provenance_slug comment, validate_dataset _check_regression docstring, and docs/live-tree-workflow.md frozen-install table all updated — the regression read is now env-located (DATASET_DIR), off BASE_DIR. Cutover-safe: deploy.sh pulls kayak_data (which has regression/) before it builds, so the read finds the same reports it served from docs/regression (byte-identical). The engine's docs/regression/ copy is now unread by code; E3 deletes it. Verified: full gate green; wheel-smoke renders the fixture report from DATASET_DIR/regression; a real build against kayak_data renders all 25 reports (25 html/svg/json), no <script> leak. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(build): warn when regression slugs declared but dataset dir absent Address the S2-E2 review's availability note: the hourly pipeline build doesn't pull kayak_data, so a stale/misconfigured DATASET_DIR checkout (no regression/ directory) would silently publish zero reports even when calc rows declare provenance_slugs. Surface that as a build-time warning, the parallel to validate-dataset's deploy-time slug↔dir check. _build_to_dir counts the non-empty CalcExpression.provenance_slug rows and threads the count through _deploy_source_files → _deploy_static_assets → _deploy_regression_artifacts; the missing-dir early return warns via the extracted _warn_regression_dir_missing helper (keeps the deploy function under the C901 complexity limit). A dataset that simply ships no reports (zero slugs) stays silent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(build): cover regression slug-count computation and three-hop threading Address the re-review's optional residual: the prior test exercised only the leaf (_deploy_regression_artifacts with an explicit count), not that the count is actually computed from the DB or threaded through the deploy hops. Extract the inline provenance_slug count into a _count_regression_slugs(session) helper (also tidies _build_to_dir) and unit-test it — including the NULL and whitespace-only slug edge cases that must not count. Add a threading test that drives the full _deploy_source_files → _deploy_static_assets → _deploy_regression_artifacts chain and asserts the warning fires, closing the untested wiring link between the leaf warn test and _build_to_dir. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
kayak-status.servicetimed out at 03:36 on 2026-05-23 (Result: timeout, killed at the 120sTimeoutStartSec). The render used only ~2.5s CPU but couldn't exit: the "Traffic 24h / per-IP" section does reverse-DNS on every client IP viasocket.gethostbyaddr— a blocking C call that ignores socket timeouts — and on black-holed PTR queries the worker threads stayed stuck. They were non-daemonconcurrent.futuresthreads, which theatexithook joins at interpreter exit, pinning the process. Duration had been creeping up (31s → 83s → >120s) as the unresolvable-IP set grew, because black-holed IPs were never cached and got re-probed every night.Two structural defects: the rDNS budget (180s) was larger than
TimeoutStartSec(120s), and stuck lookups could block process exit.Fix
_resolve_parallel): stuckgethostbyaddrlookups are abandoned at the deadline instead of joined at exit.TimeoutStartSec120s → 300s so the budget sits comfortably under the timeout.Tests
New
tests/test_analytics_rdns.py(10 tests): budget bounds wall time despite a hang, backoff escalation + cap, confirmed-result clears backoff, 4-element negative cache round-trip, legacy/TTL handling, queue-saturation doesn't poison the tail, cross-call dedup. Full suite: 934 passed; ruff + mypy clean.Deploy note
The Python change is live on the editable install for the next
levels status. The unit timeout change needs installing on the live host:Out of scope (flagged)
status.py's systemd section probes everykayak-*unit with two 15s-timeout subprocesses sequentially with no aggregate cap — same failure class, much lower probability. Worth a separate pass if it ever recurs.🤖 Generated with Claude Code