[CUDA] Honor offline_cache=False end-to-end so QD_OFFLINE_CACHE=0 actually gives a cold compile by duburcqa · Pull Request #580 · Genesis-Embodied-AI/quadrants

duburcqa · 2026-04-27T21:20:46Z

Honor offline_cache=False end-to-end on the CUDA backend (plus a `qd.init` user-guide page)

QD_OFFLINE_CACHE=0 is supposed to disable on-disk kernel caching so users can take cold-compile measurements. On CUDA today it does not: two layers of caching survive the flag and silently serve cached compilation across runs. This PR closes both leaks. It also adds a new init_options.md user-guide page documenting the commonly-tuned qd.init(...) knobs (offline_cache, cfg_optimization, fast_math, num_compile_threads, debug, check_out_of_bound) - subsumes #581 which was reduced to docs-only after cuda_jit_opt_level was dropped (no measurable benefit on the integration profile).

TL;DR

PtxCache::get_cache_mode hard-coded MemAndDiskCache and never consulted compile_config.offline_cache. Fixed: with offline_cache=False, the constructor skips loading prior metadata, dump() returns without writing, and store_ptx tags entries MemCache so the dump filter excludes them. In-process memory caching still works.
The NVIDIA driver compute cache at ~/.nv/ComputeCache is keyed by PTX content hash. CUDA_CACHE_DISABLE is captured at libcuda init time so it cannot be toggled after the Python interpreter has started. Workaround: when offline_cache=False, append a process-stable nonce comment to the PTX so the driver's content hash misses across runs but stays identical for kernels with identical PTX within one run.
New docs/source/user_guide/init_options.md user-guide page (hooked into the user-guide index under Performance). Documents offline_cache plus the other commonly-tuned qd.init(...) knobs (cfg_optimization, fast_math, num_compile_threads, debug, check_out_of_bound). No code change for those knobs - just consolidating prose that did not have a single home before.

Why

Mixed cache hits across runs were producing apparent ~5min cold compiles that were actually <30s warm compiles, hiding the real cost we wanted to profile. CUDA_CACHE_DISABLE cannot help mid-process - the driver caches the env var at cuInit time. The only in-process way to bypass the driver compute cache is to perturb the PTX hash itself.

Mechanism

PtxCache gating

Before:

CacheMode PtxCache::get_cache_mode(const CompileConfig &compile_config) {
  return CacheMode::MemAndDiskCache;
}

After:

CacheMode PtxCache::get_cache_mode(const CompileConfig &compile_config) {
  return compile_config.offline_cache ? CacheMode::MemAndDiskCache : CacheMode::MemCache;
}

Three call paths now consult this:

Constructor (ptx_cache.cpp) - skip loading metadata.tcb when not MemAndDiskCache.
dump() - early-return when not MemAndDiskCache (the per-entry filter at the loop already drops MemCache tagged entries, but skipping the lock acquisition is cleaner).
store_ptx - tag new entries with the current mode rather than always MemAndDiskCache.

Driver compute cache bypass via PTX nonce

compile_module_to_ptx calls append_compute_cache_bypass_nonce_if_disabled(buffer, config) after assembling the PTX. The helper:

void append_compute_cache_bypass_nonce_if_disabled(std::string &ptx, const CompileConfig &compile_config) {
  if (compile_config.offline_cache) {
    return;
  }
  static const std::string session_nonce =
      fmt::format("\n// quadrants-session-nonce: {}\n", static_cast<int64_t>(std::time(nullptr)));
  ptx.append(session_nonce);
}

The nonce is static so it is computed once per process. Two kernels in the same run that produce identical PTX still hit the driver cache and skip a second ptxas invocation; only cross-process reuse is broken, which is exactly the property needed for clean cold-cache measurements. PTX comments do not affect SASS codegen.

Per-flag matrix

offline_cache	PtxCache disk reads	PtxCache disk writes	Driver compute cache hits across runs
True (default)	yes	yes	yes (unchanged)
False	no	no	no (PTX hash differs per process)

Tests

tests/cpp/runtime/cuda/ptx_cache_test.cpp

PtxCache.OfflineCacheDisabledSkipsDisk - pins the PtxCache fix. With offline_cache=False: in-memory load still works, dump() does not create the ptx_cache_sm_<cc> directory, and a fresh instance returns nullopt for the previously-stored key. Fails on the unfixed tree (the sm_dir is created and the second instance loads from disk).

tests/cpp/runtime/cuda/jit_cuda_test.cpp (new file)

JitCudaSessionNonce.OfflineCacheEnabledLeavesPtxUntouched - smoke-tests the no-op branch.
JitCudaSessionNonce.OfflineCacheDisabledAppendsStableNonce - asserts the nonce is appended and that two calls within the same process produce identical output, directly pinning the documented intra-run cache hit / cross-run cache miss invariant.

Side-effect audit

Default qd.init() (offline_cache=True): all caching paths unchanged.
qd.init(offline_cache=False) / QD_OFFLINE_CACHE=0: ~/.cache/quadrants/.../ptx_cache_sm_* is never written to, ~/.nv/ComputeCache will accumulate one fresh entry per process per kernel rather than reusing the cross-run entry.
The PTX nonce is gated on !offline_cache so production users do not bust the driver cache.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 095bda5f19

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

duburcqa · 2026-04-27T21:24:05Z

@claude review

hughperkins · 2026-04-27T21:36:15Z

Makes sense, though I wonder what are the performance implications on testing if we have a single global flag for evertyhing 🤔

Anyway, we can always make more granular options later. This particular flag has a very 'global sounding' name, so seems fair to shut down the cache globaly.

Might be nice to document this option somewhere, and what it does. (I know we don't have it documented anywhere till now; but that's been generally true of our docs).

…eplace std::time(nullptr) (whole-second resolution, collides on back-to-back CI restarts) with std::chrono::system_clock::now().time_since_epoch().count() XOR std::random_device, and set compile_config.offline_cache = true in ptx_cache_test's TestBasic and TestSmVersionPartitioning so they keep exercising the disk-persistence path now that get_cache_mode honors the flag - addresses claude bot review feedback on PR #580

duburcqa · 2026-04-27T21:48:23Z

Makes sense, though I wonder what are the performance implications on testing if we have a single global flag for evertyhing 🤔

This env var and corresponding config flag can be modified between every Quadrants reset and it will be parsed again, from what I know (I think it is even more regular that this, but this is sufficient to get a test-specific behaviour).

…bleshooting.md so users know that it now bypasses both the Quadrants PtxCache and the NVIDIA driver compute cache via PTX nonce, addressing claude / codex bot review feedback on PR #580 about missing user-facing docs for the public-API behavior change

…oting.md and into a new init_options.md user-guide page so the description sits next to other qd.init knobs rather than under crash-recovery prose; troubleshooting.md keeps the single-line cross-reference, addressing user follow-up review feedback on PR #580 about troubleshooting not being the right place

… PtxCache disk reads / writes gated on compile_config.offline_cache, plus a process-stable PTX nonce that bypasses the NVIDIA driver compute cache (~/.nv/ComputeCache) without losing intra-run cubin sharing; tests cover both the disk-skip and the nonce stability invariants

…l three cache layers it gates (kernel-IR cache cross-backend, PtxCache CUDA-only, driver compute cache via PTX nonce), drop the Genesis-specific framing in the page intro, and switch to 'some of the knobs' wording so the page does not promise exhaustive coverage; addresses user follow-up review feedback on PR #580

…graph (no Genesis mention, no recently-added framing), drop the per-section subheadings for the existing compile-time knobs in favor of a tighter bullet list, and simplify the debug / check_out_of_bound sections; addresses user follow-up review feedback on PR #581

…nged or added recently' qualifier from the init_options.md intro; the page is reference documentation, not a changelog, addresses user follow-up review feedback on PR #580

… PtxCache disk reads / writes gated on compile_config.offline_cache, plus a process-stable PTX nonce that bypasses the NVIDIA driver compute cache (~/.nv/ComputeCache) without losing intra-run cubin sharing; tests cover both the disk-skip and the nonce stability invariants; init_options.md user-guide page documents the offline_cache layers

…sistent on-disk caching across Python invocations and cold-start emulation; tighten the prose to remove casual phrasing, addresses user follow-up review feedback on PR #580

…ere a cold start denotes' with 'i.e.', addresses user follow-up review feedback on PR #580

… PtxCache disk reads / writes gated on compile_config.offline_cache, plus a process-stable PTX nonce that bypasses the NVIDIA driver compute cache

…e_path rather than a hardcoded location, and pull the kernel-IR + PtxCache layers under one sentence noting the shared configurable directory; addresses user follow-up review feedback on PR #580

… PtxCache disk reads / writes gated on compile_config.offline_cache, plus a process-stable PTX nonce that bypasses the NVIDIA driver compute cache

…LINE_CACHE=0 instead of QD_CUDA_JIT_OPT_LEVEL=2) so the two PRs ship the same opening paragraph and the integration merge is conflict-free; addresses user follow-up review feedback on PR #581

… PtxCache disk reads / writes gated on compile_config.offline_cache, plus a process-stable PTX nonce that bypasses the NVIDIA driver compute cache

github-actions · 2026-04-27T23:29:53Z

Coverage Report (`5f598cf20`)

File	Coverage	Missing

Diff coverage: 0% · Overall: 71% · 0 lines, 0 missing

Full annotated report

… PtxCache disk reads / writes gated on compile_config.offline_cache, plus a process-stable PTX nonce that bypasses the NVIDIA driver compute cache

…e_path rather than a hardcoded location, and pull the kernel-IR + PtxCache layers under one sentence noting the shared configurable directory; addresses user follow-up review feedback on PR #580

github-actions · 2026-04-28T12:48:42Z

Coverage Report (`118d1b221`)

File	Coverage	Missing

Diff coverage: 0% · Overall: 73% · 0 lines, 0 missing

Full annotated report

hughperkins · 2026-04-28T13:00:32Z

@@ -0,0 +1,54 @@
+# qd.init options
+
+`qd.init(...)` accepts every field of the underlying `CompileConfig` struct as a keyword argument; the same fields are also reachable as environment variables of the form `QD_<UPPERCASE_NAME>` (e.g. `QD_OFFLINE_CACHE=0`). This page covers some of the knobs that are commonly tuned in practice. The underlying source of truth is `quadrants/program/compile_config.h`.


Is it possible to make quadrants/program/compile_config.h an actual link somehow? (I dont think we have precedent in these docs for doing this, but I think it would be very convenient for readers I feel).

hughperkins · 2026-04-28T13:02:16Z

Very detailed doc 🔥

hughperkins · 2026-04-28T13:03:02Z

Very nice doc 🔥

Just would be nice to make the linkt to the compile config an actual clickable link please, somehow. Will make it much more convenient people to use I feel.

hughperkins · 2026-04-28T13:11:11Z

checklist:

user-facing changes documented in user-facing docs
=> yes

ok to merge

… PtxCache disk reads / writes gated on compile_config.offline_cache, plus a process-stable PTX nonce that bypasses the NVIDIA driver compute cache, plus a new init_options.md user-guide page documenting the commonly-tuned qd.init knobs

github-actions · 2026-04-28T14:20:22Z

Coverage Report (`31c6bb0c2`)

File	Coverage	Missing

Diff coverage: 0% · Overall: 73% · 0 lines, 0 missing

Full annotated report

… PtxCache disk reads / writes gated on compile_config.offline_cache, plus a process-stable PTX nonce that bypasses the NVIDIA driver compute cache, plus a new init_options.md user-guide page documenting the commonly-tuned qd.init knobs

…_OFFLINE_CACHE=0 yields a true cold compile: gate PtxCache disk reads / writes on compile_config.offline_cache instead of unconditionally tagging entries MemAndDiskCache, and append a process-stable nonce comment to PTX so the NVIDIA driver compute cache (~/.nv/ComputeCache, keyed by PTX hash) cannot serve stale SASS across runs while kernels with identical PTX still share a cubin within one run