[Build] Upgrade cmake minimum version to be compatible with cmake 4.x#1
Merged
Conversation
duburcqa
reviewed
Jun 2, 2025
duburcqa
approved these changes
Jun 2, 2025
Collaborator
Author
|
Thanks! 🙌 |
CharlesMasson
approved these changes
Jun 3, 2025
Collaborator
Author
|
Thanks! |
This was referenced Mar 11, 2026
This was referenced Apr 13, 2026
hughperkins
added a commit
that referenced
this pull request
Apr 26, 2026
Introduce the helper machinery that the per-class to_torch / to_numpy methods will migrate to in subsequent commits. Existing public symbols (can_zerocopy, dlpack_to_torch, invalidate_zerocopy_cache, current_arch_is_cpu) are preserved as deprecated shims so the in-tree pre-rework callers continue to work; they will be removed once every call site is migrated. New surface: - _ZerocopyCache: per-instance container with two independent slots (torch tensor + numpy ndarray), each filled lazily on first access via torch.utils.dlpack.from_dlpack and numpy.from_dlpack respectively. Numpy zero-copy now bypasses torch entirely (closes review #6). - make_zerocopy_cache_if_supported(owner, ...): constructs a cache when zero-copy is supported and registers `owner` with `pyquadrants.cache_holders` so invalidation is wired automatically (closes review #18). - get_zerocopy_torch / get_zerocopy_numpy: thin entry points that implement the always-zerocopy-then-clone semantic (closes review #15, #16, #21) and the Apple Metal double-sync (qd.sync() on read AND torch.mps.synchronize() after .clone()/.to(); closes review #1, #22, #23). Also applies the small lints from the review: - Module-level constant for the torch>2.9.1 MPS bytes_offset probe; drops the pointless lru_cache wrapper around a zero-arg helper (closes review #2). - ASCII '...' instead of Unicode horizontal ellipsis '\u2026' in the docstring (closes review #3). - Top-level imports for numpy and torch (try/except for the no-torch CI case); no per-call lazy imports in the new code path (closes review #7, #9). The deprecated shim still does what the existing per-class methods expect; the new helpers are torch-clean. cache_holders is still empty until the next commits register Ndarray / ScalarField / MatrixField; this commit alone is no-op behaviourally.
hughperkins
added a commit
that referenced
this pull request
Apr 26, 2026
Convert Ndarray.to_torch / _ndarray_to_numpy / _ndarray_matrix_to_numpy to thin calls into _interop.get_zerocopy_torch / get_zerocopy_numpy. Behaviour changes vs. the previous PR state: - to_torch(copy=True) now ALSO uses the zerocopy export and clones the result, instead of skipping zerocopy entirely. Zerocopy DLPack is cheaper than the kernel-copy fallback even when followed by a clone (closes review #15, #16, #21). - to_torch on Apple Metal now does both syncs: qd.sync() before the read AND torch.mps.synchronize() after the .clone() so the cloned buffer is actually populated by the time the call returns (closes review #1, #22, #23). - to_numpy(copy=False) now goes through numpy.from_dlpack directly -- no torch round-trip, no torch import required (closes review #6). - to_numpy default (copy=None) keeps the current "independent copy" semantics: numpy arrays conventionally outlive their source, so the default is the safer one. Zero-copy is opt-in via copy=False. Cache invalidation switches from the legacy _qd_dlpack_tc attribute to the per-class _zerocopy_cache (cached_property) + _invalidate_zerocopy_cache method. Registration in pyquadrants.cache_holders happens automatically on first cache access (closes review #18). The existing _reset hook drops its zerocopy invalidation call: that work is now handled by the cache_holders loop in impl.reset() BEFORE C++ teardown, eliminating the use-after-free that motivated the Genesis to_numpy() always-copy workaround.
hughperkins
added a commit
that referenced
this pull request
Apr 26, 2026
Convert ScalarField.to_torch / to_numpy to thin calls into _interop.get_zerocopy_torch / get_zerocopy_numpy, mirroring the Ndarray migration in the previous commit. - Adds Field._invalidate_zerocopy_cache method (used by all subclasses) so the impl.reset() cache_holders loop has a single contract to call. - Adds ScalarField._zerocopy_cache as a cached_property; constructed once per instance instead of re-checking can_zerocopy on every call (closes review #17). - Registers ScalarField in pyquadrants.cache_holders on first cache access (closes review #18). - to_torch(copy=True) now zerocopies and clones rather than skipping zerocopy entirely (closes review #15, #16, #21); to_torch on Metal does both syncs (closes review #1, #22, #23). - to_numpy(copy=False) goes through numpy.from_dlpack directly, no torch round-trip and no torch import required (closes review #6). Default to_numpy semantic (copy=None) remains "independent copy" for lifetime safety; zero-copy is opt-in via copy=False.
hughperkins
added a commit
that referenced
this pull request
Apr 26, 2026
Convert MatrixField.to_torch / to_numpy to thin calls into _interop.get_zerocopy_torch / get_zerocopy_numpy. Adds a single _matrix_view_shape() helper that returns (expected_shape, as_vector); both to_torch and to_numpy now go through it instead of duplicating the n / m / keep_dims branching (closes review #13, #14). Same wins as the ScalarField migration: - _zerocopy_cache as cached_property (closes review #17) - automatic registration with pyquadrants.cache_holders (closes #18) - copy=True now zerocopies + clones (closes #15, #16, #21) - to_numpy(copy=False) goes through numpy.from_dlpack directly, no torch round-trip (closes #6) - Apple Metal double-sync via the centralised helpers (closes #1, #22, #23) The redundant reshape after zerocopy stays (DLPack returns the field's flat n-d shape; numpy/torch users expect the matrix dims appended) but it's now in one place per method instead of two.
hughperkins
added a commit
that referenced
this pull request
May 18, 2026
…llback
Unrecognised types in fastcache argument hashing previously had two failure modes, both bad:
- Top-level: ``[FASTCACHE][PARAM_INVALID]`` warn + return None, disabling fastcache for the whole call.
Any solver-like object carrying a single opaque metadata field (Genesis ``UID``, Pydantic config,
back-pointer) silently killed the cache.
- Nested under ``@qd.data_oriented(stable_members=True)``: silent skip. Worked for the Genesis case but is
dangerous: if someone later adds a new tensor-like type (e.g. ``BFloat16Tensor``) whose value affects
kernel codegen but forgets to register it in args_hasher's recognised set, the silent skip serves stale
cache results without any indication.
Both paths are replaced with a single ``type(v).__qualname__``-based fallback (``opaque-<module>.<qualname>``)
that emits a one-shot ``[FASTCACHE][UNKNOWN_TYPE]`` warning per type. Properties:
- Cache key stable across instances of the same opaque class (Genesis UID #1 and UID #2 produce the same
key). Kernels cannot read non-recognised Python types so opaque metadata cannot affect codegen, making
type-identity-only hashing correct for genuinely opaque members.
- Loud diagnostic for the dangerous case: any unrecognised type that ever gets hashed prints a warning
pointing at args_hasher.stringify_obj_type so a missed tensor-like registration is impossible to miss.
- ``ScalarField`` / ``MatrixField`` (recognised-but-unsupported tensor-like) still disable fastcache via
a new ``_FAIL_FASTCACHE`` sentinel — their shape/dtype affect codegen but fastcache doesn't yet handle
them. Distinct from the qualname fallback so the field path remains correct.
Also adds ``pruning_paths`` and ``parent_flat`` plumbing through ``stringify_obj_type`` / ``dataclass_to_repr`` /
``hash_args`` for the upcoming pruning-driven narrow walk (L1 cache lookup of kernel-accessed flat names);
the new parameters default to None so this commit alone is the qualname-fallback baseline.
``test_src_ll_cache_arg_warnings`` updated to assert the new ``[UNKNOWN_TYPE]`` warning (instead of the old
``[PARAM_INVALID]`` + ``[INVALID_FUNC]`` dead-end).
The ``_qd_stable_members`` flag is no longer read by args_hasher; its launch-context role
(``_mutable_nd_cached_val`` short-circuit) is unchanged in this commit and will be addressed separately.
hughperkins
added a commit
that referenced
this pull request
May 19, 2026
…llback
Unrecognised types in fastcache argument hashing previously had two failure modes, both bad:
- Top-level: ``[FASTCACHE][PARAM_INVALID]`` warn + return None, disabling fastcache for the whole call.
Any solver-like object carrying a single opaque metadata field (Genesis ``UID``, Pydantic config,
back-pointer) silently killed the cache.
- Nested under ``@qd.data_oriented(stable_members=True)``: silent skip. Worked for the Genesis case but is
dangerous: if someone later adds a new tensor-like type (e.g. ``BFloat16Tensor``) whose value affects
kernel codegen but forgets to register it in args_hasher's recognised set, the silent skip serves stale
cache results without any indication.
Both paths are replaced with a single ``type(v).__qualname__``-based fallback (``opaque-<module>.<qualname>``)
that emits a one-shot ``[FASTCACHE][UNKNOWN_TYPE]`` warning per type. Properties:
- Cache key stable across instances of the same opaque class (Genesis UID #1 and UID #2 produce the same
key). Kernels cannot read non-recognised Python types so opaque metadata cannot affect codegen, making
type-identity-only hashing correct for genuinely opaque members.
- Loud diagnostic for the dangerous case: any unrecognised type that ever gets hashed prints a warning
pointing at args_hasher.stringify_obj_type so a missed tensor-like registration is impossible to miss.
- ``ScalarField`` / ``MatrixField`` (recognised-but-unsupported tensor-like) still disable fastcache via
a new ``_FAIL_FASTCACHE`` sentinel — their shape/dtype affect codegen but fastcache doesn't yet handle
them. Distinct from the qualname fallback so the field path remains correct.
Also adds ``pruning_paths`` and ``parent_flat`` plumbing through ``stringify_obj_type`` / ``dataclass_to_repr`` /
``hash_args`` for the upcoming pruning-driven narrow walk (L1 cache lookup of kernel-accessed flat names);
the new parameters default to None so this commit alone is the qualname-fallback baseline.
``test_src_ll_cache_arg_warnings`` updated to assert the new ``[UNKNOWN_TYPE]`` warning (instead of the old
``[PARAM_INVALID]`` + ``[INVALID_FUNC]`` dead-end).
The ``_qd_stable_members`` flag is no longer read by args_hasher; its launch-context role
(``_mutable_nd_cached_val`` short-circuit) is unchanged in this commit and will be addressed separately.
hughperkins
added a commit
that referenced
this pull request
May 19, 2026
…llback
Unrecognised types in fastcache argument hashing previously had two failure modes, both bad:
- Top-level: ``[FASTCACHE][PARAM_INVALID]`` warn + return None, disabling fastcache for the whole call.
Any solver-like object carrying a single opaque metadata field (Genesis ``UID``, Pydantic config,
back-pointer) silently killed the cache.
- Nested under ``@qd.data_oriented(stable_members=True)``: silent skip. Worked for the Genesis case but is
dangerous: if someone later adds a new tensor-like type (e.g. ``BFloat16Tensor``) whose value affects
kernel codegen but forgets to register it in args_hasher's recognised set, the silent skip serves stale
cache results without any indication.
Both paths are replaced with a single ``type(v).__qualname__``-based fallback (``opaque-<module>.<qualname>``)
that emits a one-shot ``[FASTCACHE][UNKNOWN_TYPE]`` warning per type. Properties:
- Cache key stable across instances of the same opaque class (Genesis UID #1 and UID #2 produce the same
key). Kernels cannot read non-recognised Python types so opaque metadata cannot affect codegen, making
type-identity-only hashing correct for genuinely opaque members.
- Loud diagnostic for the dangerous case: any unrecognised type that ever gets hashed prints a warning
pointing at args_hasher.stringify_obj_type so a missed tensor-like registration is impossible to miss.
- ``ScalarField`` / ``MatrixField`` (recognised-but-unsupported tensor-like) still disable fastcache via
a new ``_FAIL_FASTCACHE`` sentinel — their shape/dtype affect codegen but fastcache doesn't yet handle
them. Distinct from the qualname fallback so the field path remains correct.
Also adds ``pruning_paths`` and ``parent_flat`` plumbing through ``stringify_obj_type`` / ``dataclass_to_repr`` /
``hash_args`` for the upcoming pruning-driven narrow walk (L1 cache lookup of kernel-accessed flat names);
the new parameters default to None so this commit alone is the qualname-fallback baseline.
``test_src_ll_cache_arg_warnings`` updated to assert the new ``[UNKNOWN_TYPE]`` warning (instead of the old
``[PARAM_INVALID]`` + ``[INVALID_FUNC]`` dead-end).
The ``_qd_stable_members`` flag is no longer read by args_hasher; its launch-context role
(``_mutable_nd_cached_val`` short-circuit) is unchanged in this commit and will be addressed separately.
hughperkins
added a commit
that referenced
this pull request
May 29, 2026
…tr chain Codex review #1 on PR #704 (#704 (comment)): the previous mutable-nd cache predicate only checked mutability of the top-level kernel arg. With a frozen outer container wrapping a mutable inner container that holds the ndarray (``@dataclass(frozen=True) -> @qd.data_oriented -> qd.ndarray``), the top-level arg's ``__hash__`` is not None and ``is_data_oriented`` is False, so the predicate returned False. No leaf id was folded into the launch-context cache key, so reassigning ``outer.inner.x`` between launches left the cache bound to the original ndarray. Reproducer added in ``test_frozen_outer_mutable_inner_ndarray_reassign`` (test 15b). Fix: introduce ``_chain_has_mutable_container`` that walks every parent along ``attr_chain`` from the root down to (but excluding) the leaf and OR-folds the same mutability check (``__hash__ is None`` or ``is_data_oriented``). The check stops at the first mutable parent. Only runs when ``key`` differs from the cached one, so the per-launch overhead is unchanged for the steady-state hot path.
hughperkins
added a commit
that referenced
this pull request
Jun 1, 2026
…ning & ndarray-wrapper tests Three changes addressing the PR #704 CI feedback agents: 1. **Feature-factorization fix** — moved ``Kernel._chain_has_mutable_container`` (an ``@staticmethod`` with zero coupling to ``Kernel`` internals) out of the 812-line ``kernel.py`` and into ``_template_mapper_hotpath.py`` as the free function ``chain_has_mutable_container``. This sits alongside the sibling struct-walking helpers ``_struct_nd_paths_for`` / ``_build_struct_nd_paths`` / ``_collect_struct_nd_descriptors`` that this same PR introduced — the "Check feature factorization" agent (run 26662318558) correctly noted those were placed there but the chain-walker was left behind on ``Kernel``. ``kernel.py`` now imports ``chain_has_mutable_container`` and the only call-site in ``_get_launch_ctx`` calls the free function directly. 2. **Pytest coverage for the DeprecationWarning** — added three tests to ``test_py_dataclass.py`` mirroring the probe in ``tmp/probe_template_dataclass_warning.py``: * ``test_dataclass_with_template_emits_deprecation_warning`` — asserts the warning fires (``pytest.warns`` with the message-regex anchor). * ``test_data_oriented_with_template_does_not_emit_deprecation_warning`` — asserts the warning is suppressed for ``@qd.data_oriented`` instances (the ``is_data_oriented(val)`` exclusion in ``Kernel.materialize``). * ``test_typed_dataclass_does_not_emit_deprecation_warning`` — asserts the supported flat-by-fields path emits no warning. Addresses the "Check test coverage" gap #1 (run 26662318494). 3. **Pytest coverage for the ndarray-wrapper unwrap branch** — added ``test_data_oriented_ndarray_wrapper`` to ``test_data_oriented_ndarray.py`` that constructs a ``@qd.data_oriented`` whose member is a ``qd.Tensor`` wrapper around a ``qd.ndarray`` (``qd.tensor(..., backend=qd.Backend.NDARRAY)``). Confirmed via the throwaway script in ``tmp/verify_unwrap_hit.py`` that both ``_build_struct_nd_paths`` and ``_collect_struct_nd_descriptors`` hit their ``if type(v) in _TENSOR_WRAPPER_TYPES: v = v._unwrap()`` branches (1 hit each) during this test. Addresses the "Check test coverage" gap #2. Regression: 143 tests pass (was 123 + new) across ``test_data_oriented_ndarray.py`` + ``test_data_oriented_mixed_combos.py`` + ``test_py_dataclass.py`` + ``test_tensor_wrapper_in_struct.py``, no failures.
hughperkins
added a commit
that referenced
this pull request
Jun 3, 2026
…unc-dataclass Brings in PR #704 (ndarray-on-data_oriented) plus 12 other commits from main. Resolved conflicts: - docs/compound_types.md: took main's post-PR-704 version end-to-end (the deprecation notice, "How to choose", clearer table rows, qd.Template normalisation — all authored during the PR #704 review cycle). - docs/fastcache.md: kept HEAD's pruning-driven hashing references and the flattened (no "## Appendix") doc structure; absorbed main's keying-row wording where it didn't conflict. - python/quadrants/lang/_template_mapper_hotpath.py: kept HEAD's per-instance ndarray-path cache + cycle-safe walker (Codex #3 fix, polymorphic-instance attribute structure). Folded in main's chain_has_mutable_container as a free function alongside the existing struct-walking helpers. - python/quadrants/lang/_template_mapper.py: kept HEAD's per-class _arg_disposition + _classify_disposition fast path over the slot positions only (vs main's loop over all args). main's _collect_data_oriented_nd_ids superseded by HEAD's path walk. - python/quadrants/lang/kernel.py: switched _mutable_nd_cached_val construction to use chain_has_mutable_container (deeper Codex #1 chain walk from main) while preserving HEAD's _qd_stable_members short-circuit. - python/quadrants/lang/ast/.../function_def_transformer.py: kept HEAD's cycle-safe _walk_obj recursion (the `seen` set parameter from PR #705's robustness pass). - tests/test_ad_dataclass.py, tests/test_data_oriented_ndarray.py: every <<<<<< / >>>>>> block was a wrapping-width difference between HEAD's 120c reflow and main's slightly narrower wrap — took HEAD throughout. main's new test_data_oriented_ndarray_wrapper added as a new section 23 (the wrapper- unwrap branch coverage from the PR #704 CI follow-ups; missing on this branch because the d590230 commit landed on the sibling hp/data-oriented-ndarray-fix branch and was never cherry-picked here). Verified: pre-commit clean (black + ruff + pylint), ast.parse + ruff check clean. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pulling these commits in from taichi-dev/taichi#8703