[Build] Linux x86 runner by hughperkins · Pull Request #3 · Genesis-Embodied-AI/quadrants

hughperkins · 2025-06-02T17:44:35Z

Issue: #

Brief Summary

Add linux x86 runner

runs on github standard runner
runs on any branch
for now it is 'run on demand', though I could update it to run on every commit, to a PR, eg could update that in this PR itself if we want

Notes:

Currently this doesnt include changes for tests to run cleanly (though I realized just now that the opengl changes are needed only for being able to run tests withou seg faulting I think, so we could kick that change ouf of this PR if we want?)
I found I needed to update changelog.py, in order to not crash when not making a release.

copilot:summary

Walkthrough

copilot:walkthrough

…figurations

hughperkins · 2025-06-02T18:03:02Z

Note: we have to upgrade vulkan version, because old version no longer avialble.

duburcqa · 2025-06-03T12:33:59Z

+      - uses: actions/checkout@v4
+      - name: Linux x86 Build
+        run: |
+          bash .github/workflows/scripts_new/linux_x86.sh


I do get the point of putting the CI script in a dedicated file instead of just pasting itself contain here. If the script is only used for CI, avoid unnecessary layers for indirection that are just obfuscating the logic.

It would also allow splitting it into multiple steps or jobs for better modularity and readability of failures.

Ok, so my understanding of the desires here are:

nice to see multiple steps

nice to be able to track progress across multiple steps in UI, eg something like https://github.com/duburcqa/jiminy/actions/runs/14775364707/job/41482561535

I in my turn personally like to factorize code out of the runner configuration because:

decouples code from runner configuration

means I can run/test the code easily, on arbitrary platforms, without needing to somehow trigger full CI flow etc

eg spin up a container, try running the script

run on my mac

etc ...

personally, I also find it easier to format code in standalone bash scripts, without needing to indent, pay attention to various yaml quirks etc

lastly, I feel that we shouldn't require potential developers to be able to read github runner config files, when setting up their development environment

yet I feel that the scripts used by CI are the best 'golden source' for what works today, when someone wants to install

To comply with all the above 😓 , what I have done is:

split the runner configuration into 4 steps:

prerequisites

build

install

test

split the script corresondingly into 4 parts, one part for each step

added documentation to the dev install page, linking to each of these steps, as a reference

Now I admit that we should ideally turn on the markup link checker, to run across all .md files, systematically, even if they didn't change, since someone could rename one of these scripts, and break the build page 😓 Well, baby steps. We do at least have the link checker running, and we'll turn it on, to run across all .md files, systematically, in a bit.

duburcqa · 2025-06-03T12:35:30Z

        outdir.mkdir(parents=True, exist_ok=True)
        unzip(local_cached, outdir, strip=strip)
-    elif name.endswith(".tar.gz") or name.endswith(".tgz"):
+    elif any(name.endswith(ext) for ext in [".tar.gz", ".tgz", ".xz", ".tar.bz2"]):


No need to change here, but I recommend immutable tuple instead of mutable list if mutability is irrelevant. better use the minimal required container, this makes the code more self-explanatory and may avoid silly mistakes (i.e. default values for method in Python).

Personally, I feel that list is more intuitive to read. tuple on the whole I feel represents something more analogous to a struct, even though it's technically iterable. I don't have a strong preference to keep it as a list, but I do have a gentle preference.

Default values for a method, totally agree with using tuple, instead of list, for the reason you allude to: that parameters to methods should not be mutaable, otherwise if you modify them inside the method, then you just modified thd efault parametrsr 😓

I guess stachexchange tends to align with your own pov on this: when not mutable, use a tuple https://stackoverflow.com/questions/1708510/list-vs-tuple-when-to-use-each. i'll change to tuple

actually, I might ask in #code

CharlesMasson · 2025-06-03T13:52:34Z

+      - uses: actions/checkout@v4
+      - name: Linux x86 Build
+        run: |
+          bash .github/workflows/scripts_new/linux_x86.sh


It would also allow splitting it into multiple steps or jobs for better modularity and readability of failures.

…link from the doc

hughperkins · 2025-06-04T20:01:36Z

Thank you!

Introduce the helper machinery that the per-class to_torch / to_numpy methods will migrate to in subsequent commits. Existing public symbols (can_zerocopy, dlpack_to_torch, invalidate_zerocopy_cache, current_arch_is_cpu) are preserved as deprecated shims so the in-tree pre-rework callers continue to work; they will be removed once every call site is migrated. New surface: - _ZerocopyCache: per-instance container with two independent slots (torch tensor + numpy ndarray), each filled lazily on first access via torch.utils.dlpack.from_dlpack and numpy.from_dlpack respectively. Numpy zero-copy now bypasses torch entirely (closes review #6). - make_zerocopy_cache_if_supported(owner, ...): constructs a cache when zero-copy is supported and registers `owner` with `pyquadrants.cache_holders` so invalidation is wired automatically (closes review #18). - get_zerocopy_torch / get_zerocopy_numpy: thin entry points that implement the always-zerocopy-then-clone semantic (closes review #15, #16, #21) and the Apple Metal double-sync (qd.sync() on read AND torch.mps.synchronize() after .clone()/.to(); closes review #1, #22, #23). Also applies the small lints from the review: - Module-level constant for the torch>2.9.1 MPS bytes_offset probe; drops the pointless lru_cache wrapper around a zero-arg helper (closes review #2). - ASCII '...' instead of Unicode horizontal ellipsis '\u2026' in the docstring (closes review #3). - Top-level imports for numpy and torch (try/except for the no-torch CI case); no per-call lazy imports in the new code path (closes review #7, #9). The deprecated shim still does what the existing per-class methods expect; the new helpers are torch-clean. cache_holders is still empty until the next commits register Ndarray / ScalarField / MatrixField; this commit alone is no-op behaviourally.

…olymorphism Codex #3 review on PR #704 (https://github.com/Genesis-Embodied-AI/ quadrants/pull/704#discussion_r3253281957) flags that _struct_nd_paths_cache is keyed by type(arg) and breaks if two instances of the same class have polymorphic attribute structure (ndarray-backed Tensor on one, field-backed Tensor on another at the same path). The per-instance walk redesign in PR #705 (stacked on top of #704) fixes this. Document the gap here as a FIXME so the issue isn't lost if #705 takes a different shape.

…unc-dataclass Brings in PR #704 (ndarray-on-data_oriented) plus 12 other commits from main. Resolved conflicts: - docs/compound_types.md: took main's post-PR-704 version end-to-end (the deprecation notice, "How to choose", clearer table rows, qd.Template normalisation — all authored during the PR #704 review cycle). - docs/fastcache.md: kept HEAD's pruning-driven hashing references and the flattened (no "## Appendix") doc structure; absorbed main's keying-row wording where it didn't conflict. - python/quadrants/lang/_template_mapper_hotpath.py: kept HEAD's per-instance ndarray-path cache + cycle-safe walker (Codex #3 fix, polymorphic-instance attribute structure). Folded in main's chain_has_mutable_container as a free function alongside the existing struct-walking helpers. - python/quadrants/lang/_template_mapper.py: kept HEAD's per-class _arg_disposition + _classify_disposition fast path over the slot positions only (vs main's loop over all args). main's _collect_data_oriented_nd_ids superseded by HEAD's path walk. - python/quadrants/lang/kernel.py: switched _mutable_nd_cached_val construction to use chain_has_mutable_container (deeper Codex #1 chain walk from main) while preserving HEAD's _qd_stable_members short-circuit. - python/quadrants/lang/ast/.../function_def_transformer.py: kept HEAD's cycle-safe _walk_obj recursion (the `seen` set parameter from PR #705's robustness pass). - tests/test_ad_dataclass.py, tests/test_data_oriented_ndarray.py: every <<<<<< / >>>>>> block was a wrapping-width difference between HEAD's 120c reflow and main's slightly narrower wrap — took HEAD throughout. main's new test_data_oriented_ndarray_wrapper added as a new section 23 (the wrapper- unwrap branch coverage from the PR #704 CI follow-ups; missing on this branch because the d590230 commit landed on the sibling hp/data-oriented-ndarray-fix branch and was never cherry-picked here). Verified: pre-commit clean (black + ruff + pylint), ast.parse + ruff check clean. Co-authored-by: Cursor <cursoragent@cursor.com>

The original #704 cached ndarray-attribute paths per CLASS, which is incorrect under polymorphic instance structure: ``@qd.data_oriented`` and ``qd.Tensor`` members can have different leaf-types across instances of the same class — Genesis ``DataManager`` only allocates ``*_adjoint_cache`` ndarrays when ``requires_grad=True``, and a ``qd.Tensor`` field can wrap an ``Ndarray`` on one instance and a ``MatrixField`` on another (e.g. ndarray-vs-field backend). ``_collect_struct_nd_descriptors`` then crashed reading ``element_type`` / ``shape`` off the ``MatrixField`` (``'MatrixField' object has no attribute 'element_type'``), affecting ~60 Genesis unit tests under the ndarray backend. Fix: cache the path list on the *instance* via ``object.__setattr__`` (so frozen dataclasses work too); ``__slots__`` classes without a ``__dict__`` silently fall back to the class-level cache (Genesis containers don't use slots). Also add defensive ``isinstance(v, Ndarray)`` and ``shape is None`` guards in ``_collect_struct_nd_descriptors`` for the case where a leaf changes type *within* an instance's lifetime.

hughperkins added 10 commits June 2, 2025 10:33

make_changelog doesnt crash when we arent making a release

55fed1a

add linux_x86_ondemand.yml runner

615906f

get avilable architetures no longer segfaults with certain opengl con…

eef2418

…figurations

compiler.py works with llvm-15 now

57fdc03

run on prs, and pushes to main

1ba6202

remove branches_ignore

d1cd64a

remove on push to main

59605bd

remove automatic

13479a5

upgrade vulkan version

79d16a7

re-enable run on PR

f540bff

hughperkins added 3 commits June 2, 2025 11:10

handle .xz deps

81eeca8

master => main

db96546

return None

ee692a5

hughperkins force-pushed the hp/linux-x86-runner branch from 13820c5 to ee692a5 Compare June 2, 2025 18:42

duburcqa reviewed Jun 2, 2025

View reviewed changes

Comment thread .github/workflows/scripts/ti_build/dep.py Outdated

Comment thread .github/workflows/scripts/ti_build/dep.py

Comment thread cherry_linx86.sh Outdated

hughperkins added 2 commits June 2, 2025 14:13

remove inadvertently added file

418cc7b

change to any and for loop

a00aa84

duburcqa reviewed Jun 3, 2025

View reviewed changes

CharlesMasson approved these changes Jun 3, 2025

View reviewed changes

hughperkins added 4 commits June 3, 2025 13:33

Split the build script into prerequisites, build, install, test, and …

8b32319

…link from the doc

fix urls

87f8be9

Merge remote-tracking branch 'genesis/main' into hp/linux-x86-runner

e5a5fe9

flaky test

afc1d64

duburcqa reviewed Jun 4, 2025

View reviewed changes

Comment thread .github/workflows/linux_x86.yml

duburcqa reviewed Jun 4, 2025

View reviewed changes

Comment thread tests/python/test_ad_ndarray.py Outdated

hughperkins added 2 commits June 4, 2025 10:40

remove delay

9030f28

update link phrase

451f379

hughperkins force-pushed the hp/linux-x86-runner branch from 0a047ed to 451f379 Compare June 4, 2025 19:00

duburcqa approved these changes Jun 4, 2025

View reviewed changes

hughperkins merged commit db5ce62 into main Jun 4, 2025
23 of 26 checks passed

hughperkins deleted the hp/linux-x86-runner branch June 4, 2025 20:01

This was referenced Mar 11, 2026

[Perf] CUDA graph 2: graph_do_while #406

Merged

[Perf] Streams 3: Add qd.stream_parallel() context manager #409

Closed

duburcqa mentioned this pull request Apr 13, 2026

[Lang] Add qd.precise(...) for per-op IEEE-strict FP. #476

Open

This was referenced Apr 15, 2026

[Perf] Tiles 5: add Cholesky benchmark demo #483

Merged

[Perf] Tiles 4: add SharedArray slice support #482

Merged

duburcqa mentioned this pull request Apr 16, 2026

[AutoDiff] Autodiff 10: Implement adstack for SPIR-V #490

Merged

claude Bot mentioned this pull request Apr 21, 2026

[AutoDiff] Autodiff 5: Fix adjoint-alloca placement for GlobalLoads outside the current range-for #496

Merged

duburcqa mentioned this pull request Apr 24, 2026

[SPIRV] Vulkan SPIR-V correctness: atomic-view aliasing, PSB stride, narrow storage caps, u1 cast, per-init layer recheck #513

Merged

claude Bot mentioned this pull request Apr 28, 2026

[Perf] Streams 2: Add AMDGPU/HIP stream support #408

Closed

hughperkins mentioned this pull request Jun 5, 2026

[DataOriented] Fix ndarrays on data oriented (reland of #704) #723

Merged

Uh oh!

Conversation

hughperkins commented Jun 2, 2025

Brief Summary

Walkthrough

Uh oh!

hughperkins commented Jun 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hughperkins commented Jun 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants