Skip to content

feat: add compressed SFS, shrink pipeline, multi-version Perl, and release workflow#32

Open
lbe wants to merge 1 commit into
6over3:mainfrom
lbe:main
Open

feat: add compressed SFS, shrink pipeline, multi-version Perl, and release workflow#32
lbe wants to merge 1 commit into
6over3:mainfrom
lbe:main

Conversation

@lbe

@lbe lbe commented Jun 21, 2026

Copy link
Copy Markdown

The changes in this pull request were driven by my need to have as small of a WASM file as I could create to support go-exiftool-wasm, a Go package that is based upon translating WASM to native Go. I will be the first to admit that it is very large for a pull request. I did my best to approach your original package from an extend, not rewrite approach. I think was able to follow this approach except for needing to move from patch files to perl patch scripts to support various versions of perl without having to have a patch file per version.

A few points on the perl versions:

  • While exiftool will run on perl versions back to 5.8, I was unable to get perl to compile to WASM for versions earlier that 5.16 without having to do some significant code changes.
  • This code does support all version of perl from 5.16 - 5.42 except for 5.20 which also would require very large changes to compile to WASM.
  • This pull release generates exiftool-5.42.2.wasm 14.6MB in size that is functionally equivalent to the upstream's 24.2MB file.
  • The 5.16.3 version of just perl is 9.8MB. I was actually able to get it down to less that 7MB by stripping out more stdlib. I decided that 9.8MB compiled fast enough to stick with a full perl. In Go, it compiles in < 1 min whereas the 24.2MB version was taking over 20 mins. I add the actual exiftool source in as an embedded file system so that I can version it separately from perl.

I wll stop here, just short of War & Peace. I hope that this merge meets you acceptance. Please let me know if there is anything that I can do to help. LBE


Multi-version Perl support (5.16.x and 5.42.x lines) with WASI cross-compilation, deterministic shrink pipeline, LZ4-compressed SFS embedding, and a matrix release workflow for generating variant WASM artifacts.

Pipeline & Docker

  • Extend Dockerfile and build scripts for native prefix, shrink regen, prefix prep, wasm link (NO_MATHOMS/XS init, SFS compression, embed-prefix=false paths).
  • Add build.sh orchestration script for local end-to-end builds.
  • Refresh hints-wasi, prepare-prefix, build-wasm, build-exiftool, build-native-perl, build-wasi-libs.
  • Add .dockerignore and root package.json for workspace tooling.

Shrink pipeline

  • Add tools/regen-wasm-shrink.sh, tools/trace-zeroperl-deps.pl, tools/emit-wasm-xs-bundle.pl for traced dependency collection and deterministic shrink artifact generation.
  • Add tools/wasm-shrink.env for tracer configuration.
  • Replace glob.patch with more resilient patch_glob.pl; add patch_mg.pl, patch_iperlsys.pl, patch_sv_locale.pl for version-specific source fixes.
  • Add tools/unicore-strip.pl, tools/file-glob-shim.pm, tools/perltidy-skip.txt, tools/extract-missing-paths.pl.
  • Add tools/wasm-xs-kernel.txt, tools/check-wasm-shrink-determinism.sh.

SFS compression

  • Rework tools/sfs.js for LZ4-framed entries with codec=0/1 per-file.
  • Add stubs/sfs_runtime.c, stubs/sfs_compression.c, stubs/sfs_lz4.h, stubs/sfs_runtime.h, stubs/sfs_compression.h for lazy decompression with LRU cache (20 MB cap, 4096 entries).
  • Wire ZEROPERL_SFS_COMPRESS through Dockerfile, prepare-prefix.sh, build-wasm.sh.

Smoke tests and verification

  • Add tests/smoke/core-smoke.pl, tests/smoke/core-mod-smoke.pl, tests/smoke/lib/Core/TestMod.pm, and sample corpus (sample.jpg.b64, sample.tiff.b64, sample.xmp).
  • Add tools/wasm-smoke.mjs, tools/wasm-smoke.sh, tools/verify-embedded-inc.mjs for embedded prefix verification.
  • Add tests/sfs/test-generator.js, tests/sfs/test-sfs.c, tests/sfs/Makefile for native SFS unit tests.

CI and release

  • Add zeroperl-release.yml: matrix builds 4 WASM variants in parallel (exiftool-5.16.3, exiftool-5.42.2, perl-only-5.16.3, perl-only-5.42.2) with full CI verification (shrink determinism, smoke tests, embedded @inc check) and publishes as a draft GitHub release.
  • Remove zeroperl.yml (replaced by zeroperl-release.yml).
  • Update nodefs workflow and Binaryen/WABT setup actions.
  • Update all GitHub Actions to Node.js 24 compatible versions (checkout@v7, upload-artifact@v7, download-artifact@v8, action-gh-release@v3).

Submodules and repo hygiene

  • Add zeroperl-ts git submodule; remove go-exiftool-wasm submodule.
  • Remove test-matrix.sh, test-matrix-all.sh, run-failed-and-report.sh (depended on go-exiftool-wasm).
  • Remove perl-delta-spec.md, .vscode/settings.json, check-fs.mjs.
  • Update .gitignore; add gen/, output/, test-output/, tmp/ entries.

Docs

  • Add comprehensive ARCHITECTURE.md covering build pipeline, runtime architecture, shrink modes, SFS compression, and CI verification.
  • Revise README.md with build instructions, build arg reference, and submodule setup.

…lease workflow

Multi-version Perl support (5.16.x and 5.42.x lines) with WASI
cross-compilation, deterministic shrink pipeline, LZ4-compressed SFS
embedding, and a matrix release workflow for generating variant WASM
artifacts.

Pipeline & Docker
- Extend Dockerfile and build scripts for native prefix, shrink regen,
  prefix prep, wasm link (NO_MATHOMS/XS init, SFS compression,
  embed-prefix=false paths).
- Add build.sh orchestration script for local end-to-end builds.
- Refresh hints-wasi, prepare-prefix, build-wasm, build-exiftool,
  build-native-perl, build-wasi-libs.
- Add .dockerignore and root package.json for workspace tooling.

Shrink pipeline
- Add tools/regen-wasm-shrink.sh, tools/trace-zeroperl-deps.pl,
  tools/emit-wasm-xs-bundle.pl for traced dependency collection and
  deterministic shrink artifact generation.
- Add tools/wasm-shrink.env for tracer configuration.
- Replace glob.patch with more resilient patch_glob.pl; add patch_mg.pl,
  patch_iperlsys.pl, patch_sv_locale.pl for version-specific source
  fixes.
- Add tools/unicore-strip.pl, tools/file-glob-shim.pm,
  tools/perltidy-skip.txt, tools/extract-missing-paths.pl.
- Add tools/wasm-xs-kernel.txt, tools/check-wasm-shrink-determinism.sh.

SFS compression
- Rework tools/sfs.js for LZ4-framed entries with codec=0/1 per-file.
- Add stubs/sfs_runtime.c, stubs/sfs_compression.c, stubs/sfs_lz4.h,
  stubs/sfs_runtime.h, stubs/sfs_compression.h for lazy decompression
  with LRU cache (20 MB cap, 4096 entries).
- Wire ZEROPERL_SFS_COMPRESS through Dockerfile, prepare-prefix.sh,
  build-wasm.sh.

Smoke tests and verification
- Add tests/smoke/core-smoke.pl, tests/smoke/core-mod-smoke.pl,
  tests/smoke/lib/Core/TestMod.pm, and sample corpus
  (sample.jpg.b64, sample.tiff.b64, sample.xmp).
- Add tools/wasm-smoke.mjs, tools/wasm-smoke.sh,
  tools/verify-embedded-inc.mjs for embedded prefix verification.
- Add tests/sfs/test-generator.js, tests/sfs/test-sfs.c,
  tests/sfs/Makefile for native SFS unit tests.

CI and release
- Add zeroperl-release.yml: matrix builds 4 WASM variants in parallel
  (exiftool-5.16.3, exiftool-5.42.2, perl-only-5.16.3,
  perl-only-5.42.2) with full CI verification (shrink determinism,
  smoke tests, embedded @inc check) and publishes as a draft GitHub
  release.
- Remove zeroperl.yml (replaced by zeroperl-release.yml).
- Update nodefs workflow and Binaryen/WABT setup actions.
- Update all GitHub Actions to Node.js 24 compatible versions
  (checkout@v7, upload-artifact@v7, download-artifact@v8,
  action-gh-release@v3).

Submodules and repo hygiene
- Add zeroperl-ts git submodule; remove go-exiftool-wasm submodule.
- Remove test-matrix.sh, test-matrix-all.sh, run-failed-and-report.sh
  (depended on go-exiftool-wasm).
- Remove perl-delta-spec.md, .vscode/settings.json, check-fs.mjs.
- Update .gitignore; add gen/, output/, test-output/, tmp/ entries.

Docs
- Add comprehensive ARCHITECTURE.md covering build pipeline, runtime
  architecture, shrink modes, SFS compression, and CI verification.
- Revise README.md with build instructions, build arg reference,
  and submodule setup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant