feat: PQC benchmark driver and cross-library comparison tools#10429
Open
MarkAtwood wants to merge 10 commits intowolfSSL:masterfrom
Open
feat: PQC benchmark driver and cross-library comparison tools#10429MarkAtwood wants to merge 10 commits intowolfSSL:masterfrom
MarkAtwood wants to merge 10 commits intowolfSSL:masterfrom
Conversation
Adds wolfcrypt/benchmark/pqc_bench.sh, a self-contained driver that: - Configures wolfSSL with --enable-mlkem --enable-dilithium --enable-slhdsa and memory/stack tracking options - Builds only the benchmark binary target (make wolfcrypt/benchmark/benchmark) - Runs all ML-KEM, ML-DSA, and SLH-DSA parameter sets with -csv output - Produces a clean single-header CSV file (strips wolfSSL output noise) - Handles both old (plain CSV) and new (GENERATE_MACHINE_PARSEABLE_REPORT) benchmark output formats - Detects aarch64 and adds -march=armv8-a for hardware cycle counters - POSIX sh compatible (verified with checkbashisms and dash)
pqc_parse.py: - Normalises wolfSSL benchmark CSV to NIST algorithm names and canonical operation labels (keygen/encaps/decaps/sign/verify) - Supports both wolfssl CSV format (with Library column) and pqcleo pipe-delimited format for cross-library comparison - Handles 5, 8, and 12-column CSV variants (memory/cycles optional) - Accepts stdin with '-' for composable pipeline use pqc_bench.sh: - Adds MEMORY MEASUREMENT APPROACH comment block documenting the Option A (built-in wolfSSL tracking) vs Option B (Valgrind massif) decision: Option A implemented, Option B deferred
Reproducibility documentation covering: - Prerequisites and tested environments (x86_64, aarch64) - Build and run instructions (copy-paste ready) - Output format reference for raw CSV and normalized formats - Memory measurement methodology and limitations - Timing methodology, frequency pinning, NUMA, thermal notes - Cross-library comparison with liboqs/PQC-LEO - Citation guidance
Add --input-format flag supporting wolfssl (existing), liboqs, openssl, and circl input formats, enabling a unified comparison CSV from any PQC library. All parsers normalise to the same canonical schema before the existing --format output stage. - liboqs: parses speed_kem/speed_sig fixed-width table output - openssl: parses 'openssl speed -mr' +R15..+R20 machine-readable lines - circl: parses 'go test -bench' ns/op output with GOMAXPROCS suffix stripping - wolfssl: existing parser retained unchanged, now selectable via --input-format
- liboqs 0.15 emits algorithm headers with trailing pipe fields; update regex to match and extract just the name token - liboqs 0.15 SLH-DSA names are SLH_DSA_PURE_SHA2_128S (underscores, PURE prefix); add normalisation rule to canonical SLH-DSA-SHA2-128s - OpenSSL grep-with-filename prefix stripped in parser (e.g. 'file:+R15') - liboqs _LIBOQS_ALG_RE now allows underscore in algorithm names
SLH_DSA_PURE_SHA2_128S doesn't match _PQC_ALGO_RE (^SLH-DSA) because the raw liboqs name uses underscores not hyphens. Normalise the candidate first, then test the normalised name.
- pqc_bench.sh: enable-slhdsa=yes,sha2 in configure flags; add
six SHA2 run_bench calls alongside the existing SHAKE set
- pqc_parse.py: add _WOLFSSL_SLHDSA_SHA2_RE pattern to normalise
wolfSSL SLH-DSA-SHA2-[SF] names to canonical SLH-DSA-SHA2-{bits}{sf}
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds a reproducible PQC benchmarking workflow for wolfSSL and a cross-library normalization tool, plus Rust wrapper helpers to support state-copy needs.
Changes:
- Added
pqc_bench.shto configure/build/run wolfSSL PQC benchmarks and emit a clean CSV. - Added
pqc_parse.pyto normalize wolfSSL/liboqs/OpenSSL/CIRCL outputs into a common schema (CSV or PQC-LEO PSV). - Added PQC benchmarking documentation (
README-pqc.md) and Rust SHA state-copy helpers.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
| wrapper/rust/wolfssl-wolfcrypt/src/sha.rs | Adds SHA256/SHA384 copy() helpers backed by wolfCrypt copy APIs |
| wolfcrypt/benchmark/pqc_parse.py | New Python normalizer/parser supporting multiple benchmark input formats |
| wolfcrypt/benchmark/pqc_bench.sh | New POSIX shell driver to build and run PQC benchmarks and assemble CSV |
| wolfcrypt/benchmark/README-pqc.md | Documents benchmark usage, output schema, and memory/timing methodology |
| AGENTS.md | Adds internal project/agent instructions and references for the benchmark work |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+211
to
+214
| def _detect_wolfssl_layout(header: list[str]) -> dict: | ||
| h = [f.strip().lower() for f in header] | ||
|
|
||
| def idx(name: str) -> Optional[int]: |
| return row[i].strip() | ||
|
|
||
|
|
||
| def parse_wolfssl(text: str, library: str) -> list[dict]: |
| _LIBOQS_ALG_RE = re.compile(r"^([A-Z][A-Z0-9_\-]+)\s*(\|.*)?$") | ||
|
|
||
|
|
||
| def parse_liboqs(text: str, library: str) -> list[dict]: |
| } | ||
|
|
||
|
|
||
| def parse_openssl(text: str, library: str) -> list[dict]: |
| _CIRCL_GOMAXPROCS_RE = re.compile(r"-\d+$") | ||
|
|
||
|
|
||
| def parse_circl(text: str, library: str) -> list[dict]: |
| # With GENERATE_MACHINE_PARSEABLE_REPORT compiled in, non-CSV lines get | ||
| # "###," or "!!!," prefixes. We redirect stderr to the log (it contains | ||
| # stack/heap summaries and verbose timing) and append stdout to RAW_CSV. | ||
| if ! "$BENCH" -csv "$@" >>"$RAW_CSV" 2>>"$LOG_FILE"; then |
Comment on lines
+648
to
+654
| pub fn copy(&self) -> Result<Self, i32> { | ||
| let mut dst: core::mem::MaybeUninit<sys::wc_Sha256> = core::mem::MaybeUninit::uninit(); | ||
| let rc = unsafe { sys::wc_Sha256Copy(&self.wc_sha256 as *const _ as *mut _, dst.as_mut_ptr()) }; | ||
| if rc != 0 { | ||
| return Err(rc); | ||
| } | ||
| Ok(SHA256 { wc_sha256: unsafe { dst.assume_init() } }) |
| |---|---|---| | ||
| | ML-KEM (CRYSTALS-Kyber) | FIPS 203 | ML-KEM-512, ML-KEM-768, ML-KEM-1024 | | ||
| | ML-DSA (CRYSTALS-Dilithium) | FIPS 204 | ML-DSA-44, ML-DSA-65, ML-DSA-87 | | ||
| | SLH-DSA (SPHINCS+) | FIPS 205 | All 10 SHAKE parameter sets (128s/f, 192s/f, 256s/f, and SHA-2 variants) | |
| brew install autoconf automake libtool | ||
| ``` | ||
|
|
||
| **Python 3.6+** is required to run `pqc_parse.py`. It is available by default |
Comment on lines
+103
to
+104
| usage: pqc_parse.py [-h] [--format {wolfssl,pqcleo}] [--library LIBRARY] | ||
| [--output OUTPUT] INPUT_CSV |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
wolfcrypt/benchmark/pqc_bench.sh: a self-contained benchmark driver that configures, builds, and runs wolfSSL for all NIST FIPS 203/204/205 algorithms (ML-KEM, ML-DSA, SLH-DSA-SHAKE, SLH-DSA-SHA2), producing a clean CSV output.wolfcrypt/benchmark/pqc_parse.py: normalises benchmark output from wolfSSL, liboqs, OpenSSL, and CIRCL into a common schema for cross-library comparison.wolfcrypt/benchmark/README-pqc.md: documents usage, output format, and memory measurement approach.wc_Sha256Copy/wc_Sha384Copyhelpers needed by the parser.Algorithms covered
ML-KEM-512/768/1024 (FIPS 203), ML-DSA-44/65/87 (FIPS 204),
SLH-DSA-SHAKE-{128,192,256}{s,f} and SLH-DSA-SHA2-{128,192,256}{s,f} (FIPS 205).
Cross-library results (x86_64, AWS c7i.2xlarge, Xeon 8488C)
wolfSSL vs liboqs vs OpenSSL vs CIRCL — keygen ops/sec representative sample:
wolfSSL is behind liboqs (assembly-optimised reference) and behind CIRCL on SHA2 ops. Competitive with or faster than OpenSSL across the board. On ARM/Graviton the gap to liboqs closes significantly.
Notes
pqc_bench.shuses wolfSSL's built-in--enable-trackmemory/--enable-stacksizefor heap and stack measurement. These are cumulative heap bytes and canary-based stack depth, not peak-RSS-per-op — see README for details.--enable-slhdsa=yes,sha2requiresautoreconfto have been run on a recent checkout; the generatedconfigurein older checkouts predates the comma-split loop for SHA2 tokens.