Skip to content

Add masked BLEU and RTF result audits#1

Draft
luojiaxuan wants to merge 2 commits into
mainfrom
agent/add-masked-bleu-results
Draft

Add masked BLEU and RTF result audits#1
luojiaxuan wants to merge 2 commits into
mainfrom
agent/add-masked-bleu-results

Conversation

@luojiaxuan

@luojiaxuan luojiaxuan commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add the target-term-masked BLEU computation path to the offline StreamLAAL evaluator and a release aggregation script.
  • Publish the masked-term BLEU TSVs, RASST-vs-InfiniSST comparison table, artifact map, and PDF/PNG figure under docs/results/main_result_global_cache30_30_20_20/.
  • Add the ACL tagged En-Zh InfiniSST validation TSV backfill used by the artifact map and document the audit in the root/result READMEs.
  • Publish the RAG compute RTF package under docs/results/rag_compute_rtf/, including frozen TSV data, PDF/PNG figure, plotting script, provenance README, and root README link.

Validation

  • python3 -m py_compile code/rasst/eval/offline_sst_eval/offline_streamlaal_eval.py code/rasst/analysis/main_result/compute_masked_terms_quality.py code/rasst/analysis/main_result/plot_masked_terms_bleu.py
  • python3 -m py_compile docs/results/rag_compute_rtf/plot_rag_compute_rtf.py
  • sha256sum for the new masked-BLEU TSVs/figures and RTF package matches the checksums recorded in the result READMEs.
  • masked_terms_quality.tsv contains 48 ok rows, with 4 rows for each dataset/method/language group.
  • docs/results/rag_compute_rtf/data.tsv contains LM 1-4 rows with median RAG RTF values 3.8497%, 2.2055%, 1.4778%, and 1.1366%.
  • git diff --cached --check passed before the RTF commit; generated TSV line endings were normalized to LF.

Note: raw *.log hypothesis/runtime files remain ignored by repo policy; the uploaded TSVs retain their artifact pointers and computed metrics.

@luojiaxuan luojiaxuan changed the title Add masked BLEU result audit Add masked BLEU and RTF result audits Jul 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant