Skip to content

Make result structs sendable for ASR - #46

Merged
BrandonWeng merged 1 commit into
mainfrom
sendable-asr
Jul 30, 2025
Merged

Make result structs sendable for ASR#46
BrandonWeng merged 1 commit into
mainfrom
sendable-asr

Conversation

@BrandonWeng

Copy link
Copy Markdown
Member

No description provided.

@github-actions

Copy link
Copy Markdown

Speaker Diarization Benchmark Results

Speaker Diarization Performance

Evaluating "who spoke when" detection accuracy

Metric Value Target Status Description
DER 18.7% <30% Diarization Error Rate (lower is better)
JER 22.6% <25% Jaccard Error Rate
RTF 0.05x <1.0x Real-Time Factor (lower is faster)

Diarization Pipeline Timing Breakdown

Time spent in each stage of speaker diarization

Stage Time (s) % Description
Model Download 0.000 0.0 Fetching diarization models
Model Compile 3.929 7.0 CoreML compilation
Audio Load 0.078 0.1 Loading audio file
Segmentation 13.615 24.2 Detecting speech regions
Embedding 38.572 68.6 Extracting speaker voices
Clustering 0.026 0.0 Grouping same speakers
Total 56.220 100 Full pipeline

Speaker Diarization Research Comparison

Research baselines typically achieve 18-30% DER on standard datasets

Method DER Notes
FluidAudio 18.7% On-device CoreML
Research baseline 18-30% Standard dataset performance

🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.4s meeting audio • 52.2s diarization time • Test runtime: 1m 21s • 07/29/2025, 07:22 PM EST

@github-actions

Copy link
Copy Markdown

VAD Benchmark Results

Performance Comparison

Dataset Accuracy Precision Recall F1-Score RTFx Files
MUSAN 100.0% 100.0% 100.0% 100.0% 13.4x faster 50
VOiCES 100.0% 100.0% 100.0% 100.0% 15.1x faster 50

Dataset Details

  • MUSAN: Music, Speech, and Noise dataset - standard VAD evaluation
  • VOiCES: Voices Obscured in Complex Environmental Settings - tests robustness in real-world conditions

EXCELLENT: Average F1-Score above 70%

@BrandonWeng
BrandonWeng enabled auto-merge (squash) July 29, 2025 23:28
@github-actions

Copy link
Copy Markdown

ASR Benchmark Results

Dataset WER Avg WER Med RTFx Status
test-clean 4.07% 0.00% 1.50x
test-other 8.26% 2.78% 1.44x

500 files per dataset • Test runtime: 42m36s • 07/29/2025, 08:04 PM EST

RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time
Processing time includes: Model inference on Apple Neural Engine, audio preprocessing, state resets between files, token-to-text conversion, and file I/O
Example: RTFx of 2.0x means 10 seconds of audio processed in 5 seconds (2x faster than real-time)

Expected RTFx Performance on Physical M1 Hardware:

• M1 Mac: ~28x (clean), ~25x (other)
• CI shows ~0.5-3x due to virtualization limitations

Testing methodology follows HuggingFace Open ASR Leaderboard

@BrandonWeng
BrandonWeng merged commit c643a76 into main Jul 30, 2025
5 checks passed
@BrandonWeng
BrandonWeng deleted the sendable-asr branch July 30, 2025 00:09
Alex-Wengg pushed a commit that referenced this pull request Jan 1, 2026
SGD2718 pushed a commit that referenced this pull request Jan 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants