[FEAT]: Batch LLM extraction to reduce N API calls to 1

[Acuspeedster]

---

name: 🚀 Feature Request
about: Suggest an idea or a new capability for FireForm.
title: "[FEAT]: Batch LLM extraction to reduce N API calls to 1"
labels: enhancement
assignees: ''

📝 Description

Currently, LLM.main_loop() makes one Ollama API call per form field.

For a form with N fields, this results in:

• N prompts
• N network round-trips
• N model inference cycles
• N JSON parses

This creates a scalability bottleneck, especially for multi-form submissions.

We propose adding a batch extraction method that sends all target fields in a single structured prompt and parses a single JSON response.

---

💡 Rationale

At scale, sequential per-field API calls significantly increase latency and infrastructure load.

Example:

• 1 form × 10 fields → 10 LLM calls
• 3 forms × 10 fields → 30 LLM calls

Reducing this to a single API call per form would:

• Improve performance
• Reduce response time
• Lower model inference overhead
• Improve scalability for multi-agency submissions

---

🛠️ Proposed Solution

Introduce a new method, e.g., LLM.main_loop_batch():

• Build one structured prompt containing:

  • All target fields
  • Transcript/input text

• Instruct the model to return strict JSON

• Parse once

• Strip markdown code fences (```json) if present

• Handle null and list values

• Gracefully fall back to main_loop() if JSON parsing fails

• Logic change in src/

• New prompt for Mistral/Ollama

• Unit tests in tests/test_llm.py

• Update src/filler.py to use batch method

---

✅ Acceptance Criteria

• Exactly 1 API call per form regardless of field count
• Batch method correctly populates all fields
• Handles null and list values correctly
• Gracefully falls back to sequential method if invalid JSON is returned
• Feature works in Docker container
• Documentation updated in docs/
• JSON output validates against the schema

---

📌 Additional Context

Before:
N fields → N API calls
After:
N fields → 1 API call

This change significantly improves performance while maintaining backward compatibility.

[Acuspeedster]

#103 implements this.

[Acuspeedster]

@marcvergees if you could review the pr I initiated it would be gr8...
Also people have been making redundant Prs to this issue.

[Acuspeedster]

@marcvergees do see this as well where @utkarshqz has been making redundant prs.

[utkarshqz]

Hi @Acuspeedster — I want to address this on technical grounds.

PR #153 takes a concurrent approach — firing all field extractions simultaneously via asyncio.as_completed(), bounding total latency to the slowest single call. That is a genuinely valuable improvement.

PR #210 takes a complementary approach at a different layer — a single batch prompt extracts all fields in one LLM call, eliminating all N round-trips entirely. Both address the latency problem with different tradeoffs worth comparing.

PR #210 is also not a standalone extraction patch. It delivers the complete FireForm pipeline working end-to-end for the first time — browser frontend, CORS, GET /forms/download, human-readable PDF labels, and batch extraction — in a tightly coupled dependency chain that could not be submitted as isolated PRs without leaving the codebase in a broken intermediate state.

I acknowledge I should have commented on the existing issue before opening a PR — I'll do that going forward.

Happy for @marcvergees @vharkins1 — to decide which approach to proceed with.

[utkarshqz]

@marcvergees do see this as well where @utkarshqz has been making redundant prs.

@Acuspeedster
I saw that batching was a community goal (#102/#103), but I realized that a simple intra-form fix wouldn't scale for responders who need to fill multiple agency forms (Police/Fire/EMS) from one report. I developed a more robust O(1) multi-template engine that includes human-label context for better LLM accuracy and a fallback mechanism to prevent failures. It's an evolution of the initial batching idea into a production-ready API.

redundant ?
Code Structure: My code (#241 ) in llm.py and forms.py uses different patterns, a dedicated endpoint, and a different prompt construction (the // label technique).

Architectural Scope: PR #103 only solves the "N calls for 1 PDF" problem. While PR #241 solves the "1 call for N different PDFs" problem.

also , #241 uses advanced prompt engineering (labels vs. raw keys), and includes a fault-tolerant fallback.

Feature	PR #103	PR #241
Logic Type	Intra-form (1 JSON for 1 PDF)	Inter-form (1 JSON for N PDFs)
Prompt Style	Bare programming keys (e.g. dob)	Human Labels (e.g. // Date of Birth)
Fail-Safe	None. If JSON fails, it errors.	Legacy Fallback to single-field extraction.
API Impact	Internal refactor only.	New /forms/fill/batch endpoint.