feat: libscope/lite — embeddable semantic search library (#451)

[RobertLD]

Summary

• Introduces LibScopeLite, exported from libscope/lite, for embedding semantic search directly into external apps (e.g. Bitbucket MCP) without spawning a subprocess or HTTP server
• Adds tree-sitter code-aware chunker (src/lite/chunker-treesitter.ts) supporting TypeScript, JavaScript, Python, and Go — tree-sitter is an optional peer dependency
• Adds input normalization layer (src/lite/normalize.ts) dispatching HTML/PDF/DOCX/plaintext to existing parsers
• 49 new tests (18 unit/chunker, 21 unit/lite, 10 integration) — 1492 total passing

API

const lite = new LibScopeLite({ dbPath: ':memory:', provider })
await lite.indexBatch(repoFiles, { concurrency: 4 })
const context = await lite.getContext('How does auth work?')

Test plan

• npm run format:check — clean
• npm run lint — 40 errors (baseline, no new errors)
• npm run typecheck — no new errors
• npm test — 1492/1492 passing
• npm run build — dist/lite/index.js present
• SonarCloud quality gate (CI)

Root cause note

MockEmbeddingProvider.hashToVector had a zero-hash collapse bug for long strings (200+ chars) that produced NaN vectors. sqlite-vec returns null distance for NaN vectors, causing DatabaseError in vectorSearch. Fixed with a non-zero seed and zero-magnitude guard.

Closes #451

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
libscope	Ignored	Preview	Mar 19, 2026 5:29pm

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.7% Duplication on New Code

See analysis details on SonarQube Cloud

[RobertLD]

Reviewed the libscope/lite implementation. Four critical bugs found — three in production code, one off-by-one in line number output.

[RobertLD]

Bug: indexBatch silently discards errors.

void this.index([doc]).finally(...) drops any rejection from this.index(). The .finally() callback receives no argument — it cannot tell whether the preceding promise succeeded or failed. The Promise<void> wrapping this has no reject path, so if any document fails to embed/index, indexBatch still resolves successfully, silently losing the failure.

The fix is to pass a reject callback to the outer Promise constructor and call it inside .finally() (or use .then(runNext, (err) => reject(err))). At minimum the error should be surfaced:

await new Promise<void>((resolve, reject) => {
  const runNext = (): void => {
    while (activeCount < concurrency && idx < docs.length) {
      const doc = docs[idx];
      if (!doc) break;
      idx++;
      activeCount++;
      void this.index([doc]).then(
        () => {
          activeCount--;
          if (idx >= docs.length && activeCount === 0) resolve();
          else runNext();
        },
        (err: unknown) => reject(err),
      );
    }
  };
  runNext();
});

[RobertLD]

Bug: indexRaw returns "" (empty string) as a document ID when the first chunk fails.

firstId is initialised to "" and only set on i === 0. If indexDocument throws on the first chunk, the for loop exits via the thrown exception (which propagates correctly), but if it throws on a later chunk the function has already assigned a real ID and this path is fine. The real problem is a subtler semantic issue: if normalized.chunks has length > 1 but the loop body somehow completes without entering if (i === 0) — impossible today but fragile — "" is returned as a valid ID. More importantly, the API contract says indexRaw returns the document ID of the newly created document, but when multiple chunks are indexed they each get their own document ID. Only the first is returned; the others are silently unreachable by the caller. This is a data loss bug for multi-chunk files: the caller cannot rate, retrieve, or delete the documents for chunks 2–N.

Consider either (a) returning all IDs as string[], or (b) indexing all chunks as a single document with pre-split content.

[RobertLD]

Race condition: shared mutable parser state across concurrent chunk() calls.

parserCache holds a single TSParser instance. parser.setLanguage(grammar) on line 128 mutates the parser's active grammar. If chunk() is called concurrently for two different languages (e.g., "typescript" and "python"), the sequence can be:

1. Coroutine A: getParser() → returns cached parser
2. Coroutine B: getParser() → returns same cached parser
3. Coroutine A: parser.setLanguage(typescriptGrammar)
4. Coroutine B: parser.setLanguage(pythonGrammar) ← overwrites A's language
5. Coroutine A: parser.parse(tsSource) ← parsed with Python grammar → wrong AST

This is realistic when indexBatch runs concurrency > 1 and the batch contains mixed-language files that go through indexRaw→normalizeRawInput→chunker.chunk().

The parser must either be cloned/re-created per chunk() call, or language be serialised (one active language at a time), or one parser instance be kept per language.

[RobertLD]

Off-by-one: preamble endLine is 0-based while every other endLine in the same chunk array is 1-based.

endLine: child.startPosition.row,   // line 223 — missing +1

Every other place in this file converts row to a 1-based line number with row + 1 (lines 184, 194, 214, 244, 254, 265, 269, 280). Only the preamble chunk emitted inside the large-node path uses the raw 0-based row value. For example, if the preamble ends just before a class starting at row 10 (1-based line 11), the preamble chunk will report endLine: 10 instead of endLine: 11, making the range appear to end one line early and leaving line 11 unaccounted for in any line-range display or navigation built on top of these chunks.