memory: SQLite + FTS5 keyword index over workspace/memory/ (Phase A)

[FMXExpress]

Phase A of the hybrid memory system. Mirrors openclaw's model: durable notes live as Markdown files (MEMORY.md, workspace/memory/<date>.md) that the model edits with the existing fs_write tool. SQLite is a derived index — never the source of truth, lazy-rebuilt on each memory_search call by walking mtimes. No embeddings, no memory_add tool, no ILLMProvider.Embed — all deferred to Phase B.

What's in the box

1. PasClaw.Memory.Index — cross-target SQLite + FTS5 index

IMemoryIndex interface with Open / Close / SyncDir / Search. Two backends gated by IFDEF:

Target	Driver	Build dep	Runtime dep
FPC	TSQLite3Connection + TSQLQuery (sqldb / sqlite3conn)	fcl-db (apt: fp-units-db)	libsqlite3.so
Delphi	TFDConnection + TFDQuery (FireDAC SQLite)	ships with Delphi	sqlite3.dll

If libsqlite3 can't be loaded, Open returns False and memory_search degrades to "index unavailable" — the rest of the agent keeps running.

Schema:

CREATE TABLE memory_files (
  rowid INTEGER PRIMARY KEY AUTOINCREMENT,
  path TEXT UNIQUE NOT NULL,
  mtime INTEGER NOT NULL,
  indexed_at INTEGER NOT NULL
);
CREATE VIRTUAL TABLE memory_fts USING fts5(
  path UNINDEXED, content,
  tokenize='porter unicode61'
);

Each file is one FTS5 row sharing rowid with memory_files. The reindex path deletes both rows by path and reinserts, so rowids stay in sync across edits.

SyncDir walks workspace/memory/MEMORY.md + workspace/memory/*.md, compares mtimes, reindexes whatever's newer, and drops rows for files that disappeared from disk.

Search runs MATCH and returns up to K hits with the FTS5 snippet() excerpt and bm25() score (smaller = better).

2. PasClaw.Tools.Memory — memory_search tool

{
  "name": "memory_search",
  "description": "Search the workspace memory directory (MEMORY.md + workspace/memory/*.md) with SQLite FTS5 BM25 ranking…",
  "parameters": {
    "type": "object",
    "properties": {
      "query": {"type": "string"},
      "k":     {"type": "integer", "minimum": 1, "maximum": 25}
    },
    "required": ["query"]
  }
}

Opens $PASCLAW_HOME/workspace/memory/.index.db, syncs against the filesystem, runs the FTS5 query, returns up to 25 hits as path (bm25=N.NNN) + snippet. Missing memory dir surfaces a friendly (no memory directory yet — write to MEMORY.md first) instead of an error.

No memory_add tool by design. Model writes durable memories by editing files with fs_write — same convention openclaw uses.

3. BuildMemorySection extension

The system prompt's ## Memory section used to inject only MEMORY.md. It now also injects today's and yesterday's daily notes (if either exists), wrapped under ### Today (<date>) / ### Yesterday (<date>) subsections. Mirrors openclaw's bootstrap loading; older daily notes stay on disk and reach the model only when memory_search returns them.

4. Rules update (rule 5)

Teaches the model to:

• write durable preferences/project facts to MEMORY.md,
• write episodic session context to workspace/memory/<today>.md,
• call memory_search before answering questions about prior conversations.

5. Tool registration wired into every site

RegisterMemoryTools added to PasClaw.Cmd.Agent / Gateway / TUI / Serve and both TPasClawAgent.EnsureToolsRegistered paths in PasClaw.Component. Honours --no-tools (no registry → no memory_search).

Build

Makefile adds FCLDB_DIR / SQLITE_DIR (defaulting to Debian fp-units-db paths) plus an optional LAZUTILS_DIR so Tools.FS.Masks keeps resolving on environments where it lives in Lazarus rather than fcl-base. PasClaw.dproj gets DCCReference entries for the two new units.

Verification

• make — clean build under FPC 3.2.2.
• Standalone smoke test: open / SyncDir / Search hits "pascal" → top hit is MEMORY.md / FTS5 search hits "FTS5" / reindex picks up new term after rewrite / removed file drops from index. 7 assertions, all pass.
• pasclaw --help still lists every existing command unchanged.

Deferred to Phase B

• vec0 + embeddings + hybrid RRF retrieval (sqlite-vec loadable extension + ILLMProvider.Embed).
• openclaw's "memory flush" pre-compaction prompt (PasClaw has no compaction yet).
• "Dreaming" consolidation pass.
• Eviction / decay / importance scoring.

https://claude.ai/code/session_01TBcLtmpj7dqA5tyFbGnQon

---

Generated by Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 78e29a57fc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Escape natural-language queries before MATCH

When memory_search is called with ordinary prior-conversation questions or code/path tokens containing FTS punctuation (for example what did we discuss?, C++, or foo/bar), SQLite FTS5 raises a syntax error; this method catches that and returns an empty result, so the tool reports no matches even when indexed notes contain the terms. Since the prompt tells the model to use this tool for natural-language memory lookups, sanitize/tokenize the query or fall back to a quoted/ANDed term query before passing it to MATCH (the same raw binding is duplicated in the FireDAC branch).

Useful? React with 👍 / 👎.