Skip to content

fix: filter internal prompts from chat capture observations #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wjiuxing wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix: filter internal prompts from chat capture observations #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3770626
fix: filter internal prompts from chat capture observations
May 27, 2026
3b2c790
test: add coverage for entity_extraction and rerank_request patterns
May 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions src/hooks/chat-capture.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,24 @@ const MIN_MESSAGE_LENGTH = 20;
const MAX_NARRATIVE_LENGTH = 2000;
const MAX_TITLE_CONTENT_LENGTH = 60;

/**
* Patterns that identify internal open-mem prompts injected into the chat
* stream. These should NOT be captured as user observations because they are
* implementation artifacts, not user intent.
*/
const INTERNAL_PROMPT_PATTERNS: RegExp[] = [
// The observation extraction prompt (compressor → chat.message)
/^\s*<task>\s*\n?\s*Analyze the following tool output and extract a structured observation/i,
// The session summarization prompt
/^\s*<task>\s*\n?\s*Summarize the following coding session based on its observations/i,
// The conflict evaluation prompt
/^\s*<conflict_evaluation>/i,
// The entity extraction prompt
/^\s*<entity_extraction>/i,
// The reranking prompt
/^\s*<rerank_request>/i,
];

/**
* Type guard: checks whether a value is an object with a string `text` property.
*/
Expand Down Expand Up @@ -77,6 +95,9 @@ export function persistChatMessage(input: ChatCaptureInput): boolean {
const processedText = redactSensitive(stripPrivateBlocks(text), sensitivePatterns);
if (processedText.length < MIN_MESSAGE_LENGTH) return false;

// Filter out internal open-mem prompts that leak into the chat stream
if (INTERNAL_PROMPT_PATTERNS.some((p) => p.test(processedText))) return false;

sessions.getOrCreate(sessionId, projectPath);

const truncatedContent =
Expand Down
141 changes: 141 additions & 0 deletions tests/hooks/chat-capture.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,147 @@ describe("createChatCaptureHook", () => {
expect((data.filesModified as string[]).length).toBe(0);
});

test("filters out internal observation extraction prompts", async () => {
const observations = makeMockObservations();
const sessions = makeMockSessions();
const hook = createChatCaptureHook(observations as never, sessions as never, "/tmp/proj");

await hook(
{ sessionID: "s1" },
{
message: {},
parts: [
`<task>
Analyze the following tool output and extract a structured observation.
</task>

<tool_name>bash</tool_name>

<tool_output>
Some tool output here that is definitely long enough to pass the length check
</tool_output>`,
],
},
);

expect(observations.calls).toHaveLength(0);
});

test("filters out internal summarization prompts", async () => {
const observations = makeMockObservations();
const sessions = makeMockSessions();
const hook = createChatCaptureHook(observations as never, sessions as never, "/tmp/proj");

await hook(
{ sessionID: "s1" },
{
message: {},
parts: [
`<task>
Summarize the following coding session based on its observations.
</task>

<session_id>ses_abc123</session_id>

<observations>
<obs index="1">
<title>Some observation</title>
</obs>
</observations>`,
],
},
);

expect(observations.calls).toHaveLength(0);
});

test("filters out internal conflict evaluation prompts", async () => {
const observations = makeMockObservations();
const sessions = makeMockSessions();
const hook = createChatCaptureHook(observations as never, sessions as never, "/tmp/proj");

await hook(
{ sessionID: "s1" },
{
message: {},
parts: [
`<conflict_evaluation>
<new_observation>
<title>Some title</title>
</new_observation>
</conflict_evaluation>`,
],
},
);

expect(observations.calls).toHaveLength(0);
});

test("filters out internal entity extraction prompts", async () => {
const observations = makeMockObservations();
const sessions = makeMockSessions();
const hook = createChatCaptureHook(observations as never, sessions as never, "/tmp/proj");

await hook(
{ sessionID: "s1" },
{
message: {},
parts: [
`<entity_extraction>
<observation>
<title>Some title</title>
<type>discovery</type>
<narrative>Some narrative text about entities and relationships</narrative>
</observation>
</entity_extraction>`,
],
},
);

expect(observations.calls).toHaveLength(0);
});

test("filters out internal reranking prompts", async () => {
const observations = makeMockObservations();
const sessions = makeMockSessions();
const hook = createChatCaptureHook(observations as never, sessions as never, "/tmp/proj");

await hook(
{ sessionID: "s1" },
{
message: {},
parts: [
`<rerank_request>
<query>some search query about authentication patterns</query>
<candidates>
<candidate index="0"><title>Auth middleware</title></candidate>
</candidates>
</rerank_request>`,
],
},
);

expect(observations.calls).toHaveLength(0);
});

test("does not filter normal user messages that happen to contain XML-like tags", async () => {
const observations = makeMockObservations();
const sessions = makeMockSessions();
const hook = createChatCaptureHook(observations as never, sessions as never, "/tmp/proj");

await hook(
{ sessionID: "s1" },
{
message: {},
parts: [
"Please help me debug this HTML issue with <div> tags and also fix the layout",
],
},
);

expect(observations.calls.find((c) => c.method === "create")).toBeDefined();
});
Comment on lines +358 to +374

@greptile-apps greptile-apps Bot May 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Missing test coverage for entity_extraction and rerank_request patterns

The PR adds 5 patterns to INTERNAL_PROMPT_PATTERNS but only 3 of the 5 are covered by the new "filters out…" tests. The <entity_extraction> and <rerank_request> patterns have no corresponding filter tests. If either pattern were accidentally removed or broken in a future refactor, nothing would catch the regression.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: tests/hooks/chat-capture.test.ts
Line: 311-327

Comment:
**Missing test coverage for `entity_extraction` and `rerank_request` patterns**

The PR adds 5 patterns to `INTERNAL_PROMPT_PATTERNS` but only 3 of the 5 are covered by the new "filters out…" tests. The `<entity_extraction>` and `<rerank_request>` patterns have no corresponding filter tests. If either pattern were accidentally removed or broken in a future refactor, nothing would catch the regression.

How can I resolve this? If you propose a fix, please make it concise.


test("handles mixed string and object parts", async () => {
const observations = makeMockObservations();
const sessions = makeMockSessions();
Expand Down