fix(openai-chat): deliver tool-result images to vision models - #912
fix(openai-chat): deliver tool-result images to vision models#912DevMello wants to merge 3 commits into
Conversation
|
✅ PR quality gates passed This pull request now targets The title was left unchanged. The pull request has been marked ready for review again. |
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughThe OpenAI Chat adapter forwards tool-result images in a follow-up user vision message after the tool round. Tool messages retain text and an ChangesTool-result image forwarding
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant ToolResult
participant OpenAIChatAdapter
participant OpenAIChatRequest
ToolResult->>OpenAIChatAdapter: Provide text and image content
OpenAIChatAdapter->>OpenAIChatRequest: Emit text-only tool message with [image]
OpenAIChatAdapter->>OpenAIChatRequest: Emit follow-up user vision message with image_url parts
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs-site/src/content/docs/reference/adapters.md`:
- Around line 34-36: Add the tool-result image behavior to the corresponding
openai-chat sections in both translated adapter pages: document that the
text-only role:"tool" message retains a short [image] marker, while the actual
images are delivered as image_url parts in one follow-up user vision message
after the tool round closes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 839db9c6-9a30-4061-84de-2b79886cba9c
📒 Files selected for processing (5)
docs-site/src/content/docs/ja/reference/adapters.mddocs-site/src/content/docs/ko/reference/adapters.mddocs-site/src/content/docs/reference/adapters.mdsrc/adapters/openai-chat.tstests/openai-chat-tool-result-images.test.ts
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a719509eaf
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const parts: unknown[] = []; | ||
| for (const p of content) { | ||
| if (p.type !== "image") continue; | ||
| parts.push({ type: "image_url", image_url: { url: p.imageUrl, ...(p.detail ? { detail: p.detail } : {}) } }); |
There was a problem hiding this comment.
When a tool output contains a malformed input_image with an empty image_url (the tool-output parser currently accepts any string here, including the image_url: "", file_id: "..." shape that user input omits), this new carrier forwards {"image_url":{"url":""}} to the chat provider. Vision-capable OpenAI-compatible endpoints reject malformed image_url parts, so a bad tool image that previously degraded to the [image] marker can now fail the whole request; skip non-usable URLs before adding carrier parts.
AGENTS.md reference: src/AGENTS.md:L17-L19
Useful? React with 👍 / 👎.
Images in a tool_result were flattened to a literal "[image]" marker, so vision-capable routed models (kimi-code, DeepSeek, ...) described images they never saw. role:"tool" content is text-only on chat providers, so images now ride in a follow-up user vision message released once the tool round closes, mirroring the Google and Kiro adapters. Models in noVisionModels still use the vision sidecar. Fixes lidge-jun#888
a719509 to
9c8a0c5
Compare
The tool-output parser accepts any string for image_url, including
the empty file_id shape, and a {"url":""} carrier part would fail
the whole request where the "[image]" marker degrades safely.
The first commit only updated the en, ja, and ko adapter pages.
Summary
Fixes #888. Images in a tool result were flattened to the literal string
"[image]", so vision models like kimi-k3 never saw the actual image and made up a description (Claude Code sends Read and screenshot output this way).role:"tool"messages can only carry text on chat providers, so the adapter now re-sends the image parts in a user vision message once the tool round closes, the same approach the Google and Kiro adapters already use. Models innoVisionModelsstill go through the vision sidecar.Also added a line to the adapter docs. The Japanese, Korean, Russian, and Chinese versions were done with AI translation, so a quick check from native speakers would help.
Verification
bun run test(includes the newtests/openai-chat-tool-result-images.test.ts),typecheck,lint:gui,privacy:scan."[image]".Checklist
Notes
Rebased onto current
dev(was accidentally based on thedev/mainrelease-promotion point for v2.10.0). Source commit before the rebase: a719509.Summary by CodeRabbit
New Features
[image]marker.Documentation