[PR #57] [MERGED] Fix Anthropic tool results containing images not being passed to the model #70

New issue

Closed

opened 2026-02-27 07:17:49 +03:00 by kerem · 0 comments

kerem commented

2026-02-27 07:17:49 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/jwadow/kiro-gateway/pull/57
Author: @Ry-DS
Created: 1/27/2026
Status: ✅ Merged
Merged: 2/2/2026
Merged by: @jwadow

Base: main ← Head: fix/images-in-tool-results

📝 Commits (3)

372dec8 fix(anthropic): extract images from tool_result content blocks
6340804 fix(openai): extract images from tool messages for MCP screenshot support
970fee2 Revert "fix(openai): extract images from tool messages for MCP screenshot support"

📊 Changes

3 files changed (+950 additions, -442 deletions)

View changed files

📝 kiro/converters_anthropic.py (+142 -84)
📝 kiro/models_anthropic.py (+73 -30)
📝 tests/unit/test_converters_anthropic.py (+735 -328)

📄 Description

I use browsermcp with claude code. This MCP server returns images which break CC and the API since it passes the raw base64 content as text to the model when it should be treated as images.

See the fail log:

{
  "status_code": 422,
  "error_message": "Validation error: [{'type': 'string_type', 'loc': ['body', 'messages', 6, 'content', 'str'], 'msg': 'Input should be a valid string', 'input': [{'tool_use_id': 'tooluse_9og_DFeUQembqH8I5DuaJA', 'type': 'tool_result', 'content': [{'type': 'image', 'source': {'data': '..snipped, base 64 content is here..', 'media_type': 'image/png', 'type': 'base64'}}}]"
}

This PR fixes that, recognising images inside the tool result msg.

2026-01-27 16:10:32 | INFO     | kiro.routes_anthropic:messages:144 - Request to /v1/messages (model=claude-sonnet-4-5-20250929, stream=True)
2026-01-27 16:10:32 | DEBUG    | kiro.routes_anthropic:messages:147 - Anthropic-Version header: 2023-06-01
2026-01-27 16:10:32 | DEBUG    | kiro.converters_core:extract_images_from_content:262 - Extracted 1 image(s) from content
2026-01-27 16:10:32 | DEBUG    | kiro.converters_anthropic:extract_images_from_tool_results:199 - Extracted 1 image(s) from tool_result content
2026-01-27 16:10:32 | DEBUG    | kiro.converters_anthropic:convert_anthropic_messages:330 - Converted 5 Anthropic messages: 1 tool_calls, 1 tool_results, 1 images

I also noticed there's no formatter; share the formatter you use on main and I'll run it on this branch

I didn't try too hard to make it work for openai also. Lmk if you'd like me to try figure it out for OpenAI also via opencode + browsermcp.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/jwadow/kiro-gateway/pull/57 **Author:** [@Ry-DS](https://github.com/Ry-DS) **Created:** 1/27/2026 **Status:** ✅ Merged **Merged:** 2/2/2026 **Merged by:** [@jwadow](https://github.com/jwadow) **Base:** `main` ← **Head:** `fix/images-in-tool-results` --- ### 📝 Commits (3) - [`372dec8`](https://github.com/jwadow/kiro-gateway/commit/372dec8ab3b1b952268d7414081910028f62b7c5) fix(anthropic): extract images from tool_result content blocks - [`6340804`](https://github.com/jwadow/kiro-gateway/commit/63408042e86786f073318f31e0a27166a24966ec) fix(openai): extract images from tool messages for MCP screenshot support - [`970fee2`](https://github.com/jwadow/kiro-gateway/commit/970fee24e988023f3a6a460cc2fbc7636f9ae94f) Revert "fix(openai): extract images from tool messages for MCP screenshot support" ### 📊 Changes **3 files changed** (+950 additions, -442 deletions) <details> <summary>View changed files</summary> 📝 `kiro/converters_anthropic.py` (+142 -84) 📝 `kiro/models_anthropic.py` (+73 -30) 📝 `tests/unit/test_converters_anthropic.py` (+735 -328) </details> ### 📄 Description I use browsermcp with claude code. This MCP server returns images which break CC and the API since it passes the raw base64 content as text to the model when it should be treated as images. See the fail log: ```json { "status_code": 422, "error_message": "Validation error: [{'type': 'string_type', 'loc': ['body', 'messages', 6, 'content', 'str'], 'msg': 'Input should be a valid string', 'input': [{'tool_use_id': 'tooluse_9og_DFeUQembqH8I5DuaJA', 'type': 'tool_result', 'content': [{'type': 'image', 'source': {'data': '..snipped, base 64 content is here..', 'media_type': 'image/png', 'type': 'base64'}}}]" } ``` This PR fixes that, recognising images inside the tool result msg. <img width="1870" height="1010" alt="image" src="https://github.com/user-attachments/assets/e1bce55f-6d02-48d7-bd3d-9714ebd5b2bf" /> ``` 2026-01-27 16:10:32 | INFO | kiro.routes_anthropic:messages:144 - Request to /v1/messages (model=claude-sonnet-4-5-20250929, stream=True) 2026-01-27 16:10:32 | DEBUG | kiro.routes_anthropic:messages:147 - Anthropic-Version header: 2023-06-01 2026-01-27 16:10:32 | DEBUG | kiro.converters_core:extract_images_from_content:262 - Extracted 1 image(s) from content 2026-01-27 16:10:32 | DEBUG | kiro.converters_anthropic:extract_images_from_tool_results:199 - Extracted 1 image(s) from tool_result content 2026-01-27 16:10:32 | DEBUG | kiro.converters_anthropic:convert_anthropic_messages:330 - Converted 5 Anthropic messages: 1 tool_calls, 1 tool_results, 1 images ``` I also noticed there's no formatter; share the formatter you use on main and I'll run it on this branch I didn't try too hard to make it work for openai also. Lmk if you'd like me to try figure it out for OpenAI also via opencode + browsermcp. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>