DocsUseful Pluginsimage_result_formatter

image_result_formatter

Enable vision models to see images from tool results

What it does

When a tool returns a base64-encoded image (screenshot, generated image, etc.), this plugin:

Detects base64 images

Recognizes data URLs (data:image/png;base64,...) and plain base64 strings.

Uploads to oo-api, keeps a short URL

Image bytes go to oo-api (content-addressed storage); the message history keeps only a ~70-byte /img URL, so screenshots never bloat the replayed context. Requires OPENONION_API_KEY (set up by co init).

Converts to vision format

Transforms the tool result into OpenAI vision API format so the LLM can see the image visually.

Quick Start

main.py
python
from connectonion import Agent
from connectonion.useful_plugins import image_result_formatter

def take_screenshot(url: str) -> str:
    """Take a screenshot of a webpage."""
    # Returns base64-encoded PNG
    return capture_screenshot(url)

agent = Agent(
    "vision_assistant",
    tools=[take_screenshot],
    plugins=[image_result_formatter],
    model="gpt-4o"  # Use a vision model
)

agent.input("Take a screenshot of example.com and describe what you see")
output
[Tool: take_screenshot("example.com")]
🖼️  Formatted 'take_screenshot' result as image
The screenshot shows a simple webpage with the heading "Example Domain"...

Want to customize? Run co copy image_result_formatter to get an editable copy.

Without vs With Plugin

Without Plugin

The LLM receives raw base64 text:

Tool result: "iVBORw0KGgoAAAANSUhEUgAAA..."

LLM cannot interpret this as an image!

With Plugin

The LLM receives a proper image message with a short URL:

{type: "image_url", url: "https://oo.openonion.ai/img/a3f9..."}

LLM can see the image, and the history stays ~70 bytes per screenshot!

How it works

main.py
python
def _format_image_result(agent):
    """Runs on after_tools: replace base64 tool results with an image message."""
    for trace_entry in agent.current_session['trace']:
        is_image, mime_type, base64_data = _is_base64_image(trace_entry.get('result'))
        if not is_image:
            continue

        # Upload bytes to oo-api; only a short URL enters the history
        image_url = _upload_to_oo_api(base64_data, mime_type)

        # Shorten the tool message (save tokens)
        tool_msg['content'] = "Tool returned an image (provided below)"

        # Insert a user message the vision model can see
        messages.insert(tool_index + 1, {
            "role": "user",
            "content": [
                {"type": "text", "text": f"Here is the image from '{tool_name}':"},
                {"type": "image_url", "image_url": {"url": image_url}}
            ]
        })

        if agent.io:
            agent.io.send_image(image_url)  # real-time display in oo-chat


def _upload_to_oo_api(base64_data, mime_type):
    """POST bytes to oo-api, get back a stable /img URL."""
    base = os.getenv("OPENONION_API_URL", "https://oo.openonion.ai")
    resp = requests.post(
        f"{base}/api/v1/images",
        headers={"Authorization": f"Bearer {os.environ['OPENONION_API_KEY']}"},
        files={"file": ("screenshot", base64.b64decode(base64_data), mime_type)},
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["url"]

Supported Formats

Format	Example
PNG	`data:image/png;base64,...`
JPEG	`data:image/jpeg;base64,...`
WebP	`data:image/webp;base64,...`
GIF	`data:image/gif;base64,...`
Plain base64	Long base64 strings (defaults to PNG)

Use Cases

Screenshots: Browser automation tools that capture pages
Image generation: Tools that create images (charts, diagrams)
Visual analysis: Any tool returning visual data for LLM interpretation

Events Used

Event	Handler	Purpose
`after_tools`	_format_image_result	Convert base64 to vision format

Uses after_tools (not after_each_tool) because message modification must happen after all tools complete.

Source

connectonion/useful_plugins/image_result_formatter.py

main.py
python
# The plugin is just a list with one event handler
image_result_formatter = [after_tools(_format_image_result)]

Useful Plugins

eval