eval

Debug and test your agent prompts and tools

Evaluation is opt-in. Ordinary co ai sessions keep bounded records for recent debugging but make no scoring calls. Use co ai --eval to enable completion scoring.

What it does

The eval plugin helps you debug agents during development:

Generate Expected (after_user_input)

Generates what should happen once for each user turn, unless another plugin already set it for that turn.

Evaluate (on_complete)

After agent finishes, evaluates if the task was truly completed.

Quick Start

main.py
python
from connectonion import Agent
from connectonion.useful_plugins import eval

def calculate(expression: str) -> str:
    """Calculate a math expression."""
    return str(eval(expression))

agent = Agent("assistant", tools=[calculate], plugins=[eval])

agent.input("What is 25 * 4?")
output
[Expected: Should calculate 25 * 4 and return 100]
[Tool: calculate("25 * 4")]
Result: 100
/evaluating...
✓ Task complete: Calculated 25 * 4 = 100, which matches the expected result.

Want to customize? Run co copy eval to get an editable copy.

Combined with re_act

Use re_act for intent and reflection, then let eval judge the completed turn independently:

main.py
python
from connectonion import Agent
from connectonion.useful_plugins import re_act, eval

agent = Agent("assistant", tools=[search], plugins=[re_act, eval])

agent.input("Search for Python tutorials")
# re_act: Understands the request and reflects after tools
# eval: Generates this turn's expected outcome and evaluates completion

One evaluation per turn

A continued session keeps its earlier messages and trace entries as conversation context. Evaluation evidence is narrower: it starts at the current turn's user_input entry and includes only that turn's tool results and final response.

The expected outcome and verdict are also refreshed for every new user input. A tool used in an earlier request therefore cannot create a false pass or failure in the next one.

How it works

1. Generate Expected

main.py
python
@after_user_input
def generate_expected(agent):
    # Eval first refreshes its transient state for this Agent turn.
    # Preserve an expected outcome another plugin set for this turn.
    if agent.current_session.get('expected'):
        return

    user_prompt = agent.current_session.get('user_prompt', '')
    tool_names = agent.tools.names()

    expected = llm_do(
        f"User request: {user_prompt}\nTools: {tool_names}\nWhat should happen?",
        model="co/gemini-3.7-flash"
    )

    agent.current_session['expected'] = expected

2. Evaluate Completion

main.py
python
@on_complete
def evaluate_completion(agent):
    user_prompt = agent.current_session.get('user_prompt', '')
    result = agent.current_session.get('result', '')
    expected = agent.current_session.get('expected', '')
    trace = current_turn_trace(agent)

    # Summarize actions taken
    actions = [f"- {t['name']}: {t['result'][:100]}"
               for t in trace if t['type'] == 'tool_result']

    evaluation = llm_do(
        f"Request: {user_prompt}\nExpected: {expected}\n"
        f"Actions: {actions}\nResult: {result}\n"
        f"Is this complete?",
        model="co/gemini-3.7-flash"
    )

    agent.current_session['evaluation'] = evaluation
    agent.logger.print(f"✓ {evaluation}")

Events Used

Event	Handler	Purpose
`after_user_input`	generate_expected	Set expected outcome
`on_complete`	evaluate_completion	Evaluate if task complete

Use Cases

Development: Verify your agent completes tasks correctly
Testing: Automated evaluation of agent responses
Debugging: Identify incomplete or incorrect tool usage

Source

connectonion/useful_plugins/eval.py

main.py
python
# The plugin is just a list of event handlers
eval = [generate_expected, evaluate_completion]

Useful Plugins

re_act