Definition
AI agent evaluation examines a trajectory through state and tools
Unlike a one-turn model check, agent evaluation can inspect a sequence of observations, tool calls, actions, errors, recoveries, and outcomes inside an environment.
The final result matters, but it can hide how the agent reached it. Two agents can end with similar point states after making very different decisions; another can choose a sensible action and receive an unfavorable randomized outcome. A useful evaluation preserves enough of the path to separate those stories.
WagerCall provides small, rule-bound environments where the action space, state machine, randomness identity, and public evidence are explicit. That makes a trajectory easier to inspect than an open-ended production task, while remaining only one bounded source of evidence.
Framework scope
WagerCall supplies environments and evidence primitives
| WagerCall provides | Evaluator provides |
|---|---|
| Immutable game-version identity | The behavioral question and hypothesis |
| Validated observations and strict actions | The agent, model, prompt, and wrapper definitions |
| Authoritative transitions and event records | The repetition schedule and stopping rule |
| Synthetic-point accounting evidence | Derived measures, statistical analysis, and interpretation |
| Deterministic environment support for Sessions | Threats-to-validity analysis and reporting |
Mechanism
The evidence loop has four observable stages
This loop makes tool-use behavior visible. It can show whether an agent grounded its action in current state, produced a legal schema, recovered after a conflict, or repeated a mutation safely. It does not expose private model reasoning.
- 01
Observe
The agent reads public state plus only the caller-qualified information allowed by the aggregate.
- 02
Decide
The agent selects one action from a strict, versioned contract.
- 03
Commit
The server checks identity, expected version, legality, idempotency, and point constraints before transitioning state.
- 04
Inspect
The evaluator reads the ordered state, attempt, outcome, events, and applicable point evidence.
Critical distinction
Reproducible environments do not make agents deterministic
Once agents choose different actions, later observations may differ. A comparison should identify the branch point and avoid describing every later decision as if it occurred under identical state.
| Question | Accurate answer |
|---|---|
| Can the environment be rebuilt? | Yes, in deterministic Session mode when stable versioned inputs are equal. |
| Can the canonical transcript be rebuilt? | Yes, when the environment and accepted action sequence are also equal. |
| Will two agents choose the same actions? | Not necessarily. Their trajectories can diverge from the same initial environment. |
| Does a caller-known seed create a blind test? | No. Reproducibility and seed secrecy are different properties. |
Evaluation protocol
Use a six-part comparison method
- 01
Define one observable question
Name the tool-use, decision, recovery, or coordination behavior you want to examine.
- 02
Choose the environment
Match its state, action space, and information boundary to the question.
- 03
Freeze environment variables
Record version, rules hash, engine build, RNG identity, configuration, seed policy, and trial schedule.
- 04
Document agent variables
State the model, prompt, policy, wrapper, tool interface, and the one treatment intended to change.
- 05
Run and preserve evidence
Keep successful, rejected, conflicted, retried, incomplete, and excluded runs.
- 06
Analyze outside v0
Define derived measures transparently, report uncertainty, and keep conclusions inside the environment's scope.
Interpretation
Outcome-only comparison is incomplete
The ending point state can summarize a run, but trajectory evidence explains which observable decisions, failures, and recoveries contributed to it.
- Compare legal-action rates separately from game outcomes.
- Track conflict and retry handling separately from strategic choices.
- Inspect whether decisions respond coherently to changing public state.
- Preserve rejected attempts rather than silently dropping them.
- Treat evaluator-derived measures as analysis, not as native WagerCall scores.
Environment selection
Each game exposes a different decision structure
Performance in a card-game environment does not establish broad intelligence or real-world safety. It provides structured evidence about behavior under the named rules and information boundary.
| Environment | Decision structure | Useful evidence |
|---|---|---|
| Hi-Lo v1 | Sequential direction and integer stake with public composition | Probability-sensitive decisions across a finite shoe |
| Blackjack v1 | Multi-action hand policy against a dealer with private future order | Legal-action use, hand policy, and state-dependent choices |
| Texas Hold'em v1 | Independent agents in a persistent multi-hand Room | Coordination, competition, public/private observation use, and action timing |
What this cannot prove
State the limits beside the result
- WagerCall does not reveal hidden chain-of-thought or a complete causal explanation of a model's action.
- A reproducible run is not automatically blind, independent, or statistically sufficient.
- Canonical replay is not an execution attestation or regulatory audit certification.
- Synthetic-point performance is not a monetary result and cannot be redeemed or transferred.
- One game or run does not prove general model quality, alignment, safety, or superiority.
- MCP-connected behavior in WagerCall is not proof of general MCP protocol conformance.