Skip to main content
When a remote commits a refund and the reply is lost, what does an agent framework do? This page is the measurement, not an argument: one harness, one fake remote, framework defaults, and a table rendered from the run’s own JSON. It reports behaviour, not quality. None of these projects claims to solve duplicate execution, and none is doing anything its documentation does not describe. The finding is about what an agent stack does without an effect-level guard.

​
What the run of 2026-09-04 found

  • stub-retrying, double execution: the response was lost on every attempt; 5 runs: executed_twice x5
  • stub-retrying, approval mutation: 5 runs: executed_once x5
  • stub-not-retrying, double execution: the response was lost on every attempt; 5 runs: executed_once x5
  • stub-not-retrying, approval mutation: 5 runs: executed_once x5
  • mcp-client, double execution: no retry: a bare client leaves that to its caller; 5 runs: executed_once x5
  • mcp-client, approval mutation: 5 runs: executed_once x5
  • langgraph, double execution: 5 runs: executed_once x5
  • langgraph, approval mutation: 5 runs: executed_once x5
  • openai-agents, double execution: 5 runs: executed_twice x5; effects 3-4; requests 3-4
  • openai-agents, approval mutation: 5 runs: executed_once x5
Run on 2026-09-04 against fake-mcp/1 on Python 3.12.3. The file is research/framework-probe/results/2026-09-05.json. Read the approval-mutation column carefully. executed_once there does not mean the scenario was handled well: it means the mutated action, the refund the human never approved, reached the remote and committed. Every row shows it, the stubs included, because nothing in these stacks binds a human’s approval to the exact action that then runs. outcome is a closed set with no value for the mutation landed, which is why that is said here rather than in a cell. The double-execution column is the finding. The two stub rows are the control: a harness that reported executed_twice unconditionally would have said the same about both of them.

​
Not run

An adapter exists for each of these and was never executed, so this page says nothing about them. They are listed by name rather than left out, because a reader who knows the adapter exists should not have to wonder what happened to it.
  • autogen: autogen-agentchat is not installed.
  • crewai: crewai is not installed.

​
The fairness rules

Normative, because the table carries other projects’ names.
  1. The same fake remote for every framework, a fresh instance per run.
  2. The same scenario text: prompt, tool name, description and schema byte-identical wherever the framework’s API admits it.
  3. Framework defaults. No retry setting changed, no timeout tuned, no guard added.
  4. At most one configuration change per framework, only where the scenario cannot run without it, and it appears in the table’s own column.
  5. The version is read at runtime from the installed distribution, never typed by hand.
  6. The table reports behaviour, not quality.

​
Run it yourself

The harness is in the repository, outside the package, and is never shipped in a wheel. It needs the network, real API keys and a budget, which is why its results are published by hand rather than by CI.
research/framework-probe/README.md has the scenarios, the fake remote, each framework’s documented defaults with the date read, and what a run does not establish.

​
Next

Related topics

CTRLRun and framework human-in-the-loopCTRLRun and agent oversight toolkitsneeds_approvalAdaptersFrequently asked questions