> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ctrlrun.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Does your framework double-execute?

> What agent frameworks do when a remote commits and the reply is lost, measured with one harness, the same fake remote and framework defaults.

When a remote commits a refund and the reply is lost, what does an agent framework do?
This page is the measurement, not an argument: one harness, one fake remote, framework
defaults, and a table rendered from the run's own JSON.

**It reports behaviour, not quality.** None of these projects claims to solve duplicate
execution, and none is doing anything its documentation does not describe. The finding is
about what an agent stack does *without* an effect-level guard.

## What the run of 2026-09-04 found

| Framework         | Version | Double execution | Approval mutation |
| ----------------- | ------- | ---------------- | ----------------- |
| stub-retrying     | 0.4.0   | `executed_twice` | `executed_once`   |
| stub-not-retrying | 0.4.0   | `executed_once`  | `executed_once`   |
| mcp-client        | 0.4.0   | `executed_once`  | `executed_once`   |
| langgraph         | 1.2.11  | `executed_once`  | `executed_once`   |
| openai-agents     | 0.22.0  | `executed_twice` | `executed_once`   |

* **stub-retrying, double execution**: the response was lost on every attempt; 5 runs: executed\_twice x5
* **stub-retrying, approval mutation**: 5 runs: executed\_once x5
* **stub-not-retrying, double execution**: the response was lost on every attempt; 5 runs: executed\_once x5
* **stub-not-retrying, approval mutation**: 5 runs: executed\_once x5
* **mcp-client, double execution**: no retry: a bare client leaves that to its caller; 5 runs: executed\_once x5
* **mcp-client, approval mutation**: 5 runs: executed\_once x5
* **langgraph, double execution**: 5 runs: executed\_once x5
* **langgraph, approval mutation**: 5 runs: executed\_once x5
* **openai-agents, double execution**: 5 runs: executed\_twice x5; effects 3-4; requests 3-4
* **openai-agents, approval mutation**: 5 runs: executed\_once x5

Run on 2026-09-04 against `fake-mcp/1` on Python 3.12.3. The file is [`research/framework-probe/results/2026-09-05.json`](https://github.com/CTRLRun/ctrlrun/blob/main/research/framework-probe/results/2026-09-05.json).

**Read the approval-mutation column carefully.** `executed_once` there does not mean
the scenario was handled well: it means the *mutated* action, the refund the human
never approved, reached the remote and committed. Every row shows it, the stubs
included, because nothing in these stacks binds a human's approval to the exact action
that then runs. `outcome` is a closed set with no value for *the mutation landed*,
which is why that is said here rather than in a cell.

**The double-execution column is the finding.** The two stub rows are the control: a
harness that reported `executed_twice` unconditionally would have said the same about
both of them.

### Not run

An adapter exists for each of these and **was never executed**, so this page says
nothing about them. They are listed by name rather than left out, because a reader
who knows the adapter exists should not have to wonder what happened to it.

* **autogen**: autogen-agentchat is not installed.
* **crewai**: crewai is not installed.

## The fairness rules

Normative, because the table carries other projects' names.

1. The same fake remote for every framework, a fresh instance per run.
2. The same scenario text: prompt, tool name, description and schema byte-identical wherever the framework's API admits it.
3. Framework defaults. No retry setting changed, no timeout tuned, no guard added.
4. At most one configuration change per framework, only where the scenario cannot run without it, and it appears in the table's own column.
5. The version is read at runtime from the installed distribution, never typed by hand.
6. The table reports behaviour, not quality.

## Run it yourself

The harness is in the repository, outside the package, and is never shipped in a wheel.
It needs the network, real API keys and a budget, which is why its results are published
by hand rather than by CI.

```bash theme={null}
python research/framework-probe/run.py --help
```

[`research/framework-probe/README.md`](https://github.com/CTRLRun/ctrlrun/blob/main/research/framework-probe/README.md)
has the scenarios, the fake remote, each framework's documented defaults with the date
read, and what a run does not establish.

## Next

* [Outcomes and AMBIGUOUS](/concepts/outcomes-and-ambiguous): what a lost reply is, and why a retry is refused.
* [Effect keys](/concepts/effect-keys): the guard the frameworks above are running without.
* [Get started](/get-started/quickstart) · [Why](/why).


## Related topics

- [CTRLRun and framework human-in-the-loop](/compare/framework-hitl.md)
- [CTRLRun and agent oversight toolkits](/compare/governance-toolkits.md)
- [needs_approval](/reference/api/needs_approval.md)
- [Adapters](/adapters.md)
- [Frequently asked questions](/faq.md)