Agent Replay Runtime
Durable tool execution with event replay, idempotency keys and recovery after real process crashes. Includes effect traces and cases that stop instead of retrying automatically.
Problem and relevance to agents
A tool can commit an effect just before its caller dies. Blind retries can duplicate it. The runtime persists the request before invocation and the result before moving to the next step.
This separates an agent’s recorded decisions from action recovery. A finite JSON plan becomes a validated event history that distinguishes completed operations from operations whose outcome remains uncertain.
Architecture
- Pinned plans and adapters. Plans declare unique steps, JSON arguments, adapter identities and idempotency support. Reusing a run identifier with a different plan or adapter is rejected.
- Durable event history. SQLite WAL with synchronous FULL stores each transition. A process-held file lock allows one writer per database. Replay validates history without invoking tools.
- Separate effect store. An independent SQLite ledger represents the tool. Its idempotent operation commits the receipt and effect together; the baseline operation does not deduplicate.
- Contract-aware recovery. Pending requests retain their keys. An idempotent adapter can receive the same request again. Without that contract, execution pauses and requires reconciliation backed by external evidence.
What this demonstrates
- State modeling and action recovery from a validated event history.
- Separation of the agent log, external effects and the tool’s idempotency contract.
- Reproduction of the effect-to-acknowledgment gap, with killed-process traces and a baseline that duplicates the effect.
Published results
The author’s published report records six fixed crashes: three checkpoints for each adapter type. It also includes an uninterrupted control and a blind-retry baseline.
The parent waits for a checkpoint handshake, terminates the child, starts a fresh worker and reads the independent effect store. The local recording used SIGKILL, Python 3.11.13, SQLite 3.50.4 and an Intel Xeon W-3223 with 48 GiB RAM. Workers ran serially.
Effects after recovery
| Case | Final effects | Outcome |
|---|---|---|
| Idempotent: after request | 1 | Resumed with the same key |
| Idempotent: after effect | 1 | Existing receipt recovered |
| Idempotent: after acknowledgment | 1 | Recorded result replayed |
| Non-idempotent: after request | 0 | Paused, unresolved |
| Non-idempotent: after effect | 1 | Paused, then reconciled with ledger |
| Non-idempotent: after acknowledgment | 1 | Recorded result replayed |
| Baseline: blind retry | 2 | Duplicate effect |
| Uninterrupted control | 1 | Completed |
All three idempotent cases finished with one effect. The non-idempotent case interrupted before the effect remained pending with zero effects: stopping was the expected outcome. This is neither a universal exactly-once guarantee nor a statistical reliability estimate.
The post-effect unsafe case was reconciled by automated inspection of the fixture ledger, without human review. The report preserves before/after events, receipts, termination codes and source hashes. Crash matrix and traces.
Reproduce
Python 3.11 or newer, with no runtime dependencies, models, credentials or paid APIs. The linked report uses Python 3.11.13.
The first command checks recorded evidence. The second run creates a fresh matrix by terminating its own child processes. Temporary paths, timestamps and termination codes can vary by platform. Output files contain synthetic fixtures only.
Verify the recorded report
git clone https://github.com/lucioduran/agent-replay-runtime.git
cd agent-replay-runtime
git checkout 8dbc49ef4b154cc738698714749ad133fa2c6855
python -m experiments.verify results/crash-matrix.jsonReproduce the crashes
python -m experiments.reproduce --output work/crash-matrix.json
python -m experiments.verify work/crash-matrix.jsonScope and limitations
- Deduplication depends on the tool atomically binding key, effect and receipt and retaining that receipt. An idempotent flag does not verify the contract.
- Evidence covers local process crashes, not power loss, network partitions, remote providers, expiring receipts or divergent database restores.
- Plans are finite and sequential. Model planning, dynamic branching, compensation, backoff and multi-host coordination are outside scope.
- A hung tool blocks the writer. History is unbounded and repeated validation can make long runs quadratic. Throughput was not measured.
- The hash chain detects accidental changes, not malicious rewriting. Reconciliation records an operator assertion; the runtime cannot verify external truth.
Sources and revision
Values on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report.
- Reviewed revision:
8dbc49ef4b154cc738698714749ad133fa2c6855 - Revision recorded by the experiment:
753a57995f6eb8a89659bed5d06acfd4d167f847 - Report timestamp (UTC):
- Runtime implementation SHA-256:
f1ad8eaaaa0a6f907c187b7b7c626a5cda99baba90f170a7d2e01425d206a1e7