An agent sends a consequential request to an external service. Its tool call times out. The agent has no confirmation, so it reports failure and sends the same request again. But the first request may already have been accepted, may still be running, or may have changed the external system before its response was lost.
The local timeout is real. The claimed failure of the external operation may not be.
Unknown is not failed. The phrase does not mean that every timed-out request succeeded or that retries are inherently unsafe. It means that a missing or delayed observation cannot, by itself, settle what happened beyond the observer's boundary. Recovery should be governed by what is known, what the external system guarantees, and what the next action is authorized to change.
A timeout describes the observer
A timeout usually says that a particular observer did not receive an expected response within a configured interval. It does not, on its own, specify where processing stopped. A request might have failed before dispatch, been rejected by a receiver, entered an asynchronous queue, committed a change, or remained in progress after the caller stopped waiting. The observation boundary matters: a caller may sometimes have verified evidence that dispatch never began; conversely, a service response or authoritative transaction record may establish a rejection without effect. Neither situation should be relabeled as unknown simply because some other layer reported a timeout.
These possibilities matter most when the operation has an external effect: creating a resource, publishing a message, initiating a payment, modifying a production configuration, or triggering a deployment. In those cases, treating a communication timeout as a definitive business failure can convert an ordinary uncertainty into a duplicate or conflicting action.
It is useful to separate three propositions:
- Caller status: The caller's attempt timed out or returned an error. A local error alone does not establish whether an external effect occurred.
- Operation status: The receiving system rejected, accepted, queued, executed, or committed the requested work; these are distinct service-defined states.
- Result status: The relevant external state satisfies—or does not satisfy—the intended effect at a specified observation time and within the consistency limits of that observation.
The first proposition may be established by a local record while the second and third remain unsettled. An API response, when available, must also be interpreted according to its documented semantics: acknowledgment of receipt is not always proof of completed work. Established guidance on timeouts and idempotent APIs makes this distinction especially important for side-effecting operations [1, 2].
A deployment that may still be running
Consider a hypothetical operations agent responsible for preparing a configuration rollout. An authorized operator approves an update to a specific service and environment within a defined change window. The agent submits the approved operation to a deployment coordinator. It records a request identifier and a trace of the outgoing call, but receives a timeout before confirmation.
The same local symptom is compatible with several external states. The coordinator might never have received the request. It might have rejected it without effect. It might have queued the rollout, begun applying it, or completed it while the response was lost. An initial status query might itself be unavailable or read from a delayed index.
If the agent treats the timeout as definitely failed and immediately creates a new rollout, it may initiate duplicate work. If it declares success, it may conceal an unapplied change. If it automatically sends a rollback, it may introduce yet another effect while the first operation remains unresolved.
A defensible immediate report is narrower: “Submission timed out; external outcome not established.” This is the appropriate report when the available evidence does not distinguish the possible receiver states—not a mandatory label for every timeout. The next permissible action might be to inspect the deployment coordinator using the existing request identity, wait for a reliable state transition, or escalate the case. That choice depends on the actual service contract, applicable permissions, and operational risk.
This is an illustrative scenario. It is not a report of a ZeroField implementation, a production incident, or a measured recovery test.
Do not collapse two state machines
The caller and the receiving system maintain different views of an operation. A local workflow can mark a task as timed out while the receiver continues to process the request. A durable workflow engine can also retry a task after a worker crash even when part of its external effect occurred before the worker reported completion. Temporal documents this distinction between a recorded activity completion and the possibility of repeated or partial activity execution [5].
For analytical purposes, a recovery interface may distinguish ATTEMPT RECORDED, OUTCOME UNKNOWN, CONFIRMED REJECTED, CONFIRMED APPLIED, and REVIEW REQUIRED. These names are illustrative reporting categories, not a universal ZeroField protocol, and they do not all describe the same dimension: the first four concern observations or results, while REVIEW REQUIRED describes a workflow disposition. CONFIRMED REJECTED should imply no relevant effect only if the receiver contract and evidence support that conclusion; CONFIRMED APPLIED concerns a specified effect at an identified time, not its permanent persistence. A category must describe the evidence available at that moment—not invent a receiver-side fact from caller-side telemetry.
A further distinction is needed between verified absence and absence from a search result. A query returning no matching deployment, order, or message may reflect replication delay, incomplete indexing, a mismatched identifier, limited visibility, or a genuinely absent operation. Negative evidence is only as strong as the completeness and consistency guarantees of the queried source.
The practical consequence is that uncertainty should be represented explicitly. It should not disappear because a user interface, retry loop, or status enum offers only “success” and “failure.”
Reconcile before deciding to repeat
Reconciliation means obtaining an appropriately authoritative observation that can narrow the uncertainty about an attempted operation. For the deployment example, the receiving system might expose a stable operation ID, a status endpoint, a committed configuration version, or a service-side audit record.
An evidence-aware recovery path can ask, in order: Which request identity was used? What did the receiver record for that identity? Is the operation pending, rejected, or complete? Does a sufficiently fresh check of the affected resource agree? What remains unobservable? Are the original authorization and operating conditions still valid?
These questions do not mandate endless polling. A reasonable system sets bounded waiting or review conditions and stops when it cannot establish an acceptable basis for another consequential action. Where the receiver provides no trustworthy lookup or deduplication mechanism, human review may be more appropriate than an automatic replay.
Crucially, reconciliation is not equivalent to retrying the original write. A read-only status inquiry, when actually read-only and properly authorized, can reduce uncertainty without repeating the external effect. A new write is a separate decision requiring its own justification. Reconciliation need not always precede every retransmission: if a documented, enforced idempotency contract still covers the same request identity, scope, parameters, and intended effect, replay of that request may be an authorized recovery strategy even while its first result remains uncertain. This is materially different from generating a fresh write or silently changing the request [1, 3].
Idempotency is a contract, not an assumption
Idempotency in the relevant API contract means that supported repetitions of the same logical operation do not create additional unintended effects within the contract's stated boundaries. This practical guarantee must not be confused with a universal claim that every observable response or downstream state is identical. In key-based APIs, a caller can supply an idempotency key or client request token so the receiver can recognize a retry of the intended operation. AWS explains how semantic request identity and the atomic relationship between a recorded token and a mutation matter [1]. Stripe describes provider-specific rules for key reuse, parameter matching, and retention [3].
The existence of a field called request_id or idempotency_key is not sufficient proof that a workflow can retry safely. An implementation must understand the receiver's contract:
- Scope: Which endpoint, operation, account, tenant, or resource does the key cover?
- Identity: Does the receiver bind a key to the original parameters and intent?
- Lifetime: For how long does deduplication remain effective, and what happens after records expire?
- Concurrency: How are simultaneous attempts with the same key handled?
- Effect boundary: Does the guarantee cover only accepting a request, or the relevant downstream effect as well?
A new key can represent a new operation rather than a safe replay. Reusing an old key after its validity window expires may also fail to prevent duplication. And two syntactically similar requests need not represent the same user intent. Correctness comes from documented, enforced semantics—not from a persuasive variable name.
A concrete provider caveat: Stripe documents returning the stored status code and body of the first request for a reused key, including a 500 response. That repeated error is not a fresh attempt to execute the same operation, and the response alone does not settle every downstream effect. Stripe may remove keys once they are at least 24 hours old; reuse after pruning can be treated as a new request. It also does not store an idempotent result when execution never begins because validation fails or a concurrent request conflicts. These are Stripe-specific behaviors, not universal rules for all APIs [3].
Even platform claims of exactly-once delivery have boundaries. Google Cloud's Pub/Sub guarantee concerns supported pull subscriptions, successful acknowledgment behavior, a Pub/Sub-defined message ID, and its documented regional scope. It does not itself guarantee exactly-once effects in an external database, notification service, or other downstream system; publish-side duplicate messages can also have distinct message IDs [6]. The same discipline applies to agent tools: transport guarantees cannot automatically be promoted into global business-outcome guarantees.
A retry policy is a controlled resource budget
Retries remain valuable for genuinely transient faults. A retry can recover from a brief network interruption or an overloaded dependency; forbidding retries categorically would also be poor engineering. The issue is whether repetition is eligible, bounded, and safe for the specific operation.
Backoff, jitter, attempt limits, and retry budgets help avoid amplifying outages. They do not establish the missing external outcome or make a non-idempotent action safe. A retry delayed by thirty seconds can still duplicate a completed operation. Conversely, an idempotent request may still need throttling because repeated calls consume capacity. AWS and Microsoft describe these distinct concerns in their retry guidance [2, 4].
A robust recovery policy therefore separates two decisions: May the operation be repeated? and If so, when and how many times? The first depends on authorization, outcome knowledge, and the service contract. The second depends on load, failure classification, deadlines, and bounded operational budgets. Per-operation attempt caps and an aggregate retry budget across concurrent requests solve different load problems. Multiple nested retry layers can multiply total attempts unless their budgets are coordinated [2, 4].
A system also needs a stopping rule. When its evidence or retry budget is exhausted, recording an unresolved or escalated state is often more accurate than manufacturing a success/failure conclusion.
Cancellation and compensation are not erasure
A caller may cancel its local task without proving that an externally initiated operation stopped. Cancellation can be cooperative, delayed, or unsupported by the receiver. In Temporal, an Activity must heartbeat to receive cancellation from the Temporal Service, and SDK-specific behavior determines how the activity handles that cancellation [5]. An external system may still have acted independently. Accordingly, “cancellation requested”, “worker acknowledged cancellation”, and “external effect prevented or reversed” are different claims.
Compensation introduces another distinction. A corrective change, such as restoring a previous configuration, is a new externally consequential operation. It can fail, arrive late, conflict with subsequent changes, or leave other effects unreversed. Notifications already delivered, events observed by third parties, and other non-reversible consequences may persist. The Compensating Transaction pattern is a useful architectural reference for multistep systems, but compensation must not be described as guaranteed restoration of every prior state [7].
If a change may still be running, automatic compensation may be especially problematic. The system may need to establish the operation's current state, identify what was actually changed, and obtain any required approval before initiating a corrective action.
What to test before trusting recovery
A recovery architecture should be evaluated against uncertainty, not only against clean success and clean failure. Proposed test scenarios include:
- Response lost after commit. The receiver applies a change, but the caller times out. The agent must not assume failure or initiate an unsafe duplicate.
- Delayed processing. A status endpoint initially returns “pending” while the operation continues. The agent must not confuse delayed completion with rejection.
- Stale negative lookup. A replica reports no matching operation although an authoritative source later confirms it. The agent must retain appropriate uncertainty.
- Idempotency window expired. A repeated request arrives after the service's deduplication period. The agent must not assume that the old key still prevents duplicate effects.
- Changed parameters or authority. A retry has different arguments, a different target, an expired approval, or a revoked role. It must be treated as a fresh policy decision rather than continuation by default.
- Concurrent recovery agents. Two actors attempt to recover the same timed-out workflow. The system must test whether its coordination and receiver contracts prevent competing effects.
- Cancellation arrives late. The caller requests cancellation after the receiver has begun or completed work. The reported status must not falsely claim reversal.
- Exhausted observation and retry limits. No sufficiently authoritative state can be established. The system must preserve an unresolved disposition and an appropriate review path.
- Known pre-dispatch failure or definitive no-effect rejection. Trustworthy local or receiver evidence establishes that the consequential operation did not begin or could not apply. The system should not leave this case permanently UNKNOWN; any new attempt still needs the appropriate authority and policy checks.
- Cached error on idempotent replay. Repeating a valid Stripe-style key returns a stored
500result. The agent must not mistake the replay for fresh execution or treat the cached error alone as complete evidence about external state [3].
An acceptance criterion should define the tested service semantics, implementation versions, permissible effects, expected decisions, and evidence required to judge them. Labels such as SUPPORTED, CONTRADICTED, and NOT ESTABLISHED may describe outcome claims, while RETRY ALLOWED, WAIT, and ESCALATE may describe policy decisions. They should not be collapsed into one binary signal.
These are proposed evaluation cases only. No experiments, benchmarks, or verified ZeroField test results are claimed.
Where the ZeroField model fits
ZeroField organizes intelligent systems into six responsibility domains: Context; Intelligence & Roles; Control & Governance; Coordination & Execution; State, Memory & Knowledge; and Integration & Interfaces. Uncertain recovery sits across these boundaries rather than belonging to one retry function.
Context identifies the target environment and current operating conditions. Intelligence & Roles describes which actor proposes or resumes a task. Control & Governance defines approval, permission, and stop conditions. Coordination & Execution handles attempts, sequencing, and controlled recovery. State, Memory & Knowledge preserves unresolved status without transforming earlier assumptions into current truth. Integration & Interfaces supplies the external service contracts, status queries, and boundaries of the effects being requested.
This is a conceptual mapping, not a claim that ZeroField provides a deployed scheduler, deduplication service, external transaction coordinator, or tested recovery engine.
The preceding notes make complementary distinctions. Authority Is Not Capability examines who may cause an effect. A Trace Is Not Evidence examines what recorded observations can establish. Unknown Is Not Failed deliberately moves from interpreting evidence to choosing a bounded recovery action—wait, query, reuse an authorized idempotent request, stop, or escalate—while external outcome uncertainty remains.
Scope and open questions
This note synthesizes established distributed-systems patterns for an agentic-systems context. It does not propose an original consensus algorithm, a universal recovery state machine, a new idempotency protocol, or legal guarantees about audit evidence. It does not report implementation or measured performance.
Open questions include how to determine sufficiently authoritative negative evidence from eventually consistent services; how to bind delegations to recovery after a long interruption; how to coordinate competing agents when the receiver lacks reliable request identity; and how to balance waiting, human review, operational urgency, and duplicate-effect risk.
A failed observation is not proof of a failed external operation. Good recovery begins by preserving that distinction—and refusing to make the next effect less certain than the last.
References and technical context
- AWS Builders' Library — Making retries safe with idempotent APIs. On client request tokens, semantic identity, late requests, and safe repetition.
- AWS Builders' Library — Timeouts, retries, and backoff with jitter. On transient failures, timeout design, bounded retries, backoff, and jitter.
- Stripe API Reference — Idempotent requests. Provider-specific treatment of reused keys, stored initial HTTP responses (including
500), parameter matching, retention, and pre-execution exceptions. - Microsoft Azure Architecture Center — Transient fault handling. Guidance on retry eligibility, retry budgets, idempotency and duplicate retry layers.
- Temporal Documentation — Activity Execution. Activity timeouts, retry lifecycle, cancellation delivery and externally asynchronous work.
- Google Cloud — Pub/Sub exactly-once delivery. Defines pull-subscription, regional, message-ID, and acknowledgment boundaries. See also Google Cloud's explicit separation of delivery from external side effects.
- Microsoft Azure Architecture Center — Compensating Transaction pattern. On corrective operations and limitations of undoing multistep distributed effects.