Four of the fourteen people who imported a script in the first week of September uploaded the same file again within minutes. One uploaded the same PDF three times over two days and then photographed the pages instead. The production worker logged no error for any of it.
Those numbers came out of an audit I ran on September 7, replaying every captured upload against the dev backend. Everything below is what I changed in the sixteen days after it, in the order the failures showed up, plus the two August fixes that set it up. The reason those users re-uploaded is at the end.
The pipeline, in one paragraph
Scriptmate is an iOS rehearsal app for actors. A user imports a script as a PDF, a photo, or pasted text. The app sends it to a single Cloudflare Worker, which calls Claude Haiku 4.5 with tool_choice pinned to one tool, so the model must answer with that tool’s JSON: title, characters, dialogue turns. Long scripts are chunked at 12,000 characters and parsed eight chunks at a time. The model retypes the script as JSON at 45 to 50 output tokens per second, so a real play takes 80 to 180 seconds of wall time today, and took 165 to 310 seconds until a latency pass on September 8. A three minute HTTP request from a phone on Wi-Fi is where the rest of this story comes from.
Step one: retry on the client
Until September 9 a connection drop mid import produced the same screen as a corrupt file: your PDF is the problem, pick another. That is what the re-uploaders had been seeing.
The fix was a single wire path for all three parsing endpoints. It re-sends up to twice, after one second and then three, on a dropped connection or a 502, 503 or 504. It never retries a 413, 422 or 429, because those would fail again and burn tokens. And it never retries a timeout, deliberately. The timeout is 1,600 seconds because a multi chunk parse legitimately runs for minutes, so a timeout means the worker went silent for 26 minutes, and repeating that only stretches the failure.
static let retryableStatuses: Set<Int> = [502, 503, 504]
static let retryableURLErrorCodes: Set<URLError.Code> = [
.networkConnectionLost, .notConnectedToInternet, .cannotConnectToHost,
.cannotFindHost, .dnsLookupFailed, .internationalRoamingOff, .dataNotAllowed,
]
// .timedOut is deliberately absent.The error screen became typed: “Connection lost” or “Something went wrong on our side”, each with a Try again button that re-runs the same file. The analytics events gained a retry count so the recovery rate is visible.
Then the retry went live and exposed two server bugs that had been invisible for as long as the client gave up on the first failure.
Step two: a status code is a promise about retrying
The first gap. The worker already retries a 429 or 529 from Anthropic three times with one, two and four second backoff. When a 429 outlived those retries, the worker’s rule was “upstream 4xx becomes 422”, so an account level rate limit reached the phone as a deterministic failure: try fewer pages. The app, correctly, never retries a 422. A user hitting a busy minute was told their script was too long.
The mapping became explicit about what each code means to a client with a retry policy. This is the September 9 version; a third case joins it in step six:
function statusForClaudeFailure({ filtered, upstreamStatus }) {
if (filtered) return 422 // content filter: same input fails again
const up = Number(upstreamStatus)
if (up === 429 || up === 408) return 503 // busy: worth a retry
if (up >= 400 && up < 500) return 422 // the input is the problem
return 502 // upstream broke: worth a retry
}A 503 carries Retry-After: 5 and a “service is busy” message. A 4xx tells the client to stop and a 5xx tells it to try again, and if you get that backwards the client’s retry policy works against your users.
The same rule caught one more case two weeks later. A photo with no dialogue in it, a poster say, produced an empty tool call, which the worker returned as a 502. The app re-sent it twice. Three parses for one poster, none of them billed to the user, because the worker charges a user’s allowance only on a shape valid success, but all three paid for by me. Empty dialogue now gets one retry at a higher temperature and then a 422 with a message that says what was not found. Never 5xx a deterministic outcome.
Step three: the retry that parsed twice
The second gap was worse. A repeat of an import was indistinguishable from a new one. If the reply was lost in transit, the script was parsed twice. If the connection dropped mid parse, the first run kept finishing on the server while the retry started a second.
This is the idempotency key problem, and the solution is the one payments APIs have used for a decade. The app mints one UUID when an import begins and sends it as X-Import-Attempt on every request of that run, automatic retries included. The worker keys the run on it, per user and per endpoint, in Workers KV, Cloudflare’s eventually consistent key value store. The first version, on September 9:
let record = await kv.get(key, 'json')
if (record?.state === 'done') {
return replayResponse(record, rayId) // stored for an hour, replayed with X-Idempotent-Replay
}
if (record?.state === 'in_flight' && now() - record.started < TWENTY_MINUTES) {
// another run of this attempt is live: poll for its verdict instead of parsing again
}
const started = now()
await kv.put(key, JSON.stringify({ state: 'in_flight', started, ray: rayId }), { expirationTtl })A finished reply, meaning a 200, a 413 or a 422, is stored for an hour and replayed, with the original ray ID in an X-Idempotent-Replay header so the reply is traceable to the run that produced it. A transient failure leaves no record, so the retry runs. A repeat that finds a run in flight waits for its verdict. A request with no header gets the old behaviour, so older app versions keep working.
The eventual consistency question is the first one an engineer asks, so here is the answer from the source comment: a retry from the same phone lands at the same Cloudflare data center in practice, where KV writes are read your own. The worst case of a missed read is the old behaviour, a duplicate parse, never a wrong answer.
Step four: liveness is a heartbeat, not a start time
That first version treated an in flight marker older than twenty minutes as a dead run. On September 23 a user’s connection kept dropping: ten sends in a row, every one cancelled by the client somewhere between 0.07 and 22 seconds in. The Workers runtime cancels the handler when its client disconnects, so the first run died with the connection, but its marker sat there alive. Every one of the nine re-sends waited politely on a run that no longer existed.
Two changes. The run and a KV heartbeat now go through ctx.waitUntil, the mechanism that lets a Worker keep running for up to thirty seconds after its client disconnects, so a parse that was about to finish still settles the marker for the re-send. And staleness is measured from the last beat, written every ten seconds, with a sixty second window, not from the start time. A waiter that finds a stale beat takes the run over and logs it, and a waiter that has waited fifteen minutes gives up with a 503 and a Retry-After.
const heartbeat = (async () => {
while (!settled) {
await wait(ATTEMPT_HEARTBEAT_MS)
if (settled) break
lastBeatWrite = kv.put(key, inFlight(now()), { expirationTtl: ATTEMPT_IN_FLIGHT_TTL_S }).catch(() => {})
await lastBeatWrite
}
})()The settled flag is flipped before the verdict is written and the last beat write is awaited first, so a heartbeat can never land on top of a done record. That ordering took longer to get right than the rest of the feature.
Measured on dev: a twenty second parse cut off at six seconds was replayed to the re-send after 25 seconds. An eighty second parse cut off at five seconds was taken over after 90.
Step five: a retry only helps if something changes
On September 7 I set the extraction temperature to zero. Extraction is transcription, and sampling variance had been showing up as turn counts drifting between runs and the same shared line rendered differently each time. Determinism fixed that.
It also broke the worker’s own chunk retries. When the model returned a malformed tool input, the worker retried the chunk once with the same input, and at temperature zero got byte for byte the same 10,695 tokens back. The user tried three times. Six identical replies, three 502s, about six minutes each. Retries now run at temperature 0.5, so the second attempt can actually land differently.
A chunk that is still malformed after salvage, the worker’s repair pass for an array the model returned as a string, and after a re-rolled retry is now gap eligible: a client that sends allowGaps gets the other ten pages plus a gaps entry naming the range that failed. The app does not send that flag yet, so in production that import still fails as a whole. The flag exists so the next app release can stop throwing pages away, and it is the biggest open item on my list.
The general form: if your retry sends the same bytes to a deterministic system, it is not a retry, it is a loop with a delay.
Step six: the Worker in the wrong country
Anthropic refuses calls from regions it does not serve with a 403, “Request not allowed”, and a Worker’s outbound fetch egresses from whichever Cloudflare data center received the request. On September 11 two requests from a user in Perth were served from the Hong Kong data center and refused. A week later a user who actually lives in Hong Kong made 25 refused calls in three minutes across three scripts, each failing in about ten milliseconds. The client retries never even started, because a 403 was mapped to 422, and the user could never import anything.
Durable Objects are the one Workers primitive that takes a location hint. On that 403 the worker now re-sends the identical request through a proxy object created with a hint, trying western North America, then eastern, then western Europe. A hint is best effort, which is why there are three. The object only lends its egress location and streams the reply back untouched. The region that worked is remembered in a per request map keyed by ray ID, so every later chunk of the same import goes straight there. The object is named per region and per user, so one heavy import never shares an event loop with another user’s. Only when every region refuses does the 403 propagate, and it now maps to 503 rather than 422, so the app stops blaming the PDF.
Verified on dev with a debug header that forces a region and a trace endpoint that shows the path taken. I have not yet watched a real blocked user come through it.
Why they re-uploaded
Two older failures explain the silent re-uploads in the first paragraph.
The Cloudflare account was on the Workers Free plan until September 7, which caps CPU at ten milliseconds per request and cannot be raised. Any parse of roughly 39 KB or more was CPU killed, and the runtime then throttled the whole worker for ten minutes or more, so even a 900 byte request died with the same error. Production survived only because traffic was sparse. The fix was a paid plan and a [limits] block with a five minute CPU budget that the repository’s notes say never to remove.
Before that, in August, the stream reader that accumulated the model’s tool JSON was concatenating strings per event, quadratic in output size. A 500 KB tool JSON produced gigabytes of copies. Joining an array once per block was the whole fix for the CPU burn, but even with the quadratic copy gone, reassembling eight streams does not fit in ten milliseconds, which is why the plan had to change too.
Neither of those produced an error a user could act on. They produced re-uploads.
The ledger
What makes this tractable as one person is that nothing is lost. Every upload is captured to an R2 bucket with the Cloudflare ray ID before parsing starts. Failures and successes are logged with the same ray. When an import fails, the app builds a prefilled support email with the file attached.
The worker has a hundred unit tests, with a fake KV and an injected clock. They cover the status table, every branch of the idempotency wrapper including the heartbeat takeover, each region reroute path with an injected fetch, and each malformed output shape the salvage code has seen. Deploys go through a golden eval set of real scripts, and King Lear runs by hand when the chunk pipeline changes, at about 36 cents a run.
What transfers
I spent years building payment and banking apps before this, and every rule above is one I already knew from that side of the wire. Idempotency keys on anything that costs money. Status codes as a contract with the client’s retry policy. Liveness measured by heartbeat, not by start time. Retries that change something. Never a 5xx for a deterministic outcome. An LLM backend is a payments API where the charge is tokens and the ledger is your own logs.
Scriptmate is on the App Store, built solo. The speech and audio engineering is in the case study, and the iOS 26 speech engine has a post of its own.