Poison queues and false comments: the unglamorous mechanics of production AI

Naseeb Mian

Full Stack Software Engineer, Andela Talent

Nov 2, 2026

6 min

Poison queues and false comments: the unglamorous mechanics of production AI

Summary: a pipeline that reads bank details off scanned check images passed its pilot, cleared an early-access ring, then stopped working for reasons unrelated to the model. Nothing in it is an agent, and that’s the useful part.

Three weeks in, some people stopped getting the email

A customer who still pays by paper check gets an email offering to turn it into a recurring electronic payment in one click. A background job reads the routing and account numbers off the scanned image with Azure Document Intelligence and stages a payment method to confirm.

It worked in the pilot. It worked in the early-access ring, which went live June 12 across a short list of customer tenants.

Then it didn’t. For some people, some of the time, and nothing turned red.

The messages weren’t failing anywhere anyone was watching. They were landing in a poison queue. Azure Document Intelligence caps a standard-tier resource at 15 analyze calls a second unless you ask for more. That ceiling is per resource, the default, and it’s in Microsoft’s published limits rather than a guess. Cross it, and you get a 429. Our checks never arrived at a steady rate, though. Each tenant runs its own daily sweep, the sweeps overlap, and a day’s average sat far below the ceiling while peaks went straight through.

Nothing in production here is an agent

Worth saying plainly. A timer stages items onto a queue. A function picks one up, makes one call to a managed endpoint, applies fixed thresholds, writes a row. No tool selection, no planning loop, no self-correction, no say over its call rate.

The agents here are in the build loop rather than the product: two of the controls below carry agent co-author trailers, as does the queue fix.

So why does a pipeline this dumb belong in a conversation about agentic systems? Because the wall it hit sits underneath autonomy. Something making one call per item, at a rate it doesn’t choose, crossed a documented quota hard enough to lose customer emails for weeks. An agent picks its own call rate and its own retry behavior, so the retry storm becomes a decision the model makes.

If your agentic pilot is stuck, the first thing to check is not the model.

What the technical barrier actually was

The process had no opinion about failures. Every exception out of the OCR call looked identical to the queue trigger, so a rate limit, a corrupt image and a gateway blip took one path: fail, retry, fail again, poison. The pull request says so: before the change, a 429 “bubbled up as an unhandled exception and the message fell into the poison queue after the Functions retry policy exhausted.”

The fix was a catch chain. Not clever, just specific:

catch (RequestFailedException ex) when (ex.Status is 429) { throw new DocumentOcrRateLimitException(ex); } catch (RequestFailedException ex) when (ex.Status is 503 or 504) { throw; } catch (RequestFailedException ex) { throw new DocumentOcrException( $"Document Intelligence request failed ({ex.Status}): {ex.ErrorCode}", ex); }

Three behaviors out of one surface.

A 429 becomes its own type, carrying the message Document Intelligence rate limit exceeded (429), so one call site upstream decides whether to wait. A 503 or 504 rethrows untouched, leaving the platform’s existing handling alone. Everything else is wrapped as terminal, so nobody retries it. That last branch matters, because retry-everything is the trap on the other side, and we walked into it anyway. In August, a not-found exception that nothing treated as terminal made the queue rerun a failure that could never succeed, until the message poisoned. Same outcome as the throttling, opposite cause.

The comment that wasn’t true

Items were staged onto the queue with a randomized delay, and the default ceiling on that spread was three minutes. The doc comment explained why:

/// Production overrides this per-environment, so the default is kept small.

Production didn’t override it. There was no environment override anywhere, so three minutes was the production value all along. A 180-second window absorbs at most 2,700 analyze calls before the average alone crosses 15 a second. The window is now 57,600 seconds, a 320x widening. That comment was the most dangerous artifact in the file. Code that’s wrong gets tested. A comment that’s wrong just gets believed.

The retry ladder, and the setting that does nothing

Rate-limited items go back on the queue with a growing delay. The calculator runs to fourteen lines and has two ways to give up:

if (attempt > settings.MaxAttempts) return null; var baseDelay = TimeSpan.FromSeconds(settings.StartDelaySeconds) * Math.Pow(settings.Multiplier, attempt - 1); var jitter = TimeSpan.FromSeconds(_random.NextDouble() * baseDelay.TotalSeconds); var delay = baseDelay + jitter; var cutoff = anchorDate + settings.CutoffWindow; return now + delay > cutoff ? null : delay;

Returning null drops the item rather than failing it. No row was persisted, so tomorrow’s sweep rediscovers the check with a fresh anchor. Settings: a 300-second start, a 2x multiplier, 10 max attempts, a 16-hour cutoff. To find which stop condition fires, I pointed a console harness at the compiled assembly and walked real chains to exhaustion:

zero jitter (fastest possible climb, jitter fixed at 0.0) attempt delay elapsed since anchor result 1 00:05:00 00:00:00 scheduled 6 02:40:00 02:35:00 scheduled 7 05:20:00 05:15:00 scheduled 8 - 10:35:00 STOP (no delay returned) live jitter, 100,000 simulated retry chains last attempt scheduled chains share 6 47778 47.78% 7 52222 52.22% chains that reached MaxAttempts=10: 0

MaxAttempts is a dead configuration. Attempt 10’s base delay alone runs to 42.7 hours, which can’t fit in a 16-hour window from any starting point, so the cutoff always fires first and the chain stops at 6 or 7 in all 100,000 runs.

Turn that dial to 20 and nothing changes.

Anyone tuning it in an incident is adjusting a number the system never reads.

What we turned down

Ask Azure to raise the quota. It’s free, and it’s a support ticket away, and it moves the same defect one tenant along.

Or lean on the platform’s own retry. It gives five attempts before poisoning and the default visibility timeout is zero, so a failed message reappears at once. Against a rate limit, that’s a burst generator pointed at the service that just asked you to slow down.

The controls nobody demos

This thing reads bank routing and account numbers off images of other people’s checks.

If governance means anything in a deployment, it means something here, and two of the three controls worth naming arrived as fixes rather than requirements. The first was an authorization defect: the invite was addressed on the wrong attribute of a multi-owner account, so it could reach someone who had no standing to act on it. Found inside the ring, fixed within days, and nothing came of it. It was still a question nobody had asked before shipping.

The second was the confidence threshold deciding whether a read is good enough. It moved behind a flag that defaults to off, resolves per tenant, and logs old and new decisions so they can be compared before anyone switches. Routing numbers also have to clear an ABA checksum, not a confidence score alone. Later the payment authorization stopped being recorded as offline and started carrying the consenting IP and user agent.

The ring is itself a control, and I undersold it above by calling it a rollout. Both defects surfaced inside it, which is what separated it from a demo.

Nobody funds a pipeline they can’t count

This is where the process barrier stops being technical.

Four days after the throttling work merged, our platform team cut the list of AI models engineering could use. Three tiers survived: cheap and fast, balanced, and a premium one released to selected departments only. Premium capacity was to go to what the announcement called mission-critical work.

I’m not naming the models. That roster is internal and the argument doesn’t need it.

What matters is the shape: governance as allocation, a finite premium resource handed out by argument. Organizational buy-in isn’t a mood in a room. It’s somebody with a budget deciding your work outranks whatever else the money could do, and nobody grants premium allocation because a pipeline recently stopped poisoning.

So somebody wrote the query.

On Aug. 11, it ran across a 23-tenant list: 75 converted owners, $11,645.23 in processed volume, $80.06 in convenience fees, nothing refunded. That tenant count is organizational buy-in stated as a number: the ring opened from a short list in June to 23 by August, each one somebody agreeing to point real customers at it.

Small numbers. But they’re the difference between “we hit a rate limit and paused” and “throttling is costing us conversions every week,” and only the second sentence survives a budget meeting. The ROI measurement didn’t justify the work in retrospect. It kept the work alive. That measurement was wrong twice before it was right, in the same direction both times. Two earlier versions required the owner to also have a completed check-scan row. That gate counts an owner who merely attempted enrollment, because the status flips on attempt and never reverts, and it drops everyone who converted with a payment method already on file. Both undercounted. Had either shipped, the ROI measurement would have argued against the pipeline using the pipeline’s own data.

What cuts against this

The widely quoted figure from MIT’s Project NANDA that 95% of enterprise AI pilots deliver nothing has an objection attached: it counts a pilot failed without a profit-and-loss effect inside six months, which excludes efficiency gains and every second-order benefit [1]. If six months is too short a horizon, mine is worse. My own ROI measurement came from one query, run once, 60 days past the rollout date, and some of those owners signed up the week before.

The retry shape has an objection too. Ours adds a random slice of the base delay to the base delay itself, so the wait lands anywhere on [base, 2 x base). That’s Equal Jitter in the AWS taxonomy, and the experiment behind it ranks Equal Jitter below Full Jitter on both axes [2]. We shipped the one that does more work and takes longer.

And the biggest one. My claim that autonomy makes this failure worse is an argument, not a measurement, and no agent of ours is in production hitting a rate limit. Practitioners writing on agentic retry storms land where I do [3], and their recommended fix, keeping the retry decision out of the model, is what the catch chain above already is. That’s convergence, not evidence, and their numbers come unsourced.

The part that generalizes

The technical barrier here was ordinary. A documented quota, a burst-shaped workload, a catch block, a config value. Any competent team clears it in a week.

What separated this from the pilots that never come back wasn’t the fix. It was that the ring stayed small enough for both failures to surface as support questions rather than incidents, each was specific enough to name, and somebody could put a dollar figure beside the work before the roadmap slot was defended.

Governance decides who gets resources. Measurement decides who counts as mission-critical. Neither gets easier when you add autonomy, and the 429 doesn’t care that its caller can plan.

Sources