← All posts

What ten production incidents taught us about shipping an AI content tool

Srinivasa Reddy, Founder, Zolt AI · · 7 min read

We keep an internal document called PRODUCTION_SAFETY.md. It's a list of every way this product has actually hurt, or nearly hurt, a real customer — not hypothetical risks, not a generic security checklist copied from somewhere. Every entry cites the commit that fixed it. If we can't point to a real incident, it doesn't go on the list, because a list of thirty imagined dangers reads as zero real ones.

We're publishing some of it, because "trust us, we're careful" is not an argument, and a company that's willing to describe its own mistakes precisely is at least telling you something true.

The Pet Xclusives account

A reconnect on one customer's Meta integration — call the business Pet Xclusives, because that's genuinely close to what it sold — returned the wrong Facebook Page and Instagram account. Not a broken connection. A real, working connection to the wrong business. Four posts sat queued for review. Approving one would have published Pet Xclusives' content to a completely different account, under a different name, in front of a different audience.

Nothing in the data distinguished "the Page this business owns" from "a Page this person happens to have admin rights on." A single Meta permission grant can return a dozen Pages belonging to a dozen different businesses, and the API has no field for "the one you meant." We caught it by reading the actual rows in the database before anything shipped, not because a test failed — no test could fail, because nothing about the data was wrong. It was a real connection. It just wasn't the customer's.

We added an automated check afterward — anything that catches two publishing-enabled connections on one platform for one business, which is the shape this specific mistake takes. But the check that actually matters is still a person looking at a screen and asking "is that really their account?" We wrote that down as a rule, not just a fix: some checks can't be automated away, and pretending otherwise is how this kind of mistake survives a code review.

The token that outlived the goodbye

Disconnecting a social account is supposed to mean disconnected. We found a bug where only one platform's disconnect flow actually cleared the stored credential — every other platform flipped a status flag from "connected" to "disconnected" and left the real, usable, decryptable access token sitting in the database untouched.

Three rows, on a test environment, held live posting access to a customer's real Facebook, Instagram, and Threads accounts — accounts whose owner believed, correctly by every signal we'd shown them, that access had been revoked.

The uncomfortable part of this one isn't the bug. Bugs happen. It's that "disconnected" had looked completely correct from every angle we normally check: the UI said disconnected, the status column said disconnected, a person clicking around the product would see disconnected everywhere they looked. The only way to find the actual problem was to ask a more specific question than the one the UI was answering: not does this say disconnected but can this token still do anything. We fixed the bug in one commit and wrote a test that checks the second question specifically, because the first question turned out not to prove what we thought it proved.

Sandbox is not a toy

We run two environments — a production database for real customers, and a sandbox for testing. Early on, a script pointed at what we believed was the sandbox spent several hours reading real customer connection data, because the local environment file had quietly been pointed at the wrong one. Nothing was written; this was a read-only mistake. But it produced a wrong conclusion — "there's nothing to migrate" — that we would have shipped as fact, when the real production database had exactly the migration work we'd been told to do.

The fix was procedural, not code: before any script touches a database, check which one it's actually pointed at, by asking the database itself which host it is — not by trusting an environment variable's name. "Sandbox" is not a synonym for "safe to be careless in," either. A sandbox database still holds real OAuth tokens for real third-party accounts; the boundary that matters isn't which database you're reading, it's what the token in that row can still do.

A default nobody ever sets

Business.timezone defaulted to UTC. Nobody chooses UTC on purpose for their own business; it's what happens when a form field is never filled in. We didn't notice for a while, because a default that's silently wrong doesn't throw an error — it just quietly does the wrong thing, forever, for anyone who never touched the setting.

Seven of ten production businesses were on it. Scheduled posts for one customer were firing at four in the morning, Central time, instead of nine — every single day, for as long as the schedule had been running, with no error anywhere to say so.

We now capture the creator's real timezone at signup, and separately, a running schedule that's still on UTC gets flagged with a warning rather than posting silently. The lesson generalizes past this one field: a default that's wrong for almost everyone is a bug that hasn't been reported yet, not a safe fallback. If a setting is never touched, ask whether the default is actually right, or just harmless-looking.

When the tests said yes and meant nothing

The one that stung the most, honestly: three test files were passing — green, every run — for reasons that had nothing to do with what they claimed to check. One test asserted against a value it read from a config file at runtime, instead of the fixed default it was supposed to be pinning; whatever the config happened to say, the test agreed with it. Another used a regular expression to check file contents that broke the moment a comment in that file happened to contain a character sequence the regex treated as a delimiter — so it had been silently checking nothing for months. A third's file-matching logic used forward slashes on a codebase that, on Windows, produces backslashes, so it had never actually run its own check at all.

None of these were subtle logic bugs. They were the test equivalent of a smoke detector with no battery — it looks exactly like a working smoke detector until the day it matters. We now ask, before calling anything done: is the suite green, and do I actually know why each test that used to fail now passes? A test you've learned to ignore is worse than no test, because it still reads as coverage.

What actually changed

None of this is a claim that we've reached some finished state of "safe." The honest version is narrower: we now have a real, growing, cited list of the specific ways this product has hurt someone, we check it before every production deploy, and we add to it only when something has actually happened — never a hypothetical, because a list that mixes real incidents with imagined ones stops being useful the moment you can't tell which is which.

If you're evaluating a tool that touches your customers' social accounts, your published content, or your brand's voice, "we take security seriously" is not a fact you can check. Ask what's actually gone wrong, and what changed because of it. If the answer is nothing, either nothing has shipped yet, or nobody's looking closely enough to find it.

We'd rather you saw ours.

Curious what Zolt actually does day to day? See how it works, or go straight to pricing.