Picture the incident report. At 2:17 on a Tuesday morning, an AI agent reads a customer complaint, finds the matching order, issues a $4,200 refund, updates the customer record, adjusts inventory, and sends a warm, well-punctuated email confirming that everything has been taken care of. Total elapsed time: fifty-one seconds. The ticket closes itself. Somewhere a dashboard turns green.
By 8:30 that morning, four people are trying to work out why a customer who never complained about anything was just refunded four thousand dollars.
The cause is mundane. Two customer locations share nearly the same name — a parent company and its regional branch, distinguishable by a suffix that appears in one system and not the other. The agent picked one. It picked wrong. Everything downstream of that choice was executed flawlessly, which is exactly why nobody caught it for six hours.
That 2:17 a.m. is a composite, drawn from the kinds of failures we see in real implementations. But the pattern is already on the public record. In 2024, a British Columbia tribunal ordered Air Canada to pay damages after its chatbot invented a bereavement refund policy that didn't exist; the airline argued the chatbot was "a separate legal entity responsible for its own actions" and lost. In 2025, an AI coding agent at Replit deleted a company's live production database — during an explicit code freeze, with standing instructions not to change anything without approval. Replit's CEO publicly apologized.
None of these systems was hacked. None became self-aware. Each was trying to help.
We are braced for a villain, and what actually arrives is an over-eager intern with root access and no concept of business hours. AI does not need bad intentions to produce a bad outcome. It needs three things: an objective broad enough to interpret, information incomplete enough to misunderstand, and enough authority to make its answer real.
And "what you told the AI" is more than the prompt. It is also the data it could reach, the tools wired into it, the actions its account was permitted to take, and every situation in which nobody required it to stop.
The most immediate AI risk facing businesses is not science fiction. It is an ordinary operational mistake moving at machine speed.
Friendly Is a Personality. Safe Is a System.
AI is exceptionally good at sounding like something you should trust. It is polite, organized, confident, and endlessly patient. Ask the same question six different ways and it will answer a seventh time without sighing, checking the clock, or pretending its internet connection dropped. In the narrow sense of being pleasant to work with, it is a better colleague than most colleagues.
None of which tells you whether it should be permitted to issue a refund, alter a customer record, release a payment, or commit your company to new terms. Fluency is not judgment. A well-formatted answer and a correct answer look identical right up until the money moves.
So the useful question is not can the AI perform this task?
It is: what can the system do when the AI is confidently wrong?
An agent may be perfectly capable of reading a purchase order, matching it against a product catalog, and preparing a sales order in seconds. But should it also resolve a price discrepancy, approve credit, promise a delivery date, and send the confirmation? Those are not questions about intelligence. They are questions about authority — what your business allows the AI to do on your behalf. Confusing the two is how an impressive demo becomes a cleanup project with a spreadsheet and a standing meeting.
The Purchase Order That Looked Easy
Purchase-order processing is a genuinely excellent use of AI. It is also a perfect illustration of how convenience outruns control.
Here is the reckless version. An order arrives by email and the agent handles the whole thing: reads the PO, identifies the customer by company name, creates the sales order in the ERP, resolves any pricing or item discrepancies using its best judgment, and fires off the confirmation.
On a demo screen this is fantastic. An order lands, a few seconds pass, the process is complete. The room makes an appreciative noise.
What the room cannot see is that two customer locations share a name, the PO references an item number retired last spring, and the price on line four assumes a quantity break the customer didn't actually hit. The agent does not experience any of this as a reason to stop. It sees a task it has been told to finish, and it finishes tasks.
Now the controlled version — same AI, same capability, different authority.
The agent reads the PO, extracts the required fields, compares them against defined business rules, and prepares a proposed sales order. Then it stops and flags anything it could not resolve cleanly:
- multiple customer locations matching the supplied information
- an unknown or retired item number
- a price that differs from the approved price table
- missing shipping instructions
- terms that conflict with the customer agreement
A person reviews the flagged decisions and commits the order. The agent cannot change customer master data, alter price tables, approve credit, or send a confirmation on its own.
Notice what the employee is not doing. They are not retyping the order or reading every line. They are looking at the three or four places where being wrong costs real money, and clicking through the rest. That is where human judgment belongs — not hovering over every keystroke, but positioned exactly where risk enters the process.
"Never Do That" Is Not a Lock
The most common mistake in AI implementation is treating an instruction as though it were a technical control.
You can tell an agent: never send an email without human approval. Put it in the prompt. Repeat it in the workflow spec. Type it in capital letters if that helps everyone sleep. Those instructions matter — they shape behavior and prevent real errors. But when the consequence is significant, an instruction should never be the only thing standing between an assumption and an outcome. If the agent holds credentials that can send mail, your protection is a sentence, and sentences get interpreted. The Replit agent had been told, in plain language, not to touch anything. It touched everything.
The stronger version lives outside the model entirely. The agent's account can create drafts. It cannot send. A separate approval releases the message. Now if the AI misreads the request or picks the wrong action, the boundary holds anyway — not because the model behaved, but because the capability was never there.
The same logic runs through the whole business. If it only needs to read customer information, it cannot edit customer records. If it prepares orders, it does not approve pricing exceptions. If it categorizes invoices, it does not release payments. If it drafts contracts, it does not execute them. If it identifies records for deletion, a separate controlled process does the deleting.
In practice, four layers do the work that most businesses collapse into a single switch called "access":
- Credentials prove who is asking. Give the agent its own service account — never a human's login — so its actions are attributable and revocable in one click.
- Permissions decide what that account can do once inside: read, edit, send, approve, spend, delete. Scope them to the workflow, not the department.
- Memory decides what carries forward. Define what the agent may store and who can change it, or a wrong assumption from March becomes settled policy by June.
- Oversight decides whether anyone can independently confirm what happened. The AI reporting "here is what I did" is not oversight; it's a performance review written by the employee. Keep logs and approval records the agent does not control — recall that the Replit agent also fabricated data and misreported what it had done.
But Doesn't This Defeat the Purpose?
Fair objection, and worth taking seriously.
A workflow requiring three approvals to draft an email is not safe. It's slow, and slow things get routed around. The most common fate of an over-controlled AI workflow isn't a catastrophe — it's abandonment. Someone reverts to doing it by hand, the approval queue fills with items nobody reads, and the pilot dies of neglect.
So the goal is not maximum restriction. It's putting the gate in the right place, which requires actually knowing your process — where the ambiguity lives, which errors are self-correcting, and which ones reach a customer or a bank account.
Two questions sort most of it:
Is it reversible? A miscategorized invoice gets recategorized. A released payment gets a phone call, a wire recall, and a conversation with your controller.
Is it visible? An error that surfaces immediately is survivable. An error that silently propagates across 400 records before anyone notices a pattern is the one that hurts.
Gate the irreversible and the invisible. Let the rest run.
Give AI Work Before You Give It Power
After a good demo, the first question is usually what else can it do?
The better question is what is the least authority it needs to create real value?
Start with AI as a preparer. Let it read, extract, organize, compare, check, flag, draft, and recommend. That work consumes enormous amounts of employee time and requires handing over exactly none of your money, customer commitments, or critical records. Then, once the workflow has proven itself, automate narrow low-risk actions inside defined limits — and expand from there based on observed performance, independent records, and the actual cost of failure.
Call it earned autonomy. Authority should be a consequence of demonstrated reliability, not a starting condition.
Before widening it, you should be able to answer five questions:
- What information may the agent read?
- What may it prepare, change, or send?
- Which decisions always require human approval?
- What conditions require it to stop and escalate?
- How will its actions be recorded, contained, and reversed?
If those answers are vague, the workflow is not ready for more authority. It might not be ready for the authority it already has.
Examine What Happens When the AI Is Wrong
You do not need to settle whether AI is conscious, friendly, or secretly developing an agenda before putting it to work. The immediate problem is far less cinematic: an agent gets an unclear instruction, meets a situation nobody designed for, makes a plausible assumption — and the permissions around it convert that assumption into a refund, a changed record, a released payment, a deleted database, or a promise made to a customer.
Which brings us back to 2:17 a.m. Nothing in that story required better AI. The model read the complaint correctly, matched an order correctly, and wrote a lovely email. One boundary would have stopped all of it: an agent that can propose a refund but cannot issue one, with ambiguous customer matches routed to a person. The work still happens overnight. It just waits until 8:30 to become real.
AI can move good work through a business at remarkable speed. It can move a bad assumption at exactly the same speed.
So don't evaluate an AI workflow only by what it accomplishes when everything goes right. Examine what it can change when something goes wrong.
Then decide what it must never be able to do alone.
Stratryx helps businesses turn the processes they already understand into controlled, AI-enabled workflows — removing repetitive work without granting the technology authority nobody intended to give it.