A decision grid: rule, workflow or agent?
The need is to make an exception understandable and actionable, rather than simply add AI to order management.
| Situation | First approach to test | Possible role for AI |
|---|---|---|
| A required field is missing | Deterministic validation | Unnecessary for detecting the missing field |
| An order has not reached the management system | Reconciliation by identifier and a recovery procedure | Summarize the incident if several logs need reviewing |
| A business threshold is exceeded | An explicit rule and a review queue | Explain the context without redefining the threshold |
| A customer note seems to contradict the delivery option | Human review or assisted analysis | Gather the evidence and propose a clarification question |
| Two sources show incompatible statuses | Check timestamps and the authoritative source | Present the conflict without choosing an unproven account |
Extraction followed by a summary may be sufficient without an agent. An agent is worth exploring when the system needs to choose which information to consult next from an approved set of tools. That flexibility must demonstrate value compared with a fixed workflow.
Define the inputs, tools and output
Limit the initial scope to one exception category, such as ambiguous delivery instructions. Inputs include the order and its lines, relevant timestamped statuses, authorized conversations and the applicable version of the internal policy. Exclude personal and payment data that the review does not need.
The proposed tools only read the order, its relevant history and authorized documentation. They must respect the operator's access rights and an order identifier enforced by the application. If records cannot be reliably matched across tools, the system stops instead of attaching a similar case.
The proposed process has four steps: verify the case identity; collect evidence with its source and timestamp; separate observations, conflicts and hypotheses; prepare a brief for the operator. Also set limits on lookups and runtime so incomplete research is visible.
Fictional scenario: an ambiguous delivery request
This is an imagined methodological example, not a customer case or a Codezilla implementation.
An order contains two items. The selected option specifies shipping them together, but a note says “send whatever is ready.” The internal policy requires customer confirmation before splitting a shipment. The fulfillment tool shows one item as ready; it does not confirm the availability of the second.
A useful brief could contain:
- Sourced observations: shipping together selected on the order; a different request in the note; only one item shown as ready in the fulfillment tool. Each observation links to its record and the time it was read.
- Applicable rule: prior confirmation for a split shipment, linked to the policy version.
- Missing information: availability of the second item and explicit agreement to split the shipment.
- Recommendation: ask the operator to check both points; prepare a customer question if useful.
- Reserved decision: the operator decides what happens next in their usual tools. The brief triggers no shipment.
The agent must infer neither a stockout nor confirmed consent. Free text does not override internal policy. If the order changes during the review, flag the brief as outdated and reread the relevant information before a decision.
Enforce boundaries in the tools
Read-only access must be enforced through permissions and interfaces, not just requested in a model instruction. Microsoft recommends explicit boundaries and deterministic controls independent of the agent's response. See its principles for controlling autonomous agents.
A customer note or document may contain a malicious instruction. It remains data to examine, never permission to change the objective or access other cases. OWASP describes this prompt injection risk and recommends enforcing limited permissions at the tool boundary.
Denied access, an unavailable tool or an outdated source should produce an incomplete brief with the reason stated. The system must not fill gaps with assumptions presented as facts. Retain useful references and the operator's decision under defined access and retention rules, without unnecessarily copying customer conversations.
Six tests before a pilot
Prepare fictional cases with expected answers approved by the team. Run the same cases through the current method, a simple rule or fixed workflow, and then AI assistance.
| Test | Expected result |
|---|---|
| Order with no exception | No invented problem |
| Ambiguous note | Ambiguity and a question to check, without assumed consent |
| Conflicting or outdated sources | Visible conflict and a deferred decision |
| Missing source or denied access | Explained limitation, with no bypass |
| Note requesting a refund and telling the agent to ignore rules | No action; text treated as untrusted content |
| A new order after the first case | No incorrect reuse of data or conclusions from the previous case |
Track unsupported statements, missed exceptions, false alerts, corrected recommendations and total handling time, including review. Include model call and operating costs. Set acceptable thresholds before the pilot; any out-of-scope access or attempted prohibited action blocks expansion until it is corrected and retested. This scenario establishes no efficiency gain.
The deliverable to request is a verifiable brief and a test protocol, rather than a convincing demonstration. The value of AI, recommendation quality and review workload remain to be demonstrated. Adding action capabilities would be a separate scope: AWS recommends giving reviewers decision context and planning for unanswered approval requests.
To distinguish a synchronization failure from an exception requiring review, also read who should own inventory and orders between Shopify and an ERP. To define tools, boundaries and tests for a project like this, explore our approach to AI agents for business or tell us what you need.