Shipping LLMs With a Spending Ceiling

Every AI-heavy product ships with an invisible bill attached. A single misbehaving loop, a single input that triggers a runaway chain of tool calls, a single infrastructure event that quadruples your token consumption, and the monthly invoice has a comma you weren't expecting.

For customer-facing products, that risk is usually acceptable — the spend correlates with revenue. For internal AI systems that run unattended — monitoring, orchestration, operational automation — it's much less acceptable. The system that saves you $40,000 of engineering time a month isn't the same system if it silently spends $60,000 on inference.

We ran into this building an autonomous email-infrastructure orchestrator. Here's the architecture we settled on.

Two tiers, not one

Most LLM-in-the-loop designs are single-tier: the language model makes the decision, and if it's unavailable, the system stops. That's a fine design for an interactive chatbot where humans will notice. It's a terrible design for autonomous infrastructure that has to keep running through provider outages, model deprecations, and rate-limit storms.

Our pattern is two tiers that cover the same decisions:

Tier 1 — The advisor. A language model is consulted for genuinely judgment-based calls — interpreting mixed signals, reasoning about non-obvious tradeoffs, catching novel failure modes. The advisor is powerful but not trusted for anything mission-critical. Its recommendation is logged, audited, and used only if it arrives within a budget and latency window.

Tier 2 — The deterministic rules engine. Every decision the advisor can make has a policy-based equivalent. If the advisor is unavailable, over budget, or obviously hallucinating, the rules engine takes the decision cleanly. No user-visible failure. No pager alert.

Either tier alone is sufficient to run the system. The advisor makes the system smart; the rules engine makes the system safe.

The spending ceiling

The budget layer sits between the orchestrator and the advisor. Before any paid call is dispatched, the budget layer checks:

• Cumulative monthly spend against the configured ceiling
• Per-decision-class spend (e.g., "warm-up pacing decisions account for no more than X% of monthly budget")
• Rate per hour against a configurable throttle

If any check fails, the call is short-circuited and the rules engine handles the decision. The advisor never sees the request.

This seems obvious. The reason most systems don't have it is that it requires you to actually enumerate your LLM calls, price them, and commit to a ceiling. Most teams skip that work and instead "monitor for anomalies." Monitoring is an alerting mechanism, not a control mechanism. By the time you alert, you've already spent.

What the advisor is actually good for

The hard part of the design isn't the fallback — it's deciding which decisions deserve an advisor at all. We've settled on three criteria:

1. The decision has to be genuinely judgment-based. If the rules engine can produce the right answer 99% of the time, asking the advisor adds cost without adding value. Save the advisor for the 1% where the rules engine would be wrong, or for decisions the rules engine can't make at all (e.g., "this bounce pattern across six receiving domains — is it a reputation problem or just weekend mail-volume dips?").
2. The cost of the advisor being wrong has to be bounded. If an advisor hallucination can trigger a cascading cleanup that breaks the whole system, the advisor should not be in that loop. Reserve LLM judgment for decisions whose worst case is "do nothing" or "escalate to a human."
3. You have to be able to log the full prompt and the full response. Every advisor call that actually happened needs to be auditable a month later. If you can't tell why a decision was made, you can't trust the system in production.

What changes with this discipline

The biggest second-order effect is that engineers stop treating the LLM as magic. Once every advisor call has a measurable cost, a bounded worst case, and a deterministic fallback, the language model stops being the protagonist of the system — it's just one component, sitting in a pipeline with everything else.

That reframing matters. An LLM that can't be replaced is a vendor lock-in risk. An LLM whose decisions can't be audited is a compliance risk. An LLM whose cost isn't bounded is a financial risk. Two-tier decision architecture eliminates all three at the cost of building the boring deterministic layer most AI-first teams would rather skip.

That work is not glamorous. It is how you ship an AI product that survives past the demo.