Stop Burning API Budget With 5 Architectural Patterns for LLM Token Efficiency

A lot of LLM API spend is unnecessary. The same system prompt resent on every call, whole files where one function would do, a frontier model answering questions a small one handles fine, or agent loops nobody capped.

LLM token efficiency is about cutting that waste in the architecture around the model, without making the answers worse. It matters more as agentic features turn one user action into many model calls.

According to Microsoft's own architecture guidance for AI startups, the biggest cost and latency drivers are how much context you send, how many round trips you make, and whether tools run one after another.

To fight back against ever-growing costs and latency, we will explore five patterns that fix that, what each one saves, and how each one can be misused.

Where Do LLM Applications Burn the Most Tokens?

Token waste comes from defaults: frameworks and tools send everything, every time, to whichever model was configured first. An agent pulls a full database record or a 2,000-line file into context, the model processes thousands of tokens to use a few dozen, and the cost per user action climbs with every feature you add, until the margin on an AI feature depends on how chatty your prompts are.

In a production study of AI agents reading enterprise APIs through the Model Context Protocol, Nikhil Ramashasthri of the car-sharing marketplace Turo found that 27–42% of the fields in typical API responses were infrastructure metadata, such as audit timestamps and internal IDs, that meant nothing to the agent but still consumed its token budget.

The usual leaks fall into five groups: repeated context (system prompts, tool definitions, chat history), oversized context (raw files and payloads), oversized models (one top model for every request), runaway output (long reasoning and verbose answers), and runaway loops (retries and agent iterations with no ceiling). Each of the five patterns below targets one of them.

How Do Prompt Caching and Semantic Caching Cut Repeated LLM Costs?

Prompt caching makes the provider reuse the processed start of a prompt you have already sent, so you pay a fraction of the price for it the next time. Semantic caching goes further and skips the model call entirely when a new question means the same thing as one you have already answered.

Provider-side caching is the cheapest win on this list. On current models, Anthropic bills cache reads at 0.1x the base input price (0.05x on Claude Opus 5.5 and Sonnet 5.5), charges 1.25x to write a five-minute cache entry, and refreshes the entry each time it's used. OpenAI caches automatically from 1,024 tokens on GPT-5.6 and later, with reads at 0.1x and writes at 1.25x. Anthropic's docs also state that caching has no effect on the response itself, so the saving costs you no accuracy. The one engineering rule is prompt order: stable instructions, tool definitions and reference material go first, and anything that changes per request (timestamps, user data) goes last, or it breaks the reusable prefix.

A self-hosted semantic cache sits in front of the API. Each incoming question is turned into an embedding, compared with past questions, and answered from the cache when the match is close enough. Redis supports this through the RedisVL SemanticCache, which exposes a distance threshold, a time-to-live and filterable fields, and Valkey, the open-source Redis fork, now has native vector search to build the same thing on.

Semantic caching pays off for support bots and internal Q&A, where users ask the same thing in different words. It's riskier anywhere a near-match can be wrong. The authors of a June 2026 survey of token-level optimizations point out that "How do I activate a membership?" and "How do I cancel a membership?" sit close together in embedding space but need opposite answers. A tight threshold, short TTLs and cache entries scoped to each user or tenant are the minimum.

What Is Semantic Token Compression, and How Does AST Parsing Shrink Code Context?

Semantic token compression removes the tokens that carry nothing the model needs for the task. It works at three levels: dropping irrelevant data before it reaches the prompt, summarizing what must stay, and, in research tools such as LLMLingua, deleting low-information tokens from the prompt text itself.

The first level gives the biggest and safest gains. In the Turo study, stripping infrastructure metadata from API responses cut tokens by 34.1% on average, and letting the agent request only the fields it needed cut them by 86.3%. The same paper notes that gzip and other byte-level compression shrink what travels over the network, but the model still reads every token, so the bill doesn't move. The numbers come from one company's data and the study didn't measure answer quality, so treat them as an order of magnitude rather than a forecast.

Code is where this matters most for developer tools. A naive coding assistant pastes whole files into the prompt. An AST-based one parses the codebase into an abstract syntax tree, the structure a compiler sees, and sends only the definitions and signatures that matter for the task. Aider, an open-source coding agent, uses tree-sitter to map every function, class and reference in a repository, ranks them by how they connect, and fits the most relevant ones into a repo map of about 1,000 tokens by default. The model sees the shape of a large codebase and the two functions you are changing, without the 2,000-line files around them.

For conversations, the equivalent is to summarize older turns into compact state instead of resending the full transcript, which Microsoft recommends alongside a hard context budget per request.

When Should You Route LLM Requests to Smaller, Cheaper Models?

Route to a small model whenever the task is simple and its output can be checked: classification, extraction, rewriting, short answers and boilerplate code. Keep the large model for multi-step reasoning, ambiguous requirements and complex refactoring. Sending everything to one model, as Microsoft puts it, forces simple tasks to use the same resources as complex ones, which raises both latency and cost.

There are two ways to do it. A router predicts difficulty before the call and picks one model. A cascade starts with the cheapest model, checks the result, and escalates only when the check fails. In research benchmarks, FrugalGPT matched GPT-4's performance at up to 98% lower cost, and RouteLLM cut costs by more than 2x without losing response quality. Microsoft's guidance describes the cascade in practical terms: start with a lightweight model and escalate when validation fails, confidence is low or the task needs deeper reasoning.

In a coding pipeline, that could mean Claude Haiku 5.5 writes tests, docstrings and boilerplate, while Claude Opus 5.5 handles the refactoring plan and the final review. Predicting difficulty is unreliable, though, for reasoning, code and domain-specific questions. The survey authors note that production platforms keep agent and tool-calling traffic on the main model for stability. Routing works best on high-volume, low-stakes traffic, and every routing rule needs an evaluation set that shows quality didn't drop.

How Do You Cap Reasoning and Output Tokens Without Losing Accuracy?

Set explicit budgets for reasoning and output length, and ask for reasoning only when the task needs it. Reasoning models bill for the thinking they do, and they overthink: the survey cites DeepSeek-R1 producing hundreds of reasoning tokens for "100 + 200 - 300 = ?".

Providers now expose controls for this, including Anthropic's extended thinking budget, Gemini's thinking budget and Qwen3's thinking and non-thinking modes. Combine them with an output cap such as max_completion_tokens and prompts that don't ask for justifications, citations or summaries the user will never read. Research on adaptive thinking points the same way: CoThink has a regular instruct model draft a short outline and a reasoning model expand it, which cut token use by 21.1% while keeping accuracy on four reasoning models.

Uniform budgets backfire, though. The same survey warns that one-size-fits-all compression of reasoning hurts accuracy on harder problems. Give the budget per task type, so a classification call gets almost no thinking and an architecture review gets plenty.

Which Workflow-Level Cost Controls Stop Runaway Agent Spend?

Hard limits at the workflow level stop the worst bills: a cap on agent iterations, a cap on retries, a token budget per tenant and per feature, and an alert before the budget runs out. The other four patterns lower your average cost per request, while these controls cap the worst case.

Agents compound costs. One user request can trigger retrieval, several tool calls, validation and retries, and each step feeds a longer context into the next. Microsoft's guidance is to bound retries to one or two extra steps, cap the iterations of agentic retrieval loops, and pass compact intermediate state between steps instead of full transcripts or raw tool output. It also recommends tracking cost per request, including tool calls and external services, since cost spikes usually show up in latency too.

The natural place to enforce limits is an AI gateway between your applications and the model providers. Microsoft describes it as a control plane that centralizes authentication, token budgets, rate limits, routing, failover and telemetry. The survey's production gateway enforces requests per second, tokens per minute and concurrency separately for each tenant, user, application and model, so one customer's batch job can't drain the budget for everyone else.

What Does LLM Token Optimization Require Before It Pays Off?

Token optimization needs measurement, an evaluation set and a clear view of your data obligations before any of the patterns above are worth building. Without them, you can't tell a saving from a regression.

Start with observability. Log input and output tokens, model, cache hits and cost for every request, tagged by feature and customer. Microsoft suggests a simple test: if you can't explain your 95th-percentile latency from those metrics, you don't have enough visibility yet. Then build an evaluation set from your own traffic. The survey authors note that public benchmarks suffer from data contamination and rarely match real business tasks, so a router or compression step that looks fine on a leaderboard can still break your use case.

Caches create compliance questions. A semantic cache stores copies of prompts and answers, which may include personal or regulated data, so it falls under the same GDPR, HIPAA or SOC 2 controls as your primary database, including retention, deletion requests and tenant isolation. Provider caches have retention settings of their own: OpenAI's docs say organizations without Zero Data Retention default to a 24-hour cache retention option on earlier models, which is worth checking in a regulated product.

Some of this isn't worth doing at all. If your monthly API bill is smaller than a week of an engineer's time, provider caching and output caps are probably enough. Gateways, routers and semantic caches add moving parts, and Microsoft's own advice is to treat them as deliberate optimizations after you've removed the obvious waste.

How Should a Team Start Cutting LLM Token Costs?

Token efficiency comes from treating tokens like any other metered resource: measure where they go, remove what the model doesn't need, match each task to the cheapest model that does it well, and put ceilings on everything that can loop. Instrumentation and evaluation come first, provider caching and context pruning give the quickest returns, and routing and semantic caching pay off once volume justifies the extra infrastructure. At Monterail, we design and build LLM pipelines for products in regulated industries, and we can help you review an existing pipeline, find where the spend goes and decide which of these patterns are worth it for your volume. Get in touch if your AI feature's margins depend on its token bill.