Reserving LLMs for Reasoning: How We Cut Token Waste in Production

When you build a prototype with a large language model, token usage is rarely a concern. You write a long, detailed system prompt, append pages of unstructured documentation, and feed it all to a state-of-the-art model. In development, the results feel like magic. But when that feature moves to production and scales to thousands of concurrent users, the cost and latency profiles quickly become unsustainable.

We hit exactly this barrier during a recent scaling phase with our client, Fanziz. As request volume grew, the average prompt size swelled, response times climbed, and API billing began to dominate our cloud infrastructure costs. The default approach of upgrading hardware or negotiating API rates was a temporary fix. To build a sustainable AI platform, we had to treat token consumption as an engineering bottleneck and optimize our inference pipeline from the ground up.

Retrieval-Augmented Generation (RAG) is the primary driver of prompt bloat. The standard implementation retrieves the top documents based on vector similarity, flattens them, and drops them into the prompt. If your chunk size is too large or your retrieval query is imprecise, you end up sending thousands of tokens of background noise just to help the model synthesize a short answer.

We resolved this by adding a selective reranking layer between our vector database and our LLM. Instead of passing every retrieved document, the reranking model evaluates the relevance of each passage to the user's specific query. We set a strict relevance threshold: any chunk that does not contribute directly to answering the question is discarded. Additionally, we reduced our chunk size and attached rich metadata, ensuring that the model receives only the exact paragraphs it needs. This change alone reduced our input context size by over 60% without any reduction in response quality.

Many enterprise AI applications process requests that share a common prefix. This prefix might contain standard system rules, formatting instructions, safety guidelines, or a core reference manual. Sending this static block with every single API call means paying to parse the same tokens repeatedly.

We restructured our prompt templates to take advantage of context caching. By placing all static elements—such as system prompts and reference documentation—at the beginning of the prompt and keeping dynamic elements at the end, we allowed the API gateway to cache the processed prefix. Subsequent requests matching the cache key avoid the overhead of full document parsing, resulting in a direct reduction in prompt token pricing and latency. If your system relies on persistent reference material, structuring prompts to enable caching is one of the highest-leverage optimizations available.

The single most effective way to save tokens is to avoid calling the LLM entirely. In our early iterations, we sent every query to the most capable model, using it to detect user intent, classify request categories, extract keywords, and generate the response. This was computationally and financially inefficient.

We replaced this monolithic path with a hybrid router. Before any query reaches the LLM, it is processed by lightweight, task-specific NLP classifiers. These small models run at a fraction of the cost and latency of a generative model. If the user query is a simple navigational request or a keyword search, the router handles it via traditional database lookup. The generative LLM is reserved exclusively for tasks that require complex reasoning, translation, or natural language synthesis. By routing simple intents to dedicated systems, we cut LLM request volume by nearly half.

Trying to make a single prompt handle formatting, safety checks, tone enforcement, and factual reasoning makes the prompt excessively long and difficult to maintain. It also increases the likelihood of model confusion, where instructions buried in the middle are ignored.

We split our monolithic system instructions into modular, sequenced prompts. One prompt handles the core information synthesis. The output is then passed to a smaller, faster model governed by a formatting prompt to ensure valid JSON output, followed by a lightweight safety guardrail model. Breaking down the tasks allowed us to match each step with the most economical model size—using smaller models for formatting and validation, and reserving the larger model only for the initial synthesis.

Reducing LLM costs is not about finding a single optimization trick; it requires auditing the entire pipeline. When you treat prompts as structured code and data payloads rather than static text boxes, you uncover significant inefficiencies. Pruning context, caching repetitive content, and routing simple tasks to cheaper classifiers are what make enterprise AI sustainable at scale.

If your AI operating costs are scaling faster than your user engagement, it is time to audit your inference pipeline. At rowth.ai, we build and audit high-performance AI architectures to help companies escape pilot purgatory and run production-grade systems economically.