Fanziz
Optimizing LLM costs without compromising AI quality
We redesigned Fanziz's entire inference pipeline — RAG-retrieved context, cached prompts, adaptive prompting strategies and hybrid NLP/LLM routing — so token usage and latency fell while answer quality went up.
The challenge
As Fanziz rolled out AI content recommendations, intelligent search and sports commentary, LLM request volume grew fast. Every request carried large amounts of context to keep answers accurate, so prompts kept getting longer, response times climbed and token spend rose with them — the same wall most AI products hit once features leave the demo stage. Rather than scale infrastructure, we set out to answer four questions: can we send less information and keep the same quality, does every request actually need a large language model, can repeated context be reused instead of regenerated, and can prompts be structured more intelligently instead of simply made longer?
What we built
Improvements at every layer of the pipeline rather than a single optimization. Articles and knowledge were converted into semantic embeddings so retrieval-augmented generation pulls only the passages a request actually needs — significantly smaller prompts, and often better answers, because the model is no longer reading around the point. Background context shared across many requests is cached and reused instead of rebuilt every time, cutting repeated tokens and response time together. Prompting strategy became a per-task decision: zero-shot where that suffices, one-shot or few-shot only where examples earn their tokens. And one sprawling system prompt was split into focused prompts governing formatting, safety, domain knowledge and response consistency — easier to maintain, and measurably more predictable output.
Using LLMs only where they add value
The single biggest saving came from recognising that not every problem requires a large language model. Intent detection, keyword extraction, named-entity recognition and basic text classification are now handled by lightweight NLP models, and a router picks the cheapest path that can answer a request correctly. LLMs are reserved for reasoning-intensive work where they provide clear value, which cut latency alongside operating cost.
Why it matters
By optimizing the whole pipeline rather than chasing model performance alone, Fanziz ended up with an AI platform that is faster, cheaper to run and easier to scale. High-quality responses, less wasted token consumption, and expensive models used only where they genuinely earn their place. For any business deploying AI at scale, that is the difference between an impressive pilot and sustainable AI infrastructure.