LLM Application Cost Optimization for a Team Whose AI Bill Had Grown Faster Than Their Revenue
The model worked. The bill broke the unit economics.
A team operating a customer-facing AI product whose model usage cost had grown materially faster than the product's revenue base. The product was succeeding on capability — users liked the AI features and used them — but the unit economics had degraded to the point where leadership was concerned about the trajectory.
I.Problem Statement
Without intervention, the AI cost curve would consume a meaningfully larger share of revenue than the business model could absorb. The team needed cost optimization that didn't degrade the user experience users were responding to. The leadership wanted both to be true — the cost trajectory had to bend down and the user-perceived quality had to stay where it was.
II.Methodology
A cost optimization engagement targeting the LLM application's request path.
Caching was implemented at multiple levels. Identical and near-identical queries returned cached outputs rather than generating duplicate model calls. Semantic caching — where queries that were different in surface form but equivalent in intent returned the same output — absorbed a substantial portion of the request volume that prior implementations had been treating as unique.
Model tiering was introduced. Queries were classified by complexity; the cheap model handled the common simple cases adequately; the expensive model was reserved for the cases that genuinely required it. The classification model was itself cheap, so the routing overhead was negligible against the savings.
Prompt engineering reduced per-request token consumption. The original prompts had grown organically without much attention to length; rewriting them with cost in mind reduced average input tokens substantially without measurable quality degradation. Output instructions were tightened to produce the format the application needed without verbose preamble.
Batching was introduced for the request types that could tolerate it. Several high-volume queries that didn't need to be processed in real time were batched and processed asynchronously, taking advantage of the substantial cost difference between real-time and batch model APIs.
Retrieval-augmented generation was applied to a class of queries that had been answered through the model alone. Adding retrieval gave the model the information it needed in context, allowing a smaller model to produce equivalent outputs to what a larger model had previously produced through capability alone.
Cost tracking was instrumented per use case, per user cohort, and per query pattern. The team could see exactly which patterns were driving cost and could prioritize optimization against the patterns where the savings were largest.
III.Results & Discussion
AI cost as a portion of revenue dropped substantially without measurable user-perceived quality degradation. The product's unit economics returned to a sustainable trajectory. The team retained the ability to apply more expensive models where they genuinely produced better outputs while pushing the bulk of routine traffic onto cheaper paths. The architectural changes set up the application to absorb the next stage of growth without the cost curve becoming the primary engineering problem.