Controlling LLM Costs in Production: Lessons from Running Our Own AI Products

There is a specific moment every team building on LLMs eventually hits: the first genuinely scary API invoice. Usage grew, someone added a feature that calls the model in a loop, a prompt quietly ballooned to 6,000 tokens, and suddenly the "AI feature" costs more than the servers running everything else. We know the feeling firsthand — we run a subscription AI tools platform and an automated video-clipping engine, and we pay those model bills ourselves every month. Nobody reimburses us for sloppy prompts.

That changes how you engineer. When the margin on your own product depends on tokens, cost control stops being a slide in an architecture review and becomes a weekly operational habit. The good news: in our experience the first serious optimization pass typically cuts LLM spend by half or more without any measurable quality loss, because most early-stage AI features are wildly overspecified. Here are the four levers we actually pull, in the order we pull them.

Lever 1: Model tiering — stop using a flagship for everything

The single biggest source of waste we see is uniform model choice: every call, from "classify this ticket" to "write this analysis", goes to the most capable (and most expensive) model available. Flagship and lightweight models from the same family often differ in price by 10–20x. That means one question dominates your unit economics: which calls actually need the big model?

In our clipping engine, the LLM's job is to read a transcript and select highlight-worthy segments. Early on, everything went to a top-tier Claude model. When we tested tiering, we found that a mid-tier model handled the bulk of selection fine, and only ambiguous or long-context cases benefited from escalation. The pattern that generalizes:

• Cheap model by default for classification, extraction, formatting, routing, and short summaries.
• Escalate on signal, not on vibes: low confidence, failed validation, or explicit complexity markers trigger a retry on the stronger model.
• Measure per task, not per benchmark. Public leaderboards tell you nothing about whether the small model can do your narrow job. Build a 50-example eval from real production data and let it decide.

Tiering alone routinely cuts spend 40–70% for pipelines dominated by simple calls. It is also the lever with the best effort-to-savings ratio: often a config change plus an eval run.

Your LLM bill is not a cost of doing AI. It is a report card on how much of your pipeline you have actually thought about.

Lever 2: Caching — the tokens you never send are free

Caching for LLMs comes in two flavors, and teams tend to ignore both.

Prompt caching

Both Claude and GPT-family APIs support prompt caching: if the first N tokens of your request are identical across calls — the system prompt, instructions, few-shot examples, a reference document — the provider processes them once and charges a fraction (often around a tenth) for cache reads afterwards. The catch is that it only works if your prompts are structured for it: static content first, variable content last. A prompt that interleaves the user's input into the middle of the instructions breaks the cache on every call. This is a one-day refactor for most codebases and it compounds on every single request forever.

Response caching

Duller and even more effective: many "AI features" answer the same questions repeatedly. Product Q&A, standard document summaries, category suggestions for similar inputs. A hash-keyed cache in Redis with a sensible TTL means your second identical request costs zero tokens and returns in milliseconds. On features with repetitive inputs we have seen cache hit rates high enough that the model becomes the fallback, not the default path.

Lever 3: Prompt slimming — most prompts are 40% ritual

Prompts grow the way legacy code grows: someone adds a paragraph to fix an edge case, nobody ever deletes anything, and six months later you are paying for 3,000 tokens of instructions on every call, of which the model demonstrably uses half. Because input tokens are billed per call, prompt weight is a tax on your entire volume.

Our slimming routine, applied whenever a pipeline's cost drifts up:

1. Diff against reality. Take the current prompt, cut every instruction you cannot tie to an observed failure, and run the eval. In our experience, half the "critical" instructions change nothing.
2. Kill redundant few-shot examples. Three well-chosen examples usually match eight. Each example you delete is deleted from every future call.
3. Trim the retrieval payload. If you are doing RAG, sending ten chunks "to be safe" instead of the four that matter doubles cost and often hurts accuracy by burying the answer. Rerank, then send less.
4. Cap outputs. Output tokens usually cost several times input tokens. If the downstream consumer needs a JSON object, forbid the essay around it. Structured output modes exist; use them.

None of this is glamorous. All of it shows up on the invoice.

Lever 4: Batching and async — pay off-peak prices for off-peak work

Not every LLM call needs an answer in two seconds. Nightly enrichment, backfills, report generation, transcript post-processing — anything a user is not actively waiting for is a candidate for batch APIs, which both major providers offer at roughly half price in exchange for relaxed latency (typically within 24 hours, usually much faster).

The design consequence matters more than the discount: it forces you to classify every call site as interactive or deferrable. In our video pipeline, transcription and highlight selection run as queued jobs on dedicated compute — the user expects a notification when the clips are ready, not a spinner — so the entire heavy path is architected around throughput, not latency. (We wrote up that architecture in detail in how we built our AI video-clipping engine.) Once you have a queue, batching, retries, and rate-limit handling all get dramatically simpler too.

The habit that makes the levers stick

One warning from operating this stuff: cost optimizations decay. Prompts regrow, someone switches a call back to the flagship model "temporarily", a new feature ships without caching. The fix is not heroics, it is instrumentation:

• Log tokens and cost per feature, per call site — not just the account total. An aggregate bill hides which feature is bleeding.
• Track cost per unit of user value: per clip generated, per document processed, per conversation. That is the number that must stay below your price.
• Alert on drift, review monthly, and re-run your evals whenever you downshift a model.

And a contrarian note: sometimes the right decision is to not optimize yet. If you are pre-launch and total spend is a rounding error, obsessing over caching is procrastination — validate the feature first, then engineer the margin. The sequencing argument is the same one we make in why every AI project should start with a proof of concept: prove value, then industrialize.

The takeaway: LLM costs are unusually controllable compared to most infrastructure spend, because the waste is concentrated in a few visible places — wrong model tier, cold caches, bloated prompts, synchronous everything. Four levers, applied with a decent eval and per-feature metering, will usually halve the bill. We know because it is our bill.