Token budget optimization: reduce LLM API costs by 60–75%
Meta description: Practical techniques for cutting LLM API costs at scale: tiktoken auditing, LLMLingua prompt compression, pgvector semantic caching, and request batching.
Tags: backend api microservices architecture nodejs
TL;DR
Running Claude or GPT-4o at production scale? Your token spend is almost certainly 2–3x higher than it needs to be. The four levers that matter: pre-flight token budgeting, prompt compression, semantic similarity caching, and request batching. Apply all four and you’re looking at 60–75% cost reduction with no measurable quality regression.
The problem nobody wants to talk about
The average production prompt sent to a frontier model carries 35–50% redundant tokens you’re paying for and the model largely ignores. Most teams get this wrong: they treat LLM cost as a billing problem instead of an engineering problem. They cap spend, throttle users, and call it done. The waste is structural, and it compounds with every request.
Here’s the architecture that changes this.
Layer 1: Tiktoken budget analysis
Before compressing anything, you need to know what you’re spending. tiktoken gives you per-request token visibility before the API call fires.
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
# INPUT_COST_PER_1M_TOKENS: check current pricing at your provider's pricing page
INPUT_COST_PER_1M_TOKENS = 5.00 # USD — verify before use
def audit_prompt(system: str, user: str) -> dict:
system_tokens = len(enc.encode(system))
user_tokens = len(enc.encode(user))
total = system_tokens + user_tokens
return {
"system": system_tokens,
"user": user_tokens,
"total": total,
"estimated_cost_usd": (total / 1_000_000) * INPUT_COST_PER_1M_TOKENS
}Run this across a week of production logs. In my experience, teams are consistently shocked to find their system prompts have bloated to 800–1,200 tokens through iterative editing — often three times their original size.
Layer 2: Prompt compression with LLMLingua
Microsoft’s LLMLingua uses a small local model to remove low-information tokens from prompts while preserving semantic content. The compression ratio is configurable, typically 2–4x with less than 5% quality degradation on structured tasks.
from llmlingua import PromptCompressor
compressor = PromptCompressor(model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank")
compressed = compressor.compress_prompt(
context,
rate=0.5, # keep 50% of tokens
force_tokens=["\n", ".", "?"] # preserve structure
)Compression rate vs. quality on summarization tasks:
| Compression Rate | Token Reduction | Quality Drop (ROUGE-L) | Recommended For |
|---|---|---|---|
| 0.7 | 30% | <1% | High-stakes generation |
| 0.5 | 50% | 3–5% | Standard API workflows |
| 0.3 | 70% | 8–12% | RAG context, retrieval chunks |
Apply aggressive compression only to retrieved context chunks, not to structured instructions. Compressing instruction content introduces unpredictable quality regressions that are difficult to catch at query time.
Layer 3: Semantic caching with pgvector
Exact-match caching misses the point. Two prompts asking “summarize Q3 results” and “give me a Q3 summary” should hit the same cache entry. That’s what semantic caching solves.
Store embeddings of previous prompts in PostgreSQL with the pgvector extension and do a nearest-neighbor lookup before every API call:
SELECT response, 1 - (embedding <=> $1::vector) AS similarity
FROM prompt_cache
WHERE 1 - (embedding <=> $1::vector) > 0.92
ORDER BY similarity DESC
LIMIT 1;A similarity threshold of 0.92 is a reliable starting point. Below 0.88 you start serving semantically adjacent but meaningfully different responses — a quality risk not worth the savings.
Cache invalidation matters here. Set a TTL of 24–72 hours depending on how frequently your source data changes, and invalidate cache entries explicitly on document update or re-ingestion events. A stale semantic cache hit is worse than a cache miss — it returns confident-sounding answers based on outdated context. Tie invalidation to your document pipeline, not just the clock.
Cache hit rates in document-heavy workflows (legal, finance, content pipelines) regularly reach 40–60% after the first few days of warm-up. That’s your single highest-leverage optimization.
Layer 4: Request batching
If your backend fires individual LLM calls per user event, you’re leaving efficiency on the table. Batch requests from a short time window — 200–500ms — into a single structured call:
const batchPrompt = `
Process each item and return a JSON array in the same order:
${items.map((item, i) => `[${i}] ${item.content}`).join('\n')}
`;In production deployments processing high-volume classification and extraction workloads, batching 10–20 items per call reduces per-item latency by 40–60% and cuts API round-trip overhead proportionally — the fixed network and cold-start cost gets amortized across the batch. At 200ms batching windows, throughput scales near-linearly with batch size up to the model’s context limit.
The catch: you need robust JSON parsing and per-item error handling since one malformed output can poison the batch. Validate structure before processing and implement per-item fallback retry logic.
Putting it together
The full pipeline: audit with tiktoken → compress context with LLMLingua → check semantic cache → batch remaining requests → store the response with its embedding vector for future cache hits. Each layer reduces the load on the next, which is why the compound savings outpace what any single technique achieves alone.
Where to start
-
Run tiktoken across one week of production logs. Find your top 10 system prompts by token count — most can be cut 30% with a single editing pass.
-
Deploy semantic caching first. It requires the least architectural change and delivers the fastest ROI. Use 0.92 as your similarity threshold with pgvector, and tie cache invalidation to your source document lifecycle — TTL alone is insufficient for data-sensitive workflows.
-
Apply LLMLingua only to retrieved context, not instructions. Compressing structured instructions introduces unpredictable quality regressions. Keep compression targeted at RAG chunks and long-form context windows where information density is naturally lower.
The 60–75% cost reduction is real, but it compounds across all four layers. Start with auditing, layer in caching, then compression. Each step funds the next.