anthropic-cookbook (1 finding) vs litellm (10 findings, $1,267/mo): what the 13x prompt-caching contrast reveals

Same engine. Two real public repos. 13x asymmetry. Here’s what mature LLM apps still miss about prompt caching — and why “we don’t manufacture findings” matters more than the savings number.

Why the contrast matters more than the dollar number

Any vendor selling you a static-analyzer audit can say “our engine is honest, it doesn’t manufacture findings to justify the price.” The proof is asymmetry. Run the same engine against a known-clean repo and a typical production repo. If both come back with similar finding counts, the engine is generating noise. If the clean repo comes back nearly empty and the production repo comes back loud, the engine is doing real work.

Here’s the side-by-side:

The lone finding in the anthropic-cookbook is a HIGH-severity system-prompt duplication across files. The cookbook isn’t running anywhere, so the savings number is $0/mo, which is exactly what an honest engine should report. If the engine had “found” a CRITICAL with a 4-figure savings claim, we’d know it was making things up.

What litellm gets wrong (and why it matters for your codebase)

litellm is the closest thing the OSS community has to a production-grade LLM-routing library. Tens of thousands of teams use it. Its codebase is mature, well-tested, well-reviewed. And it still has 10 prompt-caching opportunities the static analyzer caught in under one second of scan time.

The top 3 by impact:

1. CRITICAL: cache_control missing on static blocks (4 occurrences)

This is the #1 leak we see across customer audits. A 2K-token system prompt called 100K times/month at Sonnet rates is $600/mo uncached vs $75/mo cached — 87% off. litellm has 4 of these. None has the one-line cache_control: {"type": "ephemeral"} wrapper that turns the cost off.

- system="You are a helpful assistant. [...2K tokens of routing rules...]"
+ system=[{
+     "type": "text",
+     "text": "You are a helpful assistant. [...2K tokens of routing rules...]",
+     "cache_control": {"type": "ephemeral"}
+ }]

One line. 87% off the static portion. Per-call savings compounded across 100K+ requests/month = $525/mo per occurrence. Times 4 occurrences = $2,100/mo if every block hits typical volume.

2. HIGH: System prompt duplicated across files (also appears in anthropic-cookbook)

Even when cache_control is added, prompt-caching only matches byte-identical prefixes. If the same multi-paragraph system prompt is copy-pasted into 5 different files with subtle whitespace differences, you get 5 independent caches that each have to warm separately. The fix is mechanical: extract the prompt to a shared module, import everywhere.

# services/agent_a.py
SYSTEM = """You are a helpful routing assistant..."""

# services/agent_b.py
SYSTEM = """You are a helpful routing assistant..."""  # whitespace-different copy

# prompts/system.py (single source of truth):
SYSTEM_PROMPT = """...the canonical multi-paragraph prompt..."""

# all call sites:
from prompts.system import SYSTEM_PROMPT
client.messages.create(
    system=[{"type": "text", "text": SYSTEM_PROMPT,
             "cache_control": {"type": "ephemeral"}}],
    ...
)

3. MEDIUM: Oversized example blocks (3 occurrences)

Few-shot examples of 2,000+ chars passed in user messages without cache_control wrappers. These are exactly the high-token static blocks prompt-caching was designed for. Same fix shape as #1 above.

What this proves about static-analyzer audits

Three things, in order of importance:

1. The engine isn’t making things up. 1 finding on Anthropic’s reference. 10 on a production library. Same engine. If the engine were generating findings to justify the $39 audit, both repos would have similar counts.
2. Mature codebases still have low-hanging fruit. litellm is one of the most-reviewed LLM libraries in the OSS ecosystem. It still has 10 prompt-caching gaps that took 1 second to find. Your production codebase — almost certainly less-reviewed than litellm — almost certainly has more.
3. The fix is paste-able into a PR. Every finding ships with before/after code snippets. No “hire a consultant” handwaving. A senior engineer can ship the top 3 fixes in a single afternoon.

The savings math, conservatively

Sonnet pricing: $3/M input tokens, $0.30/M cache-read tokens (90% off after the first warm). A 4K-token static system prompt called 100K times/month:

• Uncached: 4,000 tokens × 100,000 calls = 400M input tokens = $1,200/mo
• Cached: ~$120 of cache-reads + ~$30 of amortized writes = $150/mo
• Savings per block: $1,050/mo

Most production codebases have 3-5 such blocks. That’s $3,150–$5,250/mo of recurring savings — verifiable in console.anthropic.com the very next billing cycle. The audit pays for itself 50–100x in 30 days.

The honesty proof, restated

If you remember one thing from this post: ask any vendor selling you a static-analyzer audit to run it on a known-clean reference repo and post the output. If they refuse, or if the output looks suspiciously similar to their pitch deck, the engine is making things up. If the clean output is genuinely empty (or 1 finding) and the production output is loud, the engine is doing real work.

13x asymmetry is the actual contract.