Discussion: Cache economics

[hugosenari]

#4 , has an emerging off-topic about cache economics

Load cost can be reduced with proper CACHE: +30% to cache - 90% when hit (claude numbers)
(@hugosenari)

Cached tokens are most of what I'm trying to save- they look cheap on paper but always turn out to be 90-95% of cost in breakdowns.
(@capocasa )

Would be cool to have a real data that confirms your observation about cache cost.
But for WRITE+30% HIT-90%¹ take:

CACHED_COST    = totalCacheWRITECost + totalCacheHITCost
CACHELESS_COST = totalCacheWRITECost - 23.05% + totalCacheHITCost * 10

CACHED_COST < CACHELESS_COST: - (30~70%) cache is OK
CACHED_COST = CACHELESS_COST: - (01~29%) cache is NOK
CACHED_COST > CACHELESS_COST: + (01~99%) failed on 3code mission

¹ numbers from cited source, they changes over time, change between providers.

"proper cache" link: https://hidekazu-konishi.com/entry/anthropic_claude_api_prompt_caching_and_token_efficiency.html

Mentions the following cache mistakes:

• "A breakpoint that never gets a cache hit"
• "Caching a prefix that changes every request"
• "TTL expiry between requests"
• "Falling outside the 20-block lookback"
• "Assuming provider parity" (ie: Claude Cache in AWS != Claude Cache in Anthropic)
• "Estimating tokens with the wrong tokenizer" (author suggest using API to calc)
• Switching models or mutating tools mid-session

I'm not saying 3code have all those problems, but is valid to check.