Caching Agent Tool Results to Control Latency and Cost

Caching Agent Tool Results to Control Latency and Cost

Tool-result caching as a distinct problem from prompt caching and semantic caching

Enterprise LLM spending grew from $3.5 billion to $8.4 billion between late 2024 and mid-2025, and 72% of organizations plan to spend even more in 2026 Reduce LLM Cost and Latency: A Comprehensive Guide for 2026. Most teams still don't have a real plan for controlling that spend, or the latency that comes with it Reduce LLM Cost and Latency: A Comprehensive Guide for 2026. Part of the reason is that "caching" gets treated as one thing when it's actually three, and each one solves a different problem.

Prompt caching, or prefix caching, stores the KV tensors for a repeated chunk of prompt text at the provider level. Semantic caching stores full LLM responses, keyed to how similar a new query is to one seen before. Tool-result caching caches the outputs of external tool calls (APIs, web fetches, database queries) between agent steps.

Mixing these up isn't just a naming problem. Apply prefix-caching logic to a tool result and you'll cache something that goes stale in seconds. Apply semantic-caching logic to a tool result and the question looked familiar even though the world had changed underneath it, so you'll serve a confident, wrong answer. Get the layer wrong and the invalidation logic is wrong, the expiry window is wrong, and the savings you thought you banked quietly evaporate. The rest of this piece treats each layer on its own terms, then builds up to where tool-result caching actually sits in an agent's execution loop.

What prompt/prefix caching covers (and where it stops)

Prefix caching works by having the provider store the computed KV tensors for a prompt prefix it's already seen. The next time that same prefix appears, only the new part gets processed fresh; the static portion is billed at a steep discount, and the output is byte-identical to what a full, uncached run would have produced. No quality is traded away for the savings, which is part of why it caught on so fast.

The rollout came quickly once providers committed to it. Anthropic put prompt caching into public beta on August 14, 2024, and moved it to general availability on December 17, 2024, pricing cache reads at $0.30 per million tokens against $3.00 per million for a fresh read Prompt Caching Infrastructure. OpenAI followed with automatic caching on October 1, 2024. Neither company asked developers to opt into anything complicated; the discount just applies when the system recognizes a repeated prefix.

The headline numbers are strong. A PwC evaluation, published as the "Don't Break the Cache" study, found that prompt caching reduces API costs by 41 to 80% and improves time-to-first-token by 13 to 31% across providers on multi-turn agentic benchmarks Reduce LLM Cost and Latency: A Comprehensive Guide for 2026 Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks Prompt Caching Infrastructure. The cached content needs to actually be stable for those figures to hold. A separate paper on Anthropic's cache architecture, referred to as CAPC, found a two-tier structure with a threshold near 3,500 tokens Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching https://github.com/0h-n0/0h-n0.github.io/pull/1942. Below that threshold, hit rate plateaus around 0.83 rather than the 1.0 the literature tends to assume, so the savings on shorter prefixes are smaller than the idealized numbers suggest Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching.

Prefix caching is built for the parts of a prompt that don't change, system prompts, tool schemas, instructions. It was never built to handle a tool's output, which by definition can change from one call to the next. That's a different job, and it needs a different mechanism. Provider rollout timeline (from the PwC arXiv paper and Maxim guide) is discussed in the section on what prompt/prefix caching actually covers (and where it stops).

Semantic caching's fit and its limits with tool outputs

Semantic caching works off vector embeddings. A new query gets compared against past queries, and if it's close enough in meaning, the system serves the old cached response instead of hitting the model again Reduce LLM Cost and Latency: A Comprehensive Guide for 2026 Prompt Caching Infrastructure. This matters more than it might sound: roughly 31% of enterprise LLM queries are semantically identical to something already asked Reduce LLM Cost and Latency: A Comprehensive Guide for 2026 Prompt Caching Infrastructure. When it hits, the payoff is real. Response times can drop from roughly 850ms to roughly 120ms (a 7x speedup), with 40–70% cost reductions on workloads with repetitive queries Reduce LLM Cost and Latency: A Comprehensive Guide for 2026.

None of that helps with tool outputs, though, and the reason is structural rather than a tuning problem. Semantic similarity has no sense of time. A stale embedding scores exactly as well as a fresh one, so a cache built this way can't tell "weather yesterday" from "weather right now" if the two questions are phrased the same way Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching. Getting the similarity threshold right is already a balancing act on its own: set it too low and the system serves wrong answers with total confidence; set it too high and the hit rate collapses, taking the savings with it.

Tool outputs make the problem worse because they're not really "answers" in the way a chat response is. They come from executing external code or calling external APIs, and they carry environmental state that no embedding captures. No embedding captures that. Caching a tool result at the semantic layer quietly conflates "this looks like the same question" with "this is still the same answer," and those are not the same claim. The tool-result layer needs invalidation logic tied to the real world, the actual freshness of the actual source, not to how similar two queries sound Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching.

The tool-result caching layer in the agent loop

Agents spend a lot of their runtime just waiting. Web fetches, database queries, search APIs, file reads, all of it takes time, and the result of a call like get_stock_price("AAPL") or fetch_page("https://...") is valid only for a specific window. Tool-result caching sits right between the agent's reasoning loop and the tools it reaches for. It intercepts an outbound call and, if a recent call was structurally or semantically equivalent, hands back the stored result instead of making the call again.

This matters more than it sounds like it should, because agents repeat themselves. Within a single trajectory, an agent may call the same tool several times, sometimes with inputs that are nearly, but not exactly, identical. Tool-level caching catches those redundant hops and skips them, and it does this without touching LLM inference at all. It's a separate layer, doing separate work.

Semantic caching differs from this in a specific way. Semantic caching matches a query to a cached LLM response. Tool-result caching matches a tool invocation to a cached execution output. Layer it out and the architecture looks like four distinct tiers: exact-match caching at the response layer, semantic caching at the intent layer, tool-result caching at the execution layer, and RAG chunk caching at the knowledge layer https://github.com/HKUDS/LightRAG/issues/2355. Tool-result caching is the one this piece is built around. There's no single headline number to lead with here; the numbers that matter come next, in TTL choices and invalidation rules.

TTL strategy: matching expiry windows to how fast different tool outputs change

Time-to-live, or TTL, is the simplest way to invalidate a cache: assign an expiration time to each entry and let it expire on schedule. It's easy to build. The tradeoff is that you either evict too early and lose savings, or you keep serving something stale until the clock finally runs out.

The fix is dropping the idea of a default TTL entirely. It's dropping the idea of a default entirely. TTL should track how fast the underlying source actually changes, and different tool categories change at wildly different speeds. Stock prices expire in seconds. Weather data holds for minutes to hours. Scraped web pages might stay valid for hours or days, depending on how often the source itself gets updated. Static reference material, company registration records, historical filings, can stay valid for days or even weeks. Treating all four the same way is where naive caching setups go wrong https://github.com/HKUDS/LightRAG/issues/2355.

A workable heuristic: instrument tool calls to read response headers or source metadata wherever they're available, and let observed change frequency set the TTL instead of guessing at a number. Web scraping is a good place to see why this matters concretely. Freshness here is measured in days, not months. A RAG system built on a week-old government policy document will hand out wrong answers with total confidence, and a job-matching tool pulling stale listings misrepresents what's actually available in the market right now.

TTL gets you most of the way there, but it's a starting point, not the whole plan. It handles predictable decay. It does nothing for the cases where a source changes without warning.

Invalidation beyond TTL: event-driven and fingerprint-based approaches

Event-driven invalidation flips the model: instead of waiting for a clock to run out, the system actively removes or updates a cache entry the moment the underlying data changes. Doing this right means tracking dependencies between what's cached and where it came from, which is correct in principle and genuinely fiddly to build. If a document in a knowledge base gets updated, any tool result that depended on the old version needs to be thrown out immediately, not left to expire on its own schedule.

A nastier problem produces all of this: the silent edit. Plenty of web sources change their content without touching a timestamp anywhere. TTL won't catch it, because nothing about the expiry clock knows the content moved. Semantic similarity won't catch it either, because the new page probably still reads as "the same page" to an embedding model.

A paper called FreshCache, from July 2026, tackles this directly with fingerprinting. The idea: cache the tool result alongside a SHA-256 hash of the underlying source text. On the next cache hit, run a lightweight check, comparing the stored hash against a fresh hash pulled from the live source. A mismatch triggers invalidation even when nothing about the timestamp looks off. FreshCache frames the whole question as a risk-constrained decision rather than a binary hit-or-miss, gating cache reuse with a staleness-probability model split across three tiers, answers, URL lists, and page content, each with its own tolerance for how much staleness is acceptable. Run that way, the approach reportedly saves around 97% of search calls while keeping the stale-error rate down near 0.1%.

Graph-based dependency tracking models tool-result dependencies as a graph, so invalidating one node automatically propagates out to everything downstream that relied on it. Put the pieces together and a sane build order looks like this: start with TTL for sources that decay predictably, add fingerprint checks for scraped content where timestamps can't be trusted, and layer in event-driven invalidation wherever change events from an internal system are actually observable.

Cost-latency tradeoffs of caching tool results

Creating a cache entry isn't free. The first request that populates it carries overhead, and that overhead only pays off if later requests actually hit the cache instead of missing it. For tool calls that return something different every time, or that are unlikely to be repeated at all, caching adds a step without adding any benefit. The real question before turning caching on for a given tool type is simple: what's the realistic re-use rate for calls like this one?

Some categories are obvious wins. Web scraping and page fetches tend to get requested repeatedly, since the same URL is often useful to more than one user or more than one step in an agent's trajectory. Search results for common queries fall into the same bucket. If 31% of enterprise LLM queries repeat something asked before, the tool calls feeding those queries are repeating just as often Reduce LLM Cost and Latency: A Comprehensive Guide for 2026 Prompt Caching Infrastructure. Brand and company intelligence lookups tend to get pulled repeatedly across different agents and different users looking at the same target. Reference data, regulatory filings, product catalogs, documentation, holds still long enough that caching it is close to free money.

Other categories aren't worth the effort. Highly personalized calls, where the input is different for every user, rarely repeat closely enough to hit. Sources that change faster than any TTL can reasonably track, live auction prices being the clean example, will just serve stale data no matter how tight the expiry window gets. And tool calls made once inside a single agent session, with no reason to expect a repeat, are pure overhead if cached.

There's also a subtler failure mode tied back to prompt caching. The PwC study found that placing dynamic tool results inside the cached prompt prefix can actually make things slower, because it triggers cache writes for content that's never going to be reused, adding work without adding any payoff. The fix is architectural: keep tool results out of the stable prefix entirely. The decision to cache or not is a per-tool-type judgment call. It's a per-tool-type judgment call, and a single agent might cache some categories aggressively while skipping the cache entirely for others.

Agentic Plan Caching as an emerging complement to tool-result caching

A paper published at NeurIPS 2025 by Zhang, Wornow, and Olukotun out of Stanford introduces something called Agentic Plan Caching, or APC, and it attacks a different part of the problem entirely. Instead of caching what a tool returns, APC extracts structured plan templates from agent runs that already finished, and reuses those templates the next time a semantically similar task comes up.

Across five different agent workloads and two Plan-Act architectures, the results are notable: agent serving costs down 50.31%, latency down 27.28% on average, while the system still held onto 96.61% of optimal application performance Reduce LLM Cost and Latency: A Comprehensive Guide for 2026 Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents | OpenReview Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents Prompt Caching Infrastructure. The overhead of running the plan cache itself came out to just 1.04% of total serving cost Reduce LLM Cost and Latency: A Comprehensive Guide for 2026 Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents | OpenReview Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents Prompt Caching Infrastructure. Those are strong numbers, but APC doesn't remove the need for tool-result caching.

Stacked together, the two layers cover different ground and don't compete with each other: APC handles planning and tool-result caching handles execution. APC is still research-stage work, while tool-result caching and prompt caching are both already running in production systems.

How web scraping as a tool

Web scraping earns its own look here because it sits at the rough edge of everything covered above, and staleness, cost, and detection risk all appear together in the same call. A scraping tool that re-fetches a page every time an agent asks about it burns tokens, burns time, and burns goodwill with whatever server is on the other end. A scraping tool with a caching layer built in lets an agent re-extract information from a page it already pulled, without going back to the source, which cuts both the token bill and the number of requests hitting someone else's infrastructure.

The freshness math here is unforgiving in a way that other tool categories aren't. Days matter, not months. A retrieval system grounded in a week-old policy document gives out wrong information with the same confidence as one grounded in something current, and a job board pulling listings that already closed is actively misleading whoever's using it. Get that combination right, TTL tuned to the source's real update cadence, plus a hash check for the silent edits, and web scraping becomes one of the clearest wins in the entire tool-result caching layer.