Graph-Based Retrieval Augmentation With Knowledge Graphs
Graph-based retrieval handles multi-hop questions vector search alone cannot answer correctly.
Standard RAG pipelines embed documents as chunks, retrieve by similarity, and stuff the top results into a prompt. That works fine when the answer sits inside one chunk. It falls apart the moment an answer needs facts pulled from several documents at once, and that gap is why knowledge graphs are showing up inside more retrieval systems now.
What a knowledge graph adds to the retrieval layer
A knowledge graph is simple at its core: nodes stand for entities, edges stand for typed relationships between them, and together they spell out factual structure instead of leaving it implied. A flat vector index has none of that. It just knows which chunks of text sit near each other in embedding space, and near each other is not the same thing as connected.
That difference plays out in three concrete ways. A knowledge graph states relationships as facts, instead of making the retriever guess them from how close two sentences happen to sit in a document. It supports multi-hop traversal, so a retriever can follow a chain of connections across documents instead of stopping wherever the first match runs dry. And it organizes concepts into a hierarchy, so a system knows one entity is a type of another, or part of a larger whole, rather than treating everything as loose, co-located text.
KG²RAG, published at NAACL in 2025, shows the mechanism in practice. The pipeline starts with ordinary semantic retrieval to find seed chunks, runs a knowledge-graph-guided expansion step that follows fact-level relationships between those chunks, then organizes the results into coherent paragraphs using the graph structure. Tested on HotpotQA and its variants, it showed improvements over plain vector retrieval on multi-hop question answering tasks.
None of this replaces vector search, and it shouldn't try to. The graph adds a relational layer on top: seed retrieval is still semantic, and the graph decides what to pull in next. Vector search finds content that looks similar. Graph traversal finds content that's actually connected. Those are different jobs, and most real questions need both done.
How GraphRAG handles the local-to-global problem flat RAG cannot
Microsoft Research's GraphRAG paper, "From Local to Global: A Graph RAG Approach to Query-Focused Summarization," laid out what graph structure buys beyond chunk-by-chunk lookup, and the code shipped under an MIT license so teams could build on it directly.
The core idea: build communities of related entities when the graph is indexed, summarize each community at several levels of abstraction, then retrieve at whatever level matches the scope of the question. A narrow question about one entity gets entity-level detail. A broad question about patterns across an entire corpus gets a community-level summary instead of a pile of disconnected fragments. That's what makes "sensemaking" queries possible at all, the kind that ask what a corpus is about rather than what one document says.
Even graph-based systems trip over their own structure, though. HiRAG, out of a team at CUHK, names two specific failure points. First, entities that are semantically close can end up structurally far apart: a graph built straight from documents might place "big data" and "recommendation system" nowhere near each other, even though both sit clearly under "data mining." Second, there's a knowledge gap between local and global retrieval. Local search returns entity-level facts, say a company's subsidiaries or its leadership team. Global search returns a community summary, say that company's role in an industry. Without a way to reconcile the two, an LLM ends up holding two different scopes of information at once, and the answer it produces doesn't hang together.
HiRAG's fix borrows an organizing principle from how people naturally structure knowledge, building a hierarchy that bridges local and global layers instead of bolting them together as two separate jobs. Whichever paper wins this argument matters less than what it confirms: graph-based retrieval is still a contested design space, and anyone telling you there's one settled architecture hasn't read past the abstract.
Where graph retrieval outperforms flat retrieval on benchmarks, and where it does not
Vector RAG wins at single-hop, detail-oriented lookups. Full stop. If someone tells you to rip out a working vector pipeline because graphs are the future, they're solving a problem you don't have. Head-to-head comparisons across a unified set of tasks in 2025 found vector RAG holding its ground on those simple lookups, while GraphRAG pulled ahead specifically on multi-hop and global sensemaking questions.
That gap opens up right where hallucination risk runs highest. The ORAN benchmark tested vector RAG, GraphRAG, and a hybrid graph approach in 2025, and all three scored around 0.77 on faithfulness for easy questions, essentially tied. Move to medium and hard questions and the graph-based methods pull away: faithfulness came in at 0.59 versus 0.55 for vector RAG, and factual correctness landed at 0.50 for GraphRAG and 0.58 for the hybrid version, against 0.48 for vector RAG. The gap is real but modest at that level, and no one should oversell it.
Where the gap stops being modest: one knowledge-graph-language-model benchmark found vector RAG collapsing to 0% accuracy on Metrics & KPIs and Strategic Planning categories on complex multi-entity queries, while GraphRAG held steady past ten entities per query. Zero percent isn't a rounding error, it's a system that has stopped functioning. A 2025 SDK update pushed accuracy on that same benchmark from 56.2% past 90%, a substantial gain achieved through improvements to the retrieval approach itself.
On FinanceBench, presented at the GenAIK Workshop at COLING 2025, GraphRAG cut hallucinations by 6% and cut token usage by 80% against conventional RAG. So the honest read is this: keep vector RAG for lookup-style questions, because it's cheaper and it's correct there. Spend graph-level compute where the questions actually connect entities across documents. Spending it on a single-hop question is just waste, and pretending graphs win everywhere is the same mistake as pretending they win nowhere.
Why GraphRAG was too expensive to deploy in 2024, and how the cost curve shifted
The 2024 cost problem was real, and it wasn't a rounding error either. Microsoft's original hierarchical community approach could run $33,000 or more just to index a large enterprise corpus, a number that knocked most teams out of the running before anyone got to evaluate the accuracy gains. Fewer than 15% of enterprises had deployed graph-based retrieval in production as of 2025, despite benchmark after benchmark showing it worked. That gap between "it works" and "it shipped" is the whole story of last year.
Four approaches have chipped away at that bill since, each going after a different piece of it. LazyGraphRAG defers community summarization until query time instead of doing it upfront at indexing time, and gets comparable quality at a dramatically lower cost, a contrast that should make anyone who paid the 2024 price wince. HippoRAG, out of a university NLP research group, delivers multi-hop reasoning accuracy at a fraction of the cost of the full hierarchical pipeline by skipping most of the heavy indexing work. OG-RAG adds ontology grounding on top of retrieval and improves answer correctness by 40%, also without the full indexing pipeline behind it.
None of that closes the build-time gap. Constructing a knowledge graph still takes weeks to months of ontology work, while standing up a vector pipeline takes days, and cheaper inference doesn't buy back that upfront design time. So the real allocation question for a team is whether it can afford GraphRAG now." It's whether the architecture fits the corpus size and the kind of questions users are actually asking. A small corpus answering simple lookup queries has no business paying for a graph at all, cheap or not.
Building the knowledge graph itself: how LLMs changed the construction pipeline
Building a knowledge graph used to demand specialized NLP expertise, months of manual annotation, and infrastructure only a research team could justify. It wasn't something a developer picked up on a Tuesday and shipped by Friday.
LLMs changed that math. Given a handful of examples, an LLM can now pull entities and relationships out of raw text at accuracy that can rival traditional extraction pipelines that once required extensive labeled data to get there. A 2025 project built KG-FM, a knowledge graph spanning 2 million nodes, out of over 100,000 research papers, using Qwen2-72B to do the extraction and enabling accurate question answering through graph-augmented inference. That's a real demonstration that LLM-driven construction scales past toy examples, not a proof of concept stuck in a lab notebook.
Tooling caught up too. Neo4j's LLM Knowledge Graph Builder, launched in June 2024, became the fourth most popular source of user activity on AuraDB Free, and its latest version lets users pass extra prompts to steer extraction toward specific entities, themes, or sections of a document instead of extracting everything indiscriminately. A 2025 academic workshop on text-to-knowledge-graph construction, its fourth run, covered building graphs from research papers, legal filings, newswire text, and social media, with the conversation centered on scalable extraction, keeping the graph internally consistent, and catching errors before they spread downstream.
That shift, from "can this be done" to "how do we do this reliably at scale," says something about where the field actually stands. One fact hasn't moved through any of it: graph quality at construction time sets a hard ceiling on retrieval quality at inference time. A fragmented, sloppy graph produces fragmented, sloppy retrieval, no matter how clever the algorithm sitting on top of it is.
How agentic architectures use knowledge graphs as dynamic reasoning tools
Retrieval used to be a fixed pipeline: embed the query, fetch the top-k chunks, done. Agentic RAG breaks that pattern outright. The LLM now decides when to retrieve, what to retrieve, and whether what came back is good enough, looping back for another pass if it isn't.
Knowledge graphs fit that loop naturally. Graph traversal gets exposed to the agent as just another tool it can call, letting it run structured multi-hop reasoning as one step inside a longer chain, rather than firing off a single retrieval and hoping it lands. Work on GNN-based retrieval approaches uses graph neural networks to handle multi-hop reasoning directly and shows accuracy gains on multi-hop benchmarks.
A NeurIPS 2025 spotlight paper on GraphFlow tackles a specific pain point head-on: knowledge-graph retrieval methods often struggle to pull accurate, diverse information out of text-heavy graphs once the query gets genuinely complex. GraphFlow trains a retrieval policy and a flow estimator together using a flow-matching objective, and beats strong knowledge-graph RAG baselines, including GPT-4o, by 10% on average in hit rate and recall on the STaRK benchmark, while generalizing well to graphs it never trained on.
Techniques like ReAct and function calling let an agent interleave reasoning with tool calls, steering graph traversal step by step as new gaps in its knowledge turn up mid-task. Sub-question decomposition methods like Self-Ask and Search-o1 map onto this cleanly, since a multi-hop question naturally splits into a sequence of smaller lookups that trace a path through the graph. MemGraphRAG's memory-aware retrieval shows a multi-agent system, using shared memory, retrieving in a way that reflects the graph's overall structure instead of grabbing isolated fragments one at a time.
A survey posted to arXiv in October 2025 names dynamic knowledge memory for agentic systems as a key open research direction. That framing matters: knowledge graphs are looking more like memory infrastructure for agents, and less like an index you search once and forget. GraphRAG-Bench exists now, built specifically to check end-to-end output, retrieval correctness, and graph-level properties all at once, because the field has gotten complicated enough to need a benchmark that does all three at the same time.
What the retrieval pipeline needs from web data to keep knowledge graphs current
A knowledge graph is only as good as what it was built from. Feed it outdated documents, and it encodes outdated relationships as if they were still true. That's a quieter failure than a bad retrieval call, but it's no less damaging for being quiet, and arguably worse, since nobody notices until the wrong answer ships.
The same hallucination and incoherence problems graph retrieval was built to fix come right back the moment the source corpus goes stale. Three things break in practice. Entity attributes shift, a company changes leadership, a product spec updates, a regulatory status flips, so the edges that were accurate on the day the graph was built now point at facts that no longer hold. New entities and relationships show up that were never in the graph to begin with, and an agent can't reason over something the graph never captured in the first place. Documents that seeded the graph get revised or pulled entirely, and the graph just carries those errors forward as if nothing changed.
Current web content is a necessity for an agentic system running RAG over a knowledge graph. It's a basic input requirement, and the construction pipeline needs a steady, scheduled feed of fresh documents to stay useful at all. The same structured-extraction approach that builds the graph in the first place, pulling data into a defined schema, can serve the retrieval layer directly: when a query needs something that happened after the graph's last update, a live, schema-grounded web fetch fills that specific gap without forcing a full re-index of the whole thing.
Treat the knowledge graph as long-term relational memory. Pair it with a live retrieval layer that keeps that memory honest, because memory that never gets checked against the present just calcifies into confident wrongness.
