Agentic RAG is a routing problem, not an intelligence problem
·3 min read
There is a systematization from March this year that models agentic retrieval as a sequential decision problem rather than a pipeline. Formally, a finite-horizon partially observable Markov decision process. Which is an academic way of saying: the agent is making a series of choices under uncertainty, and the choices matter as much as the destination.
I think that framing is right, and I think it points at something most teams are getting backwards.
The expensive default
Watch what actually happens when a team adopts an agent framework. Every question goes through the agent. "What are your office hours" gets a planner, three tool calls, a reflection step, and nine seconds of latency, to return something a keyword index would have answered in thirty milliseconds.
Then the team concludes agents are slow and expensive.
Agents are slow and expensive. That is fine. They are supposed to be the thing you escalate to, not the thing you start with.
Route by question shape
Almost every question falls into one of a few buckets, and you can usually classify it with something small and cheap before you commit real money:
| Shape | Example | Send it to |
|---|---|---|
| Exact lookup | order 88213 status | Keyword or SQL. No LLM. |
| Single fact | what is the refund window | Hybrid retrieval, one pass |
| Comparison | how does plan A differ from B | Retrieval, two targeted queries |
| Corpus-level | what do customers complain about most | Graph or aggregation, not top-k |
| Multi-hop or task | find the contract and check if it expired | Agent, with a step budget |
The router is the product decision. The agent is just one branch of it.
On that fourth row: Microsoft's GraphRAG reported 70% to 80% win rates over naive RAG on comprehensiveness for global questions. Real result, and worth noting it is a claim about that question class specifically. Graph indexing costs a lot to build and maintain. Route to it, do not default to it.
Bound everything
An agent with no ceiling will find a way to spend your money. Things I now put in from the start:
- A hard step limit. Five is usually plenty. If it needs more, that is a signal, not a setting to raise.
- A loop detector. Same tool, same arguments, twice in a row means stop.
- A cost ceiling per request, enforced in code, not in the prompt.
- An escalation path to a human that the agent is allowed to take early.
The last one gets skipped constantly and it is the one that saves you.
The open problem
The honest state of things: we do not have a good way to evaluate trajectories. We can score the final answer. Scoring whether the agent took a sensible route to get there is mostly vibes and manual trace reading.
That research names it directly, alongside cost-aware orchestration and stable adaptive retrieval, as unsolved. It matches what I see. Two agents produce the same correct answer, one took two steps and one took eleven, and every dashboard I have used scores them identically.
Until that is solved, read your traces. Actually read them. Twenty per week, by hand. It is tedious and it is currently the only reliable method anyone has.
Where I land
Start with hybrid retrieval and citations. Add a router when you can see the buckets in your own logs. Add an agent for the specific shape that genuinely needs iteration, with a budget and a kill switch.
Adding agents because the field moved is how you get a slower product with a bigger attack surface and a worse debugging story.