itssujeeth Distinguished Architect / Technology Director · 2025 Prototype
Analytics Agent for Narrative Business Data
A prototype conversational analytics agent that lets leaders ask natural-language questions of millions of short free-text field notes, with every number computed by tools and every answer linked to its evidence.
An enterprise wanted leaders to ask questions like which competitors are showing up most often when customers talk about pricing, across millions of short notes written by field staff after customer conversations. Retrieving a handful of similar comments cannot establish a count or a trend across all records, so a standard RAG design would have produced confident answers with no basis.
I designed and built the prototype myself: the agent, its tools, query validation, and the evaluation approach. Aggregate questions use structured extraction into a knowledge graph and graph queries; open-ended exploration uses similarity retrieval. Known business concepts resolve through deterministic taxonomy and alias matching first, and a language model handles only what the taxonomy cannot. The agent orchestrates and explains, tools compute every number, model-written queries run only after validation and are kept read-only and bounded, and every answer links back to the conversations behind it.
Rehearsal complete on a 10,000-record sample; the full-history proof of concept on more than 20 million records was about to start on the same code at my departure in September 2026, and the agent was not yet in business users' hands. Exit criteria were written down: at least 90% on the benchmark with p95 latency under 10 seconds, at least 85% sentiment agreement with hand labels, sign-off from four business participants, and cost documented against the incumbent paid tool. Built as a Google ADK agent with 13 tools and a fail-closed validator for model-written graph queries. A tiered benchmark evaluates correctness, consistency, latency, and cost, and ranks configurations by cost per correct answer. The benchmark also exposed where structured retrieval was weak on open-ended questions, which is why a semantic retrieval path was added. I do not claim a production deployment or a realized business outcome for this work.
The conversational layer was familiar territory and so was the graph. What was different was the raw material. A document gives you knowledge to organize. These notes required deciding what counted as knowledge in the first place. A generic named-entity model will tag a competitor as an organization and a price complaint as a monetary value, and neither of those is the fact the graph needs. The work was connecting the language people use in the field to the business concepts the enterprise already had a taxonomy for.
The architectural split that made the agent trustworthy was separating aggregate reporting from semantic exploration. Counts, trends, and co-occurrence come from structured extraction and graph queries over all records. Similarity retrieval supports open-ended questions where the user is exploring rather than measuring. The agent never computes a number itself. It chooses tools, explains, and shows the evidence.
The evaluation was built as a tiered benchmark rather than a single accuracy score, because the question that actually decides a configuration is cost per correct answer under the latency the user will tolerate. That benchmark is also what revealed the gap in structured retrieval on open-ended questions and justified adding the semantic path.
I wrote up the extraction side of this problem, with the business domain, taxonomy, and examples changed to synthetic ones, in Before the Agent Could Answer Anything, It Needed Something Worth Knowing. Client and employer details are confidential.