Every task the agent executed was scored against benchmarks built with 70 of the bureau’s subject-matter specialists.
Case study • 2024 — now · Data Scientist / AI Product Lead
From a call center to an AI agent businesses trust
Businesses across Kazakhstan used to phone the statistics bureau’s call center with questions, for example how to fill in reporting forms. AI Stat now answers them directly, and it only ships what it can answer without hallucinating.
AI StatBureau of National Statistics of Kazakhstan
- Dec 2024 — now
- Center for Behavioural Economics & Big Data Analysis
- Production LLM agent
- Launch bar set with 70 SMEs
~10Kmonthly users,
including businesses
including businesses
300–500requests
per day
per day
~93–95%task success against
SME benchmarks
SME benchmarks
Starting point
Businesses are the bureau’s primary users. With questions, for example how to fill in reporting forms, they called the bureau’s call center and asked an operator.
Every answer meant the same hidden job: go to the data storage, extract the right data, understand what it describes, and map it to the question the business actually asked.
My role
Data Scientist and AI Product Lead at the bureau’s Center for Behavioural Economics & Big Data Analysis, an experimental division that works as the bureau’s AI lab.
Owned the KPIs, the scenarios, the knowledge base and retrieval design, and the launch bar that decided when the agent could roll out. Worked with the engineering team and with 70 of the bureau’s subject-matter specialists.
What I did
- Turned the operator’s job into the agent’s scenario
Defined the core scenario: find the data, extract it, understand what it is about, and map it to the user’s question. Prioritized the top three user journeys, starting with how to fill in the reporting forms.
- Set the launch bar with 70 SMEs
Defined the KPIs and the launch bar together with 70 subject-matter specialists: explicit no-go rules, and a task success rate of about 93–95% against their benchmarks.
- Designed knowledge and retrieval
Designed the knowledge base and the RAG retrieval layer: hybrid search across meaning and exact wording, with a second, agentic search step when the first pass is not enough.
- Built explicit feedback into the product
Every answer can be liked or disliked, with a comment on why it missed or which piece of data was missing, so failures are visible and specific.
- Rolled out in stages
Piloted the agent during the Agricultural Census on roughly 8,500 real questions before the full rollout.
How we measured it
If it doesn’t work, it doesn’t roll out. If it hallucinates, it doesn’t roll out.
A single invented answer costs the user’s trust, and the business goes back to calling the operator. That makes hallucination a release blocker.
The launch bar named what blocks a release outright, before any average score was allowed to count.
Invested heavily in recall: finding the right data is the precondition for every correct answer.
Likes, dislikes and written comments show exactly where the agent missed after launch.
~93–95%task success against SME benchmarks
~8.5Kcensus questions in the staged pilot
What changed
AI Stat went from pilot to full production and moved businesses from call-center support to self-serve answers. It serves around 10K monthly users and handles 300–500 requests per day. The bureau officially launched it on stat.gov.kz in May 2026.
Knowledge base→RAG retrieval→Agent task→SME benchmark→ProductionEvery task scored against expert answers · ~8.5K census questions
Summary
In a public institution, trust is the product. The release bar was set on hallucinations before features, because one wrong answer sends a business back to the phone line.
Building something
ambitious?
Tell me about the product and where it’s stuck, and I’ll reply by email.
Tell me about your product