Evaluation for all three scenarios, starting with contract review: expert-labeled ground-truth sets, scoring rubrics, side-by-side model and prompt comparisons, and a failure taxonomy.
Case study • 2021 — 2022 · Lead ML/AI Product Manager
Deeper search sessions at 75M+ scale
Search results and the Alice assistant reach 50M+ people. The work was to make each session go further, with more follow-up queries and longer sessions, and to carry that through to ad revenue. Alongside it came two new discovery surfaces in the SuperApp and a legal AI product incubated from zero.
YandexSearch & AI platform
- Mar 2021 — Dec 2022
- Search & AI platform · 75M+ MAU
- 4 teams aligned
- Legal AI incubation · 15-person team
+12%in-session search
engagement
engagement
~$20Mincremental
ad revenue
ad revenue
+18%LTV
1.5Mdiscovery MAU
in 2 months
in 2 months
Starting point
Yandex is a search and AI platform with 75M+ monthly users. Search results (SERP) and the Alice assistant reach 50M+ of them.
In-session engagement, measured as follow-up queries and session length, depended on four teams moving together: Search Ranking, NLU, Ads and Design.
My role
Lead ML/AI Product Manager on the Search & AI platform.
Aligned Search Ranking, NLU, Ads and Design around shared outcomes, launched two recommendation and discovery surfaces in the Yandex SuperApp, and incubated a legal AI product with a 15-person team.
What I did
- Aligned four teams on one outcome
Brought Search Ranking, NLU, Ads and Design around in-session engagement across SERP and Alice, measured as follow-up queries and session length.
- Carried engagement through to revenue
Tied the engagement work to ads and lifetime value: about 1.1% more ad revenue, roughly $20M incremental, and LTV up 18% through attribution and experimentation tooling.
- Won distribution for discovery
Launched two recommendation and discovery surfaces in the Yandex SuperApp, winning prioritization and distribution against 40+ internal services.
- Incubated a legal AI product
Built Lawrify 0→1 with a 15-person team, a legal AI agent for the back offices of law firms. The incubation has its own chapter below.
What changed
In-session search engagement rose 12% across SERP and Alice. Ad revenue grew about 1.1%, roughly $20M incremental, and LTV rose 18%. The two discovery surfaces reached about 1.5M MAU in two months, and Lawrify earned its first ~$10K in four months.
Search Ranking+NLU+Ads+Design→+12% engagementFollow-up queries and session length · SERP & Alice · 50M+ users
Incubated at Yandex: Lawrify
~$10K revenue in 4 months
A legal AI agent built for the back office
- 15-person team
- Legal domain
- $150 per seat
The wedge. No dataset and no idea to start with, so the work started with interviews with practicing lawyers. Law firms split into a front office, which presents the case and owns the outcome, and a back office, which prepares cases and searches data and precedents. The front office would not risk its reputation on an AI mistake. The back office already double-checks everything, so reviewing AI output fits its habits. Lawrify was built for the back office.
Three scenarios. Find similar legal cases inside and outside the organization. Find and extract data such as citations of specific laws. Review contracts for vague clauses, risks and typos.
The outcome. Back offices of large law firms and small boutiques became the core users at $150 per seat. Lawrify earned its first ~$10K in revenue in four months, and response time dropped from about 60 seconds to about 15.
~$10Krevenue in 4 months
76%F1 on expert-labeled evals
60→15sresponse time
How we measured Lawrify
Recall over precision: the agent should not miss a risky clause.
The back office reviews every flag anyway, so the agent was tuned to find more rather than miss.
Evaluated both the datasets the model produced and the outcome of the agent’s full task.
Lawyers double-checked false positives and false negatives in flagged clauses.
Human evals drove the quality and latency trade-off, cutting response time from about 60 seconds to about 15.
76%F1 on expert-labeled sets
~15sresponse time, from ~60s
Summary
At platform scale, engagement moves when the teams that shape a session move together. Four teams on one outcome lifted search engagement 12% and carried it through to about $20M in ad revenue, while a small incubation showed legal AI could earn on its own.
Building something
ambitious?
Tell me about the product and where it’s stuck, and I’ll reply by email.
Tell me about your product