Metrics and evaluations for conversational quality
I partnered with internal quality and engineering teams to define how Oracle measures single-turn and multi-turn AI conversations, on readability, completeness and word count, and how teams write test cases and run evaluations.
Roles & responsibilities
Conversation design: metric definitions and anchor examples
Evaluation: test-case guidance
Collaboration
Quality team
Engineering
Tools
Figma
Codex
Claude
Internal environment
Impact
Published internally
The decision
Score a reply against the whole conversation, not only the latest message.
Why. A reply can be right on its own and wrong in context. Scoring across turns catches that.
The problem
The response quality guideline said what good looks like, not how to prove it. Teams judged agents by reading a few transcripts, and two reviewers could disagree on the same one. Multi-turn made it worse: a reply can be right on its own and wrong in context.
What we measure
We looked at both single-turn and multi-turn conversations. Across both, three measures matter most.
Readability
How easily a person can read and scan the response.
Completeness
Whether the response covers everything the user asked.
Word count
Whether the length fits the question: long enough to help, short enough to read.
The solution
One way to measure quality that every team uses. I owned the metric definitions, anchor examples and test-case guidance. Partner teams brought the evaluation infrastructure.
I just moved from the Austin office to London.
Welcome to London. I can help with benefits, payroll or time off.
How many vacation days do I get this year?
Full-time employees get 15 days of paid vacation a year, plus 10 company holidays. You can see your balance under Me > Time Off.
Answers with the US policy. The user said London in turn 1.
- Readability
- 5
- Completeness
- 5
- Word count
- 5
PassClear, complete, right length.
In the UK you get 25 days of paid vacation, plus 8 bank holidays. You can see your balance under Me > Time Off.
Single-turn scoring
Each response is scored on its own, with a definition, a scale and anchor examples.
Multi-turn scoring
The same measures applied across a whole conversation, so a reply that is fine alone can still be caught when it fails in context.
Test-case guide
How to write a case someone else can run: goal, context, observable criteria and failure modes, with a template.
Evaluation procedure
Human versus model grading, rater calibration, sample sizes, and reporting that compares across releases.
What I learned
Defining a metric is a conversation design problem. The hard part is not the scale, it is writing anchor examples so two reasonable people score the same transcript the same way.