Internship: unconscious bias in conversational AI
I ran a user study to find where a digital assistant falls short for speakers of African American Vernacular English, and turned what we learned into guidelines for a more inclusive model.
Roles & responsibilities
UX research: moderated study with 12 participants
Inclusive design: guidelines for a more inclusive assistant
Tested with
Oracle Digital Assistant
Expense skill
Duration
May to Aug 2022
Impact
12 one-hour moderated sessions
Participants rated the assistant 7 out of 10
Final presentation, Aug 2022
Timeline
One summer, May to August 2022, from first question to final presentation.
The problem
A digital assistant is only as inclusive as the data it learns from. To understand every English speaker, it has to be trained on a wide range of utterances from a diverse group of users. So the question I worked on: how do we find biases in a conversational AI and design an inclusive model?
The study
A moderated user study, built to see how people really talk to the assistant.
Scope
I started with the dialects used across the US, then focused on one for this project: African American Vernacular English (AAVE).
Research statement
If the assistant accurately recognizes and responds to written utterances from the group, the model is inclusive of that group.
Sessions
One-hour moderated sessions with 12 AAVE participants, each interacting with Oracle Digital Assistant.
Scenarios
Expense tasks: uploading a receipt, creating an expense without one, modifying a previous expense, and creating reports.
What we found
Participants rated their experience with the digital assistant 7 out of 10. What shaped it most:
- People code-switch. Most change how they speak depending on the environment.
- Tone sets the tone. Responses were largely influenced by the style and tone of the assistant.
- Money makes people careful. They speak carefully when working with money, prefer to be concise, and communicate differently when speaking than when writing.
“I choose to speak in a different way when I am in a work setting vs when I am home or talking to my friends.”
“Once the DA responded create after I clicked create, I knew the conversation would not be as organic as I would have if the greeting had been more conversational.”
Usability feedback from the same sessions:
- Some instructions were too wordy.
- Users want to know how long approval takes.
- Voice input listens for too short a time.
- Payment method should offer more options.
- The assistant could not follow the context when the user moved off topic.
Challenges. More than half of the invited participants did not show up, some chose to speak a certain way with the assistant, and the Expense skill has little room for conversation.
Recommendations
Five guideline highlights for building a more inclusive assistant:
- Use more natural language that people can relate to when writing responses.
- Vary responses to the same question, for a natural feel to the conversation flow.
- Test models with a diverse range of data, with both corporate and informal words.
- Diversify data collection, data handling and processing.
- Collect and compare utterances from voice and text to see efficiency.
Next steps. Re-train the model with the utterances collected, document the usability feedback so bugs can be filed, and write up the guidelines as best practices. Going forward: a bigger study with different skills and more diverse user groups, and a metric to assess the model for inclusivity.
The final presentation
I presented the work as a final presentation in August 2022.
Download the full deck (PDF)55 slides, about 24 MB