Towards Fairness in Practice: A Practitioner-Oriented Rubric for Evaluating Fair ML Toolkits


In order to support fairness-forward thinking by machine learning (ML) practitioners, fairness researchers have created toolkits that aim to transform state-of-the-art research contributions into easily-accessible APIs. Despite these efforts, recent research indicates a disconnect between the needs of practitioners and the tools offered by fairness research. By engaging 20 ML practitioners in a simulated scenario in which they utilize fairness toolkits to make critical decisions, this work aims to utilize practitioner feedback to inform recommendations for the design and creation of fair ML toolkits. Through the use of survey and interview data, our results indicate that though fair ML toolkits are incredibly impactful on users’ decision-making, there is much to be desired in the design and demonstration of fairness results. To support the future development and evaluation of toolkits, this work offers a rubric that can be used to identify critical components of Fair ML toolkits.

Brianna Richardson
Spotify, New York, New York, United States
Jean Garcia-Gathright
Spotify, Boston, Massachusetts, United States
Samuel F. Way
Spotify, New York, New York, United States
Jennifer Thom
Spotify, Boston, Massachusetts, United States
Henriette Cramer
Spotify, San Francisco, California, United States




会議: CHI 2021

The ACM CHI Conference on Human Factors in Computing Systems (

セッション: Human, ML & AI

[A] Paper Room 14, 2021-05-10 17:00:00~2021-05-10 19:00:00 / [B] Paper Room 14, 2021-05-11 01:00:00~2021-05-11 03:00:00 / [C] Paper Room 14, 2021-05-11 09:00:00~2021-05-11 11:00:00
Paper Room 14
13 件の発表
2021-05-10 17:00:00
2021-05-10 19:00:00