Comparing Zealous and Restrained AI Recommendations in a Real-World Human-AI Collaboration Task

Abstract

When designing an AI-assisted decision-making system, there is often a tradeoff between precision and recall in the AI's recommendations. We argue that careful exploitation of this tradeoff can harness the complementary strengths in the human-AI collaboration to significantly improve team performance. We investigate a real-world video anonymization task for which recall is paramount and more costly to improve. We analyze the performance of 78 professional annotators working with a) no AI assistance, b) a high-precision "restrained" AI, and c) a high-recall "zealous" AI in over 3,466 person-hours of annotation work. In comparison, the zealous AI helps human teammates achieve significantly shorter task completion time and higher recall. In a follow-up study, we remove AI assistance for everyone and find negative training effects on annotators trained with the restrained AI. These findings and our analysis point to important implications for the design of AI assistance in recall-demanding scenarios.

Authors
Chengyuan Xu
University of California, Santa Barbara, Santa Barbara, California, United States
Kuo-Chin Lien
Appen, Sunnyvale, California, United States
Tobias Höllerer
University of California, Santa Barbara, Santa Barbara, California, United States
Paper URL

https://doi.org/10.1145/3544548.3581282

Video

Conference: CHI 2023

The ACM CHI Conference on Human Factors in Computing Systems (https://chi2023.acm.org/)

Session: Human AI Collaboration_A

Hall B
6 items in this session
2023-04-25 09:00:00
2023-04-25 10:30:00