RiskRAG: A Data-Driven Solution for Improved AI Model Risk Reporting

要旨

Risk reporting is essential for documenting AI models, yet only 14% of model cards mention risks, out of which 96% copy content from a small set of cards, leading to a lack of actionable insights. Existing proposals for improving model cards do not resolve these issues. To address this, we introduce RiskRAG, a Retrieval Augmented Generation risk reporting solution guided by five design requirements we identified from literature and co-design with 16 developers: identifying diverse model-specific risks, clearly presenting and prioritizing them, contextualizing for real-world uses, and offering actionable mitigation strategies. Drawing from 450K model cards and 600 real-world incidents, RiskRAG pre-populates contextualized risk reports. A preliminary study with 50 developers showed that they preferred RiskRAG over standard model cards, as it better met all the design requirements. A final evaluation with 38 developers, 40 designers, and 37 media professionals showed that RiskRAG improved the quality of their way of selecting the AI model for a given task, encouraging a more careful and deliberative decision-making.

著者
Pooja S. B.. Rao
University of Lausanne, Lausanne, VD, Switzerland
Sanja Scepanovic
Nokia Bell Labs, Cambridge, United Kingdom
Ke Zhou
Nokia Bell Labs, Cambridge, Cambridgeshire, United Kingdom
Edyta Paulina. Bogucka
Nokia Bell Labs, Cambridge, Cambridgeshire, United Kingdom
Daniele Quercia
Nokia Bell Labs, Cambridge, United Kingdom
DOI

10.1145/3706598.3713979

論文URL

https://dl.acm.org/doi/10.1145/3706598.3713979

動画

会議: CHI 2025

The ACM CHI Conference on Human Factors in Computing Systems (https://chi2025.acm.org/)

セッション: AI Ethics and Concerns

G314+G315
7 件の発表
2025-04-30 01:20:00
2025-04-30 02:50:00
日本語まとめ
読み込み中…