Risk reporting is essential for documenting AI models, yet only 14% of model cards mention risks, out of which 96% copy content from a small set of cards, leading to a lack of actionable insights. Existing proposals for improving model cards do not resolve these issues. To address this, we introduce RiskRAG, a Retrieval Augmented Generation risk reporting solution guided by five design requirements we identified from literature and co-design with 16 developers: identifying diverse model-specific risks, clearly presenting and prioritizing them, contextualizing for real-world uses, and offering actionable mitigation strategies. Drawing from 450K model cards and 600 real-world incidents, RiskRAG pre-populates contextualized risk reports. A preliminary study with 50 developers showed that they preferred RiskRAG over standard model cards, as it better met all the design requirements. A final evaluation with 38 developers, 40 designers, and 37 media professionals showed that RiskRAG improved the quality of their way of selecting the AI model for a given task, encouraging a more careful and deliberative decision-making.
https://dl.acm.org/doi/10.1145/3706598.3713979
The ACM CHI Conference on Human Factors in Computing Systems (https://chi2025.acm.org/)