A Qualitative Study on How Usable Security and HCI Researchers Judge the Size and Importance of Odds Ratio and Cohen's d Effect Sizes

要旨

Researchers often place a strong focus on statistical significance when reporting the results of statistical tests. However, effect sizes are reported less frequently, and interpretation in the context of the study and the research field is even rarer. These interpretations of effect sizes are, however, necessary to understand the practical importance of a result for the community. To explore how Usable Security & Privacy (USP) and HCI researchers interpret effect sizes and make judgments on practical importance, we conducted survey and interview studies with a total of 63 researchers at CHI and SOUPS 2023. Our studies focused on Cohen's d and odds ratios in two USP and one HCI scenario. We analyzed which artifacts researchers consider when judging effect size, and found misconceptions and variation between the participants, highlighting how difficult judging statistics can be. Based on our findings, we make concrete recommendations for improved reporting practices around effect sizes.

受賞
Best Paper
著者
Anna-Marie Ortloff
University of Bonn, Bonn, Germany
Julia Angelika Grohs
University of Bonn, Bonn, Germany
Simon Lenau
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
Matthew Smith
University of Bonn, Bonn, Germany
DOI

10.1145/3706598.3714022

論文URL

https://dl.acm.org/doi/10.1145/3706598.3714022

動画

会議: CHI 2025

The ACM CHI Conference on Human Factors in Computing Systems (https://chi2025.acm.org/)

セッション: HCI Methods

G416+G417
7 件の発表
2025-04-28 20:10:00
2025-04-28 21:40:00
日本語まとめ
読み込み中…