Metrics and Methods

会議の名前
CHI 2023
Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022
要旨

In recent years, various initiatives from within and outside the HCI field have encouraged researchers to improve research ethics, openness, and transparency in their empirical research. We quantify how the CHI literature might have changed in these three aspects by analyzing samples of 118 CHI 2017 and 127 CHI 2022 papers---randomly drawn and stratified across conference sessions. We operationalized research ethics, openness, and transparency into 45 criteria and manually annotated the sampled papers. The results show that the CHI 2022 sample was better in 18 criteria, but in the rest of the criteria, it has no improvement. The most noticeable improvements were related to research transparency (10 out of 17 criteria). We also explored the possibility of assisting the verification process by developing a proof-of-concept screening system. We tested this tool with eight criteria. Six of them achieved high accuracy and F1 score. We discuss the implications for future research practices and education. This paper and all supplementary materials are freely available at https://doi.org/10.17605/osf.io/n25d6.

受賞
Best Paper
著者
Kavous Salehzadeh Niksirat
University of Lausanne, Lausanne, VD, Switzerland
Lahari Goswami
University of Lausanne, Lausanne, Switzerland
Pooja S. B.. Rao
University of Lausanne, Lausanne, VD, Switzerland
James Tyler
University of Lausanne, Lausanne, Vaud, Switzerland
Alessandro Silacci
University of Lausanne, Lausanne, Vaud, Switzerland
Sadiq Aliyu
University of Lausanne , Lausanne, Vaud , Switzerland
Annika Aebli
Department of Information Systems, Lausanne, Switzerland
Chat Wacharamanotham
Swansea University, Swansea, United Kingdom
Mauro Cherubini
University of Lausanne (UNIL), Lausanne, VD, Switzerland
論文URL

https://doi.org/10.1145/3544548.3580848

動画
Complex Daily Activities, Country-Level Diversity, and Smartphone Sensing: A Study in Denmark, Italy, Mongolia, Paraguay, and UK
要旨

Smartphones enable understanding human behavior with activity recognition to support people’s daily lives. Prior studies focused on using inertial sensors to detect simple activities (sitting, walking, running, etc.) and were mostly conducted in homogeneous populations within a country. However, people are more sedentary in the post-pandemic world with the prevalence of remote/hybrid work/study settings, making detecting simple activities less meaningful for context-aware applications. Hence, the understanding of (i) how multimodal smartphone sensors and machine learning models could be used to detect complex daily activities that can better inform about people’s daily lives, and (ii) how models generalize to unseen countries, is limited. We analyzed in-the-wild smartphone data and ~216K self-reports from 637 college students in five countries (Italy, Mongolia, UK, Denmark, Paraguay). Then, we defined a 12-class complex daily activity recognition task and evaluated the performance with different approaches. We found that even though the generic multi-country approach provided an AUROC of 0.70, the country-specific approach performed better with AUROC scores in [0.79-0.89]. We believe that research along the lines of diversity awareness is fundamental for advancing human behavior understanding through smartphones and machine learning, for more real-world utility across countries.

著者
Karim Assi
École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
Lakmal Meegahapola
Idiap Research Institute, Martigny, Switzerland
William Droz
Idiap Research Institute, Martigny, Switzerland
Peter Kun
IT University of Copenhagen, Copenhagen S, Denmark
Amalia de Götzen
Aalborg University , Copenhagen, Denmark
Miriam Bidoglia
London School of Economics and Political Science (LSE), London, United Kingdom
Sally Stares
City, University of London, London, United Kingdom
George Gaskell
London School of Economics and Political Science (LSE), London, United Kingdom
Altangerel Chagnaa
National University of Mongolia, Ulaanbaatar, Mongolia
Amarsanaa Ganbold
National University of Mongolia, Ulaanbaatar, Mongolia
Tsolmon Zundui
National University of Mongolia, Ulaanbaatar, Mongolia
Carlo Caprini
U-Hopper, Trento, Italy
Daniele Miorandi
U-Hopper srl, Trento, Italy
José Luis Zarza Aguilera
Universidad Católica "Nuestra Señora de la Asunción", Asunción, Paraguay
Alethia Hume
Universidad Católica "Nuestra Señora de la Asunción", Asunción, Paraguay
Luca Cernuzzi
Catholic University of Asuncion, Asuncion, Paraguay
Ivano .. Bison
University of Trento, Trento, Italy
Marcelo Dario. Rodas Britez
University of Trento, Trento, Trento, Italy
Matteo Busso
University of Trento, Trento, Italy
Ronald Chenu-Abente
University of Trento, Trento, Italy
Fausto Giunchiglia
University of Trento, Trento, Italy
Daniel Gatica-Perez
Idiap-EPFL, Lausanne, Switzerland
論文URL

https://doi.org/10.1145/3544548.3581190

動画
A Field Test of Bandit Algorithms for Recommendations: Understanding the Validity of Assumptions on Human Preferences in Multi-armed Bandits
要旨

Personalized recommender systems suffuse modern life, shaping what media we read and what products we consume. Algorithms powering such systems tend to consist of supervised-learning-based heuristics, such as latent factor models with a variety of heuristically chosen prediction targets. Meanwhile, theoretical treatments of recommendation frequently address the decision-theoretic nature of the problem, including the need to balance exploration and exploitation, via the multi-armed bandits (MABs) framework. However, MAB-based approaches rely heavily on assumptions about human preferences. These preference assumptions are seldom tested using human subject studies, partly due to the lack of publicly available toolkits to conduct such studies. In this work, we conduct a study with crowdworkers in a comics recommendation MABs setting. Each arm represents a comic category, and users provide feedback after each recommendation. We check the validity of core MABs assumptions—that human preferences (reward distributions) are fixed over time—and find that they do not hold. This finding suggests that any MAB algorithm used for recommender systems should account for human preference dynamics. While answering these questions, we provide a flexible experimental framework for understanding human preference dynamics and testing MABs algorithms with human users. The code for our experimental framework and the collected data can be found at https://github.com/HumainLab/human-bandit-evaluation.

著者
Liu Leqi
Carnegie Mellon University, Pittsburgh, Pennsylvania, United States
Giulio Zhou
Carnegie Mellon University, Pittsburgh, Pennsylvania, United States
Fatma Kilinc-Karzan
Carnegie Mellon University, Pittsburgh, Pennsylvania, United States
Zachary Lipton
Carnegie Mellon University, Pittsburgh, Pennsylvania, United States
Alan Montgomery
Carnegie Mellon University, Pittsburgh, Pennsylvania, United States
論文URL

https://doi.org/10.1145/3544548.3580670

Impact of Privacy Protection Methods of Lifelogs on Remembered Memories
要旨

Lifelogging is traditionally used for memory augmentation. However, recent research shows that users' trust in the completeness and accuracy of lifelogs might skew their memories. Privacy-protection alterations such as body blurring and content deletion are commonly applied to photos to circumvent capturing sensitive information. However, their impact on how users remember memories remain unclear. To this end, we conduct a white-hat memory attack and report on an iterative experiment (N=21) to compare the impact of viewing 1) unaltered lifelogs, 2) blurred lifelogs, and 3) a subset of the lifelogs after deleting private ones, on confidently remembering memories. Findings indicate that all the privacy methods impact memories' quality similarly and that users tend to change their answers in recognition more than recall scenarios. Results also show that users have high confidence in their remembered content across all privacy methods. Our work raises awareness about the mindful designing of technological interventions.

著者
Passant ElAgroudy
German Research Centre for Artificial Intelligence (DFKI), Kaiserslautern, Germany
Mohamed Khamis
University of Glasgow, Glasgow, United Kingdom
Florian Mathis
University of Glasgow, Glasgow, United Kingdom
Diana Irmscher
LMU Munich, Munich, Germany
Ekta Sood
University of Stuttgart, Stuttgart , Germany
Andreas Bulling
University of Stuttgart, Stuttgart, Germany
Albrecht Schmidt
LMU Munich, Munich, Germany
論文URL

https://doi.org/10.1145/3544548.3581565

動画
Home Is Where the Smart Is: Development and Validation of the Cybersecurity Self-Efficacy in Smart Homes (CySESH) Scale
要旨

The ubiquity of devices connected to the internet raises concerns about the security and privacy of smart homes. The effectiveness of interventions to support secure user behaviors is limited by a lack of validated instruments to measure relevant psychological constructs, such as self-efficacy - the belief that one is able to perform certain behaviors. We developed and validated the Cybersecurity Self-Efficacy in Smart Homes (CySESH) scale, a 12-item unidimensional measure of domain-specific self-efficacy beliefs, across five studies (N=1247). Three pilot studies generated and refined an item pool. We report evidence from one initial and one major, preregistered validation study for (1) excellent reliability (𝛼=0.90), (2) convergent validity with self-efficacy in information security (𝑟SEIS=0.64, p<.001), and (3) discriminant validity with outcome expectations (𝑟OE=0.26, p<.001), self-esteem (𝑟RSE=0.17, p<.001), and optimism (𝑟LOT−R=0.18, p<.001). We discuss CySESH's potential to advance future HCI research on cybersecurity, practitioner user assessments, and implications for consumer protection policy.

著者
Nele Borgert
Ruhr University Bochum, Bochum, Germany
Oliver D.. Reithmaier
Leibniz University Hannover, Hannover, Germany
Luisa Jansen
Ruhr University Bochum, Bochum, Germany
Larina Hillemann
Ruhr University Bochum, Bochum, Germany
Ian Hussey
Ruhr University Bochum, Bochum, Germany
Malte Elson
Ruhr University Bochum, Bochum, Germany
論文URL

https://doi.org/10.1145/3544548.3580860

動画
Literature Reviews in HCI: A Review of Reviews
要旨

This paper analyses Human-Computer Interaction (HCI) literature reviews to provide a clear conceptual basis for authors, reviewers, and readers. HCI is multidisciplinary and various types of literature reviews exist, from systematic to critical reviews in the style of essays. Yet, there is insufficient consensus of what to expect of literature reviews in HCI. Thus, a shared understanding of literature reviews and clear terminology is needed to plan, evaluate, and use literature reviews, and to further improve review methodology. We analysed 189 literature reviews published at all SIGCHI conferences and ACM Transactions on Computer-Human Interaction (TOCHI) up until August 2022. We report on the main dimensions of variation: (i) contribution types and topics; and (ii) structure and methodologies applied. We identify gaps and trends to inform future meta work in HCI and provide a starting point on how to move towards a more comprehensive terminology system of literature reviews in HCI.

著者
Evropi Stefanidi
University of Bremen, Bremen, Germany
Marit Bentvelzen
Utrecht University, Utrecht, Netherlands
Paweł W. Woźniak
Chalmers University of Technology, Gothenburg, Sweden
Thomas Kosch
HU Berlin, Berlin, Germany
Mikołaj P.. Woźniak
University of Oldenburg, Oldenburg, Germany
Thomas Mildner
University of Bremen, Bremen, Germany
Stefan Schneegass
University of Duisburg-Essen, Essen, Germany
Heiko Müller
University of Oldenburg, Oldenburg, Germany
Jasmin Niess
University of St. Gallen, St. Gallen, Switzerland
論文URL

https://doi.org/10.1145/3544548.3581332

動画