Qualitative researchers use tools to collect, sort, and analyze their data. Should qualitative researchers use large language models (LLMs) as part of their practice? LLMs could augment qualitative research, but it is unclear if their use is appropriate, ethical, or aligned with qualitative researchers’ goals and values. We interviewed twenty qualitative researchers to investigate these tensions. Many participants see LLMs as promising interlocutors with attractive use cases across the stages of research, but wrestle with their performance and appropriateness. Participants surface concerns regarding the use of LLMs while protecting participant interests, and call attention to an urgent lack of norms and tooling to guide the ethical use of LLMs in research. We document the rapid and broad adoption of LLMs across surfaces, which can interfere with intentional use vital to qualitative research. We use the tensions surfaced by our participants to outline recommendations for researchers considering using LLMs in qualitative research and design principles for LLM-assisted qualitative research tools.
Statistical reporting, especially of effect sizes, is at the root of many methodological issues in quantitative research at CHI. Effect sizes are necessary for assessing practical relevance of results, a-priori power analysis, and meta-analyses, but currently, they are often not reported. Interpretations in the context of the study and the research field are also rare. To aid to researchers in reporting and contextualizing their effect sizes within their research field as well as choosing effect sizes for power analysis, we conducted a meta-study of quantitative CHI papers. We extracted statistics from all quantitative CHI papers published between 2019-2023 (N=1692). Based on effect sizes and the papers' CCS categories, we present effect size distributions in 12 CHI research fields. Through an additional qualitative analysis of 67 quantitative CHI'23 publications, we identify five categories of approaches that researchers take when interpreting effect size: Comparing test-specific values, assigning size labels, using a statistical or methodological reference frame, comparing different observations and interpreting for the big picture.
Researchers often place a strong focus on statistical significance when reporting the results of statistical tests. However, effect sizes are reported less frequently, and interpretation in the context of the study and the research field is even rarer. These interpretations of effect sizes are, however, necessary to understand the practical importance of a result for the community. To explore how Usable Security & Privacy (USP) and HCI researchers interpret effect sizes and make judgments on practical importance, we conducted survey and interview studies with a total of 63 researchers at CHI and SOUPS 2023. Our studies focused on Cohen's d and odds ratios in two USP and one HCI scenario. We analyzed which artifacts researchers consider when judging effect size, and found misconceptions and variation between the participants, highlighting how difficult judging statistics can be. Based on our findings, we make concrete recommendations for improved reporting practices around effect sizes.
Researchers ask a lot from their study participants: data, time, attention, ideas, and (almost) anything that helps them to pursue their research goals. But what do they give back? This question becomes especially critical in longer-term participatory research with low-resourced communities. This paper offers methodological reflections on a collaboration with a Men’s Shed that was tailored around both my research agenda and the interests of my community partner. As part of my research, we designed a booklet that eventually became their promotion brochure. By reviewing both the trouble and the gains of this process for both partners, I argue for re-imagining community-based participatory research as an opportunity for fostering give-and-take relationships with participants. The case demonstrates the method's capacity to critically extend existing HCI work on Men’s Sheds while also making participation worthwhile for my partners. The careful documentation of this process contributes methodological nuance to discussions around configuring participation.
Although the literature on Human-Computer Interaction (HCI) catalogues many theories, it offers surprisingly few tools for theorising. This paper critiques dominant approaches to engaging with theory and proposes a working model for theorising in HCI. We then present graphical causal modelling as an effective theorising tool. This includes a step-by-step guide to building causal models and examples of their use in different stages of the research process. We explain how causal models help develop method-agnostic representations of research problems using directed acyclic graphs, identify potential confounders, and construct alternative interpretations of data. Finally, we discuss their limitations and challenges for adoption by the HCI community.
This paper introduces the Robotability Score (R), a novel metric that quantifies the suitability of urban environments for autonomous robot navigation. Through expert interviews and surveys, we identify and weigh key features contributing to R for wheeled robots on urban streets. Our findings reveal that pedestrian density, crowd dynamics and pedestrian flow are the most critical factors, collectively accounting for 28% of the total score. Computing robotability across New York City yields significant variation; the area of highest R is 3.0 times more "robotable'' than the area of lowest R. Deployments of a physical robot on high and low robotability areas show the adequacy of the score in anticipating the ease of robot navigation. This new framework for evaluating urban landscapes aims to reduce uncertainty in robot deployment while respecting established mobility patterns and urban planning principles, contributing to the discourse on harmonious human-robot environments.
We examine how micro-phenomenology, a qualitative research method developed to attend to, articulate, and analyse lived experience in fine detail, can be employed to study the experience of using digital systems. Micro-phenomenological interviews unpack the specific experiences of interviewees in fine-grained detail and have previously been acknowledged as a potent tool for Human-Computer Interaction Research. More recently, the method has been extended to comprise a structured analysis method to systematically analyse the temporal unfolding and qualitative dimensions of experiences captured by the interviews. This is the first paper demonstrating the combined use of interviews and analysis via a case in which they were employed to examine the experience of using WeUsedTo, a website for sharing experiences related to the COVID-19 pandemic. On this basis, we discuss the potentials of the method for eliciting and understanding experiential aspects of interactive systems, particularly pertaining to embodiment, temporality, attention, agency, and the systemic nature of experience.