Qualitative inductive methods are widely used in CSCW and HCI research for their ability to generatively discover deep and contextualized insights, but these inherently manual and human-resource-intensive processes are often infeasible for analyzing large corpora. Researchers have been increasingly interested in ways to apply qualitative methods to "big" data problems, hoping to achieve more generalizable results from larger amounts of data while preserving the depth and richness of qualitative methods. In this paper, we describe a study of qualitative researchers' work practices and their challenges, with an eye towards whether this is an appropriate domain for human-AI collaboration and what successful collaborations might entail. Our findings characterize participants' diverse methodological practices and nuanced collaboration dynamics, and identify areas where they might benefit from from AI-based tools. While participants highlight the messiness and uncertainty of qualitative inductive analysis, they still want full agency over the process and believe that AI should not interfere. Our study provides a deep investigation of task delegability in human-AI collaboration in the context of qualitative research, and offers directions for the design of AI assistance that honor serendipity, human agency, and ambiguity.
https://doi.org/10.1145/3449168
‘Big data’ corpora are typically the purview of quantitative scholars, who may work with computational tools to derive numerical and descriptive insights. Recent work asks how technology, such as AI, can support qualitative scholars in developing deep and complex insights from large datasets. Jiang et al. address this question, finding that qualitative scholars are generally opposed to using AI to support their practices of data analysis. However, in this paper, we provide nuance to these earlier findings, showing that the stage of qualitative analysis matters for how scholars feel AI can, and should be, used. Through interviews with 15 CSCW and HCI researchers who engage with qualitative analysis of large corpora, we examine AI use at different stages of qualitative analysis. We find that qualitative scholars are open to using AI in diverse ways, such as for data exploration and coding, as long as it supports rather than automates their analytic work practice. Based on our analysis, we discuss how the incorporation of AI can shift some qualitative analysis practices as well as how designing for human-AI collaboration in qualitative analysis necessitates considering the tradeoffs in designing for scale, abstraction, and task delegation.
https://doi.org/10.1145/3479856
Despite the promises of data-driven artificial intelligence (AI), little is known about how we can bridge the gulf between traditional physician-driven diagnosis and a plausible future of medicine automated by AI. Specifically, how can we involve AI usefully in physicians’ diagnosis workflow given that most AI is still nascent and error-prone (e.g., in digital pathology)? To explore this question, we first propose a series of collaborative techniques to engage human pathologists with AI given AI’s capabilities and limitations, based on which we prototype Impetus—a tool where an AI takes various degrees of initiatives to provide various forms of assistance to a pathologist in detecting tumors from histological slides. We summarize observations and lessons learned from a study with eight pathologists and discuss recommendations for future work on human-centered medical AI systems.
https://doi.org/10.1145/3449084
The need for advanced automation and artificial intelligence (AI) in various fields, including text classification, has dramatically increased in the last decade, leaving us critically dependent on their performance and reliability. Yet, as we increasingly rely more on AI applications, their algorithms are becoming more nuanced, more complex, and less understandable precisely at a time we need to understand them better and trust them to perform as expected. Text classification in the medical and cybersecurity domains are good examples of this. Human experts lack the capacity to deal with the high volume and velocity of data that needs to be classified, and ML techniques are often unexplainable and lack the ability to capture the required context needed to make the right decision and take action. We propose a new abstract configuration of Human-Machine Learning (HML) that focuses on reciprocal learning, where the human and the AI are collaborating partners. We employ design-science research (DSR) to learn and design an application of the HML configuration, which incorporates software to support combining human and artificial intelligences. We define the HML configuration by its conceptual components and their function. We then describe the development of a system called Fusion that supports human-machine reciprocal learning. Using two case studies of text classification from the cyber domain, we evaluate Fusion and the proposed HML approach, demonstrating benefits and challenges. Our results show a clear ability of domain experts to improve the ML classification performance over time, while both human and machine, collaboratively, develop their conceptualization, i.e., their knowledge of classification. We generalize our insights from the DSR process as actionable principles for researchers and designers of 'human in the learning loop' systems. We conclude the paper by discussing HML configurations and the challenge of capturing and representing knowledge gained jointly by human and machine, an area we feel has great potential.
https://doi.org/10.1145/3479587
Chatbots are regarded as a promising technology for delivering guidance. Prior studies show that chatbots have the potential of coaching users to learn different skills; however, several limitations of chatbot-based approaches remain. People may become disengaged from using chatbot-guided systems and fail to follow the guidance for complex tasks. In this paper, we design chatbots with (HC) and without (OC) human support to deliver guidance for people to practice journaling skills. We conducted a mixed-method study with 35 participants to investigate their actual interaction, perceived interaction, and the effects of interacting with the two chatbots. The participants were randomly assigned to use one of the chatbots for four weeks. Our results show that the HC participants followed the guidance more faithfully during journaling practices and perceived a significantly higher level of engagement and trust with the chatbot system than the OC participants. However, after finishing the journaling-skill training session, the OC participants were more willing to keep using the learned skills than the HC participants. Our work provides new insights into the design of integrating human support into chatbot-based interventions for delivering guidance.
AI-based educational technologies may be most welcome in classrooms when they align with teachers’ goals,preferences, and instructional practices. Teachers, however, have scarce time to make such customizationsthemselves. How might the crowd be leveraged to help time-strapped teachers? Crowdsourcing pipelineshave traditionally focused on content generation. It is, however, an open question how a pipeline might bedesigned so the crowd can succeed in a revision/customization task. In this paper, we explore an initial versionof a teacher-guided crowdsourcing pipeline designed to improve the adaptive math hints of an AI-basedtutoring system so they fit teachers’ preferences, while requiring minimal expert guidance. In two experimentsinvolving 144 math teachers and 481 crowdworkers, we found that such an expert-guided revision pipelinecould save experts’ time and produce better crowd-revised hints (in terms of teacher satisfaction) than twogeneration conditions. The revised hints however, did not improve on the existing hints in the AI tutor, whichwere already highly rated, though with room for improvement and customization. Further analysis revealedthat the main challenge for crowdworkers may lie in understanding teachers’ brief written comments andimplementing them in the form of effective edits, without introducing new problems. We also found thatteachers preferred their own revisions over other sources of hints, and exhibited varying preferences over hintsin AI-tutor. Overall, the results confirm that there is a clear need for customizing hints to individual teachers’preferences, but also highlight the need for more elaborate scaffolds so the crowd has specific knowledge ofthe requirements that teachers have for hints. The study represents a first exploration in the literature of howto support crowds with minimal expert guidance in revising and customizing instructional materials.
Software bots are used to streamline tasks in Open Source Software (OSS) projects' pull requests, saving development cost, time, and effort. However, their presence can be disruptive to the community. We identified several challenges caused by bots in pull request interactions by interviewing 21 practitioners, including project maintainers, contributors, and bot developers. In particular, our findings indicate noise as a recurrent and central problem. Noise affects both human communication and development workflow by overwhelming and distracting developers. Our main contribution is a theory of how human developers perceive annoying bot behaviors as noise on social coding platforms. This contribution may help practitioners understand the effects of adopting a bot, and researchers and tool designers may leverage our results to better support human-bot interaction on social coding platforms.
https://doi.org/10.1145/3476042
As algorithmic risk assessment instruments (RAIs) are increasingly adopted to assist decision makers, their predictive performance and properties vis-a-vis fairness have come under scrutiny. However, while most studies examine these tools in isolation, researchers have come to recognize that assessing their impact requires understanding the behavior of their human interactants. In this paper, building off of recent crowdsourcing studies focusing on criminal justice, we conduct a vignette study in which laypersons are tasked with predicting future re-arrests. Our key findings are as follows: (1) Participants often predict that an offender will be rearrested even when they deem the likelihood of re-arrest to be well below 50%; (2) Participants do not anchor on the RAI's predictions; (3) The time spent on the survey varies widely across participants and most cases are assessed in less than 10 seconds; (4) Judicial decisions, unlike participants' predictions, depend in part on factors that are orthogonal to the likelihood of recidivism. These results highlight the influence of several crucial but often overlooked design decisions and concerns around generalizability when constructing crowdsourcing studies to analyze the impacts of RAIs.
https://doi.org/10.1145/3479572