The ACM CHI Conference on Human Factors in Computing Systems (https://chi2024.acm.org/)

Better to Ask Than Assume: Proactive Voice Assistants’ Communication Strategies That Respect User Agency in a Smart Home Environment
Jeesun Oh (KAIST, Daejeon, Korea, Republic of)Wooseok Kim (KAIST, Daejeon, Korea, Republic of)Sungbae Kim (KAIST, Daejeon, Korea, Republic of)Hyeonjeong Im (KAIST, Daejeon, Korea, Republic of)Sangsu Lee (KAIST, Daejeon, Korea, Republic of)
Proactive voice assistants (VAs) in smart homes predict users’ needs and autonomously take action by controlling smart devices and initiating voice-based features to support users’ various activities. Previous studies on proactive systems have primarily focused on determining action based on contextual information, such as user activities, physiological state, or mobile usage. However, there is a lack of research that considers user agency in VAs’ proactive actions, which empowers users to express their dynamic needs and preferences and promotes a sense of control. Thus, our study aims to explore verbal communication through which VAs can proactively take action while respecting user agency. To delve into communication between a proactive VA and a user, we used the Wizard of Oz method to set up a smart home environment, allowing controllable devices and unrestrained communication. This paper proposes design implications for the communication strategies of proactive VAs that respect user agency.
Charting Ethical Tensions in Multispecies Technology Research through Beneficiary-Epistemology Space
Steven David. Benford (University of Nottingham, Nottingham, United Kingdom)Clara Mancini (The Open University, Milton Keynes, United Kingdom)Alan Chamberlain (University of Nottingham, Nottingham, United Kingdom)Eike Schneiders (University of Nottingham, Nottingham, United Kingdom)Simon D. Castle-Green (University of Nottingham, Nottingham, Nottinghamshire, United Kingdom)Joel E. Fischer (University of Nottingham, Nottingham, United Kingdom)Ayse Kucukyilmaz (University of Nottingham, Nottingham, United Kingdom)Guido Salimbeni (University of Nottingham, Nottingham, United Kingdom)Victor Zhi Heung. Ngo (University of Nottingham, Nottingham, United Kingdom)Pepita Barnard Stringer (University of Nottingham, Nottingham, United Kingdom)Matt Adams (Blast Theory, Brighton, United Kingdom)Nick Tandavanitj (Blast Theory, Brighton, United Kingdom)Ju Row Farr (Blast Theory, Brighton, United Kingdom)
While ethical challenges are widely discussed in HCI, far less is reported about the ethical processes that researchers routinely navigate. We reflect on a multispecies project that negotiated an especially complex ethical approval process. Cat Royale was an artist-led exploration of creating an artwork to engage audiences in exploring trust in autonomous systems. The artwork took the form of a robot that played with three cats. Gaining ethical approval required an extensive dialogue with three Institutional Review Boards (IRBs) covering computer science, veterinary science and animal welfare, raising tensions around the welfare of the cats, perceived benefits and appropriate methods, and reputational risk to the University. To reveal these tensions we introduce beneficiary-epistemology space, that makes explicit who benefits from research (humans or animals) and underlying epistemologies. Positioning projects and IRBs in this space can help clarify tensions and highlight opportunities to recruit additional expertise.
SoniWeight Shoes: Investigating Effects and Personalization of a Wearable Sound Device for Altering Body Perception, Behavior and Emotion
Amar D'Adamo (Universidad Carlos III de Madrid, Madrid, Spain)Marte Roel Lesur (Universidad Carlos III de Madrid, Madrid, Spain)Laia Turmo Vidal (Universidad Carlos III de Madrid, Madrid, Spain)Mohammad Mahdi Dehshibi (Universidad Carlos III de Madrid, Madrid, Spain)Daniel De La Prida (Universidad Carlos III de Madrid, Madrid, Spain)Joaquin R.. Diaz Duran (Universidad Carlos III de Madrid, Madrid, Spain)Luis Antonio Azpicueta-Ruiz (Universidad Carlos III de Madrid, Madrid, Spain)Aleksander Väljamäe (University of Tartu, Tartu, Estonia)Ana Tajadura-Jiménez (Universidad Carlos III de Madrid, Leganés, Madrid, Spain)
Changes in body perception influence behavior and emotion and can be induced through multisensory feedback. Auditory feedback to one's actions can trigger such alterations; however, it is unclear which individual factors modulate these effects. We employ and evaluate SoniWeight Shoes, a wearable device based on literature for altering one's weight perception through manipulated footstep sounds. In a healthy population sample across a spectrum of individuals (n=84) with varying degrees of eating disorder symptomatology, physical activity levels, body concerns, and mental imagery capacities, we explore the effects of three sound conditions (low-frequency, high-frequency and control) on extensive body perception measures (demographic, behavioral, physiological, psychological, and subjective). Analyses revealed an impact of individual differences in each of these dimensions. Besides replicating previous findings, we reveal and highlight the role of individual differences in body perception, offering avenues for personalized sonification strategies. Datasets, technical refinements, and novel body map quantification tools are provided.
No More Angry Birds: Investigating Touchscreen Ergonomics to Improve Tablet-Based Enrichment for Parrots
Rebecca Kleinberger (Northeastern University, Boston, Massachusetts, United States)Jennifer Cunha (Northeastern University, Boston, Massachusetts, United States)Megan McMahon (Northeastern University, Boston, Massachusetts, United States)Ilyena Hirskyj-Douglas (The University of Glasgow, Glasgow, United Kingdom)
Touchscreen devices, ubiquitous in humans' day-to-day lives, offer a promising avenue for animal enrichment. With advanced cognitive abilities, keen visual perception, and adeptness to engage with capacitive screens using dexterous tongues, parrots are uniquely positioned to benefit from this technology. Additionally, pet parrots often lack appropriate stimuli, supporting the need for inexpensive solutions using off-the-shelf devices. However, the current human-centric interaction design standards of tablet applications do not optimally cater to the tactile affordances and ergonomic needs of parrots. To address this, we conducted a study with 20 pet parrots, examining their tactile interactions with touchscreens and evaluating the applicability of existing HCI interaction models. Our research highlights key ergonomic characteristics unique to parrots, which include pronounced multi-tap behavior, a critical size threshold for touch targets, and greater effectiveness of larger targets over closer proximity. Based on these insights, we propose guidelines for tablet-based enrichment systems for companion parrots.
Teachers, Parents, and Students' perspectives on Integrating Generative AI into Elementary Literacy Education
Ariel Han (UC Irvine, Irvine, California, United States)Xiaofei Zhou (University of Rochester, Rochester, New York, United States)Zhenyao Cai (University of California Irvine, Irvine, California, United States)Shenshen Han (University of California, Irvine, Irvine, California, United States)Richard Ko (UCI, Irvine, California, United States)Seth Corrigan (UC Irvine, Irvine, California, United States)Kylie A. Peppler (University of California, Irvine, Irvine, California, United States)
The viral launch of new generative AI (GAI) systems, such as ChatGPT and Text-to-Image (TTL) generators, sparked questions about how they can be effectively incorporated into writing education. However, it is still unclear how teachers, parents, and students perceive and suspect GAI systems in elementary school settings. We conducted a workshop with twelve families (parent-child dyads) with children ages 8-12 and interviewed sixteen teachers in order to understand each stakeholder's perspectives and opinions on GAI systems for learning and teaching writing. We found that the GAI systems could be beneficial in generating adaptable teaching materials for teachers, enhancing ideation, and providing students with personalized, timely feedback. However, there are concerns over authorship, students’ agency in learning, and uncertainty concerning bias and misinformation. In this article, we discuss design strategies to mitigate these constraints by implementing an adults-oversight system, balancing AI-role allocation, and facilitating customization to enhance students’ agency over writing projects.
Me, My Health, and My Watch: How Children with ADHD Understand Smartwatch Health Data
Elizabeth Ankrah (University of California, Irvine, Irvine, California, United States)Franceli L.. Cibrian (Chapman University, Orange, California, United States)Lucas M.. Silva (University of California, Irvine, Irvine, California, United States)Arya Tavakoulnia (University of California Irvine, Irvine, California, United States)Jesus Armando. Beltran (UCI, Irvine, California, United States)Sabrina Schuck (University of California Irvine, Irvine, California, United States)Kimberley D. Lakes (University of California Riverside, Riverside, California, United States)Gillian R. Hayes (University of California, Irvine, Irvine, California, United States)
Children with ADHD can experience a wide variety of challenges related to self-regulation, which can lead to poor educational, health, and wellness outcomes. Technological interventions, such as mobile and wearable health systems, can support data collection and reflection about health status. However, little is known about how ADHD children interpret such data. We conducted a deployment study with 10 children, aged 10 to 15, for six weeks, during which they used a smartwatch in their homes. Results from observations and interviews during this study indicate that children with ADHD can interpret their own health data, particularly at the moment. However, as ADHD children develop more autonomy, smartwatch systems may require alternatives for data reflection that are interpretable and actionable for them. This work contributes to the scholarly discourse around health data visualization, particularly in considering implications for the design of health technologies for children with ADHD.
Uncovering Lemur Cross-Species Usage of an Interactive Audio Device In Zoos
Vilma Kankaanpää (The University of Glasgow, Glasgow, United Kingdom)Fay E. Clark (University of Bristol, Bristol, United Kingdom)Ilyena Hirskyj-Douglas (The University of Glasgow, Glasgow, United Kingdom)
Computer technology for animals is typically oriented toward isolated individuals, seldom attending to such group-living factors as accommodating differences between individuals. To address this shortcoming of research and practice, the authors designed and developed an audio-based system that lets lemurs in group accommodation voluntarily trigger audio via a novel device dubbed LemurLounge and listen to it on their own. This interactive system was deployed for 14 lemurs, of three species (black-and-white, brown, and ring-tailed), in their normal habitat. The device's presence clearly influenced lemurs' visits to the relevant portion of the enclosure. Alongside a general preference for audio over silence, assessment of individual- and species-level differences revealed significant differences at both levels, though no particular sound type (rainfall, traffic, either upbeat or relaxing music, or white noise) was favoured. The findings and design work highlight the need for customisable and adaptive computer technology for animals living in group settings, with important implications for lemurs and other primates, humans included.
Cooking With Agents: Designing Context-aware Voice Interaction
Razan Jaber (Stockholm University , Stockholm, Sweden)Sabrina Zhong (University College London, London, United Kingdom)Sanna Kuoppamäki (KTH Royal Institute of Technology, Stockholm, Sweden)Aida Hosseini (KTH Royal Institute of Technology, Stockholm, Sweden)Iona Gessinger (University College Dublin, Dublin, Ireland)Duncan P. Brumby (University College London, London, United Kingdom)Benjamin R.. Cowan (University College Dublin, Dublin, Ireland)Donald McMillan (Stockholm University , Stockholm, Sweden)
Voice Agents (VAs) are touted as being able to help users in complex tasks such as cooking and interacting as a conversational partner to provide information and advice while the task is ongoing. Through conversation analysis of 7 cooking sessions with a commercial VA, we identify challenges caused by a lack of contextual awareness leading to irrelevant responses, misinterpretation of requests, and information overload. Informed by this, we evaluated 16 cooking sessions with a wizard-led context-aware VA. We observed more fluent interaction between humans and agents, including more complex requests, explicit grounding within utterances, and complex social responses. We discuss reasons for this, the potential for personalisation, and the division of labour in VA communication and proactivity. Then, we discuss the recent advances in generative models and the VAs interaction challenges. We propose limited context awareness in VAs as a step toward explainable, explorable conversational interfaces.
EcoSanté Lifestyle Intervention: Encourage Reflections on the Connections between Health and Environment
Pei-Yi (Patricia) Kuo (National Tsing Hua University, Hsinchu, Taiwan)Mike Horn (Northwestern University, Evanston, Illinois, United States)
EcoSanté is a mobile lifestyle intervention that encourages individual behavior change while also helping participants understand the deep connections between daily lifestyle choices and our collective impact on the planet. Informed by research on “small” intervention approaches, we sent participants daily behavioral challenges that demonstrated connections between personal health and environmental impact at large. Through a 20-day mobile intervention study, 139 participants uploaded 1,920 submissions documenting their attempts to engage in these challenges. We found that participants’ self-reported healthy eating behavior and general self-efficacy improved significantly immediately after the intervention. Moreover, 30 days after the intervention, participants’ self-reported eating, exercise, and general self-efficacy all significantly improved compared to the beginning of the study. Participants had a more negative reaction when being asked to come up with their own challenges. Based on quantitative and qualitative findings, we provide implications for future researcher on mobile behavior intervention research.
Call of the Wild Web: Comparing Parrot Engagement in Live vs. Pre-Recorded Video Calls
Ilyena Hirskyj-Douglas (The University of Glasgow, Glasgow, United Kingdom)Jennifer Cunha (Parrot Kindergarten, Jupiter, Florida, United States)Rebecca Kleinberger (Northeastern University, Boston, Massachusetts, United States)
The concept of the animal Internet has flourished, with many conceptualisations proceeding from the premise that connecting animals online may enrich their social life. Yet we remain unaware of how -- or even whether -- online interactions (either live or with pre-recorded material) might affect how animals engage with other animals. We implemented a system for parrots to trigger live video calls with other birds or playback from a pre-recorded video call. The goal was to identify differences in engagement and behaviours. Over a six-month study, parrots triggered significantly more live calls and engaged longer in that setting relative to the playback condition, while the animals' caregivers found greater value in the latter but preferred the live alternative for the birds under their care. The results begin to question what animals make of online remote connections, putting forward considerations as to how the internet can affect animals' experiences.
Ellie Talks About the Weather: Toward Evaluating the Expressive and Enrichment Potential of a Tablet-Based Speech Board in a single Goffin’s Cockatoo
Jennifer Cunha (Northeastern University, Boston, Massachusetts, United States)Corinne C. Renguette (Indiana University at Purdue University in Indianapolis, Indianapolis, Indiana, United States)Nikhil Singh (Massachusetts Institute of Technology, Cambridge, Massachusetts, United States)Lily Stella (Indiana University at Purdue University in Indianapolis, Indianapolis, Indiana, United States)Megan McMahon (Northeastern University, Boston, Massachusetts, United States)Hao Jin (Northeastern University, Boston, Massachusetts, United States)Rebecca Kleinberger (Northeastern University, Boston, Massachusetts, United States)
Augmentative and alternative communication devices (AACs) are designed to assist humans with complex communication needs. Recently, AAC use has been reported in non-human animals. Such tools may potentially provide enrichment and increase interspecies connection. However, there is no evaluation framework and little data available to assess AAC potential. Here, we examine seven months of a single parrot’s sustained use of a tablet-based AAC totaling 129 sessions within 190 days. After devising a coding schema, we propose a framework to explore the expressive potential and enrichment value for the parrot. Our results suggest that the choice of destination words cannot be simply explained based on random selection or icon location alone, and 92\% of corroborable selections are validated by behaviors. The parrot interactions also appear significantly skewed toward social and cognitive enrichment. This work is a first step toward assessment of AAC use for parrot enrichment.
Exploring the Lived Experience of Behavior Change Technologies: Towards an Existential Model of Behavior Change for HCI
Amon Rapp (University of Turin, Torino, Italy)Arianna Boldi (University of Turin, Torino, ITALY, Italy)
The majority of behavior change and persuasive technologies are exclusively addressed to modify a specific behavior. However, the focus on behavior may cloud the “existential aspects” of the process of change. To explore the lived and meaning-laden experience of behavior change, we interviewed 23 individuals who have used behavior change technology in their everyday life. The study findings highlight that behavior change is tied to meanings that point to existential matters, relates to a nexus of life circumstances, and unfolds over long periods of time. By contrast, the technology used by the participants appears mostly to focus on the present target behavior, ignoring its links to the participants’ life “context” and “time,” also providing scarce help for sense-making. Based on these findings, we surface a preliminary “existential model of behavior change,” identify several barriers that may prevent the modification of behavior and propose some design suggestions to overcome them.
Putting Things into Context: Generative AI-Enabled Context Personalization for Vocabulary Learning Improves Learning Motivation
Joanne Leong (MIT, Cambridge, Massachusetts, United States)Pat Pataranutaporn (MIT, Boston, Massachusetts, United States)Valdemar Danry (MIT, CAMBRIDGE, Massachusetts, United States)Florian Perteneder (Independent, Hagenberg, Austria)Yaoli Mao (Columbia University, New York, New York, United States)Pattie Maes (MIT Media Lab, Cambridge, Massachusetts, United States)
Fostering students' interests in learning is considered to have many positive downstream effects. Large language models have opened up new horizons for generating content tuned to one's interests, yet it is unclear in what ways and to what extent this customization could have positive effects on learning. To explore this novel dimension, we conducted a between-subjects online study (n=272) featuring different variations of a generative AI vocabulary learning app that enables users to personalize their learning examples. Participants were randomly assigned to control (sentence sourced from pre-existing text) or experimental conditions (generated sentence or short story based on users’ text input). While we did not observe a difference in learning performance between the conditions, the analysis revealed that generative AI-driven context personalization positively affected learning motivation. We discuss how these results relate to previous findings and underscore their significance for the emerging field of using generative AI for personalized learning.
The Sound of Support: Gendered Voice Agent as Support to Minority Teammates in Gender-Imbalanced Team
Angel Hsing-Chi Hwang (Cornell University, Ithaca, New York, United States)Andrea Stevenson Won (Cornell University, Ithaca, New York, United States)
The present work explores the potential of leveraging a teamwork agent's identity -- signaled through its gendered voice -- to support marginalized individuals in gender-imbalanced teams. In a mixed design experiment (N = 178), participants were randomly assigned to work with a female and a male voice agent in either a female-dominated or male-dominated team. Results show the presence of a same-gender voice agent is particularly beneficial to the performance of marginalized female members, such that they would contribute more ideas and talk more when a female agent was present. Conversely, marginalized male members became more talkative but were less focused on the teamwork tasks at hand when working with a male-sounding agent. The findings of the present experiment support existing literature on the effect of social presence in gender-imbalanced teams, such that gendered agents serve similar benefits as human teammates of the same gender identities. However, the effect of agents' presence remains limited when participants have experienced severe marginalization in the past. Based on findings from the present study, we discuss relevant design implications and avenues for future research.
Designing Multispecies Worlds for Robots, Cats, and Humans
Eike Schneiders (University of Nottingham, Nottingham, United Kingdom)Steven David. Benford (University of Nottingham, Nottingham, United Kingdom)Alan Chamberlain (University of Nottingham, Nottingham, United Kingdom)Clara Mancini (The Open University, Milton Keynes, United Kingdom)Simon D. Castle-Green (University of Nottingham, Nottingham, Nottinghamshire, United Kingdom)Victor Zhi Heung. Ngo (University of Nottingham, Nottingham, United Kingdom)Ju Row Farr (Blast Theory, Brighton, United Kingdom)Matt Adams (Blast Theory, Brighton, United Kingdom)Nick Tandavanitj (Blast Theory, Brighton, United Kingdom)Joel E. Fischer (University of Nottingham, Nottingham, United Kingdom)
We reflect on the design of a multispecies world centred around a bespoke enclosure in which three cats and a robot arm coexist for six hours a day during a twelve-day installation as part of an artist-led project. In this paper, we present the project's design process, encompassing various interconnected components, including the cats, the robot and its autonomous systems, the custom end-effectors and robot attachments, the diverse roles of the humans-in-the-loop, and the custom-designed enclosure. Subsequently, we provide a detailed account of key moments during the deployment and discuss the design implications for future multispecies systems. Specifically, we argue that designing the technology and its interactions is not sufficient, but that it is equally important to consider the design of the `world' in which the technology operates. Finally, we highlight the necessity of human involvement in areas such as breakdown recovery, animal welfare, and their role as audience.
MoiréWidgets: High-Precision, Passive Tangible Interfaces via Moiré Effect
Daniel Campos Zamora (University of Washington, Seattle, Washington, United States)Mustafa Doga Dogan (MIT CSAIL, Cambridge, Massachusetts, United States)Alexa Siu (Adobe Research, San Jose, California, United States)Eunyee Koh (Adobe Research, San Jose, California, United States)Chang Xiao (Adobe Research, San Jose, California, United States)
We introduce MoiréWidgets, a novel approach for tangible interaction that harnesses the Moiré effect—a prevalent optical phenomenon—to enable high-precision event detection on physical widgets. Unlike other electronics-free tangible user interfaces which require close coupling with external hardware, MoiréWidgets can be used at greater distances while maintaining high-resolution sensing of interactions. We define a set of interaction primitives, e.g., buttons, sliders, and dials, which can be used as standalone objects or combined to build complex physical controls. These consist of 3D printed structural mechanisms with patterns printed on two layers—one on paper and the other on a plastic transparency sheet—which create a visual signal that amplifies subtle movements, enabling the detection of user inputs. Our technical evaluation shows that our method outperforms standard fiducial markers and maintains sub-millimeter accuracy at 100 cm distance and wide viewing angles. We demonstrate our approach by creating an audio console and indicate how our approach could extend to other domains.
One Size Does Not Fit All: Designing and Evaluating Criticality-Adaptive Displays in Highly Automated Vehicles
Yaohan Ding (University of Pittsburgh, Pittsburgh, Pennsylvania, United States)Lesong Jia (University of Pittsburgh , Pittsburgh, Pennsylvania, United States)Na Du (University of Pittsburgh, Pittsburgh, Pennsylvania, United States)
To promote drivers' overall experiences in highly automated vehicles, we designed three objective criticality-adaptive displays:IO display highlighting Influential Objects, CO display highlighting Critical Objects, and ICO display highlighting Influential and Critical Objects differently. We conducted an online video-based survey study with 295 participants to evaluate them in varying traffic conditions. Results showed that low-trust propensity participants found ICO display more useful while high-trust propensity participants found CO displays more useful. When interacting with vulnerable road users (VRUs), participants had higher situational awareness (SA) but worse non-driving related task (NDRT) performance. Aging and CO displays also led to slower NDRT reactions. Nonetheless, older participants found displays more useful. We recommend providing different criticality-adaptive displays based on drivers' trust propensity, age, and NDRT choice to enhance driving and NDRT performance and suggest carefully treating objects of different categories in traffic.
Saharaline: A Collective Social Support Intervention for Teachers in Low-Income Indian Schools
Rama Adithya Varanasi (Cornell University, Ithaca, New York, United States)Nicola Dell (Cornell Tech, New York, New York, United States)Aditya Vashistha (Cornell University, Ithaca, New York, United States)
This paper presents Saharaline, an intervention designed to provide collective social support for teachers in low-income schools. Implemented as a WhatsApp-based helpline, Saharaline enables teachers to reach out for personalized, long-term assistance with a wide range of problems and stressors, including pedagogical, technological, and emotional challenges. Depending on the support needed, teachers' requests are routed to appropriate domain experts--- staff employed by educational non-profit organizations who understand teachers' on-the-ground realities---who offer localized and contextualized assistance. Via a three-month exploratory deployment with 28 teachers in India, we show how Saharaline's design enabled a collective of diverse education experts to craft and deliver localized solutions that teachers could incorporate into their practice. We conclude by reflecting on the efficacy of our intervention in low-resource work contexts and provide recommendations to enhance collective social support interventions similar to Saharaline.
UI Mobility Control in XR: Switching UI Positionings between Static, Dynamic, and Self Entities
Siyou Pei (University of California Los Angeles, Los Angeles, California, United States)David Kim (Google Research, Zurich, Switzerland)Alex Olwal (Google Research, Mountain View, California, United States)Yang Zhang (University of California Los Angeles, Los Angeles, California, United States)Ruofei Du (Google Research, San Francisco, California, United States)
Extended reality (XR) has the potential for seamless user interface (UI) transitions across people, objects, and environments. However, the design space, applications, and common practices of 3D UI transitions remain underexplored. To address this gap, we conducted a need-finding study with 11 participants, identifying and distilling a taxonomy based on three types of UI placements --- affixed to static, dynamic, or self entities. We further surveyed 113 commercial applications to understand the common practices of 3D UI mobility control, where only 6.2% of these applications allowed users to transition UI between entities. In response, we built interaction prototypes to facilitate UI transitions between entities. We report on results from a qualitative user study (N=14) on 3D UI mobility control using our FingerSwitches technique, which suggests that perceived usefulness is affected by types of entities and environments. We aspire to tackle a vital need in UI mobility within XR.
VirtuWander: Enhancing Multi-modal Interaction for Virtual Tour Guidance through Large Language Models
Zhan Wang (The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China)Linping Yuan (The Hong Kong University of Science and Technology, Hong Kong, China)Liangwei Wang (The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China)Bingchuan Jiang (Information Engineering University, Zheng Zhou, China)Wei Zeng (The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, China)
Tour guidance in virtual museums encourages multi-modal interactions to boost user experiences, concerning engagement, immersion, and spatial awareness. Nevertheless, achieving the goal is challenging due to the complexity of comprehending diverse user needs and accommodating personalized user preferences. Informed by a formative study that characterizes guidance-seeking contexts, we establish a multi-modal interaction design framework for virtual tour guidance. We then design VirtuWander, a two-stage innovative system using domain-oriented large language models to transform user inquiries into diverse guidance-seeking contexts and facilitate multi-modal interactions. The feasibility and versatility of VirtuWander are demonstrated with virtual guiding examples that encompass various touring scenarios and cater to personalized preferences. We further evaluate VirtuWander through a user study within an immersive simulated museum. The results suggest that our system enhances engaging virtual tour experiences through personalized communication and knowledgeable assistance, indicating its potential for expanding into real-world scenarios.
ProInterAR: A Visual Programming Platform for Creating Immersive AR Interactions
Hui Ye (City University of Hong Kong, Hong Kong, Hong Kong)Jiaye Leng (City University of HongKong, HongKong, China)Pengfei Xu (Shenzhen University, Shenzhen, Guangdong, China)Karan Singh (University of Toronto, Toronto, Ontario, Canada)Hongbo Fu (City University of Hong Kong, Beijing, Beijing, China)
AR applications commonly contain diverse interactions among different AR contents. Creating such applications requires creators to have advanced programming skills for scripting interactive behaviors of AR contents, repeated transferring and adjustment of virtual contents from virtual to physical scenes, testing by traversing between desktop interfaces and target AR scenes, and digitalizing AR contents. Existing immersive tools for prototyping/authoring such interactions are tailored for domain-specific applications. To support programming general interactive behaviors of real object(s)/environment(s) and virtual object(s)/environment(s) for novice AR creators, we propose ProInterAR, an integrated visual programming platform to create immersive AR applications with a tablet and an AR-HMD. Users can construct interaction scenes by creating virtual contents and augmenting real contents from the view of an AR-HMD, script interactive behaviors by stacking blocks from a tablet UI, and then execute and control the interactions in the AR scene. We showcase a wide range of AR application scenarios enabled by ProInterAR, including AR game, AR teaching, sequential animation, AR information visualization, etc. Two usability studies validate that novice AR creators can easily program various desired AR applications using ProInterAR.
MineXR: Mining Personalized Extended Reality Interfaces
Hyunsung Cho (Carnegie Mellon University, Pittsburgh, Pennsylvania, United States)Yukang Yan (Carnegie Mellon University, Pittsburgh, Pennsylvania, United States)Kashyap Todi (Reality Labs Research, Redmond, Washington, United States)Mark Parent (Meta, Toronto, Ontario, Canada)Missie Smith (Reality Labs Research, Redmond, Washington, United States)Tanya R.. Jonker (Meta Inc., Redmond, Washington, United States)Hrvoje Benko (Meta Inc., Redmond, Washington, United States)David Lindlbauer (Carnegie Mellon University, Pittsburgh, Pennsylvania, United States)
Extended Reality (XR) interfaces offer engaging user experiences, but their effective design requires a nuanced understanding of user behavior and preferences. This knowledge is challenging to obtain without the widespread adoption of XR devices. We introduce MineXR, a design mining workflow and data analysis platform for collecting and analyzing personalized XR user interaction and experience data. MineXR enables elicitation of personalized interfaces from participants of a data collection: for any particular context, participants create interface elements using application screenshots from their own smartphone, place them in the environment, and simultaneously preview the resulting XR layout on a headset. Using MineXR, we contribute a dataset of personalized XR interfaces collected from 31 participants, consisting of 695 XR widgets created from 178 unique applications. We provide insights for XR widget functionalities, categories, clusters, UI element types, and placement. Our open-source tools and data support researchers and designers in developing future XR interfaces.
Touching the Moon: Leveraging Passive Haptics, Embodiment and Presence for Operational Assessments in Virtual Reality
Florian Dufresne (Arts et Métiers Institute of Technology, Changé, France)Tommy Nilsson (European Space Agency (ESA), Cologne, -, Germany)Geoffrey Gorisse (Arts et Métiers Institute of Technology, Changé, France)Enrico Guerra (University Duisburg-Essen , Duisburg , Germany)André Zenner (Saarland University, Saarland Informatics Campus, Saarbrücken, Germany)Olivier Christmann (Arts et Métiers Institute of Technology, Changé, France)Leonie Bensch (German Aerospace Center (DLR), Cologne, North Rhine-Westphalia, Germany)Nikolai Anton. Callus (European Space Agency, Cologne, Germany)Aidan Cowley (European Space Agency, Cologne, North Rhine-Westphalia, Germany)
Space agencies are in the process of drawing up carefully thought-out Concepts of Operations (ConOps) for future human missions on the Moon. These are typically assessed and validated through costly and logistically demanding analogue field studies. While interactive simulations in Virtual Reality (VR) offer a comparatively cost-effective alternative, they have faced criticism for lacking the fidelity of real-world deployments. This paper explores the applicability of passive haptic interfaces in bridging the gap between simulated and real-world ConOps assessments. Leveraging passive haptic props (equipment mockup and astronaut gloves), we virtually recreated the Apollo 12 mission procedure and assessed it with experienced astronauts and other space experts. Quantitative and qualitative findings indicate that haptics increased presence and embodiment, thus improving perceived simulation fidelity and validity of user reflections. We conclude by discussing the potential role of passive haptic modalities in facilitating early-stage ConOps assessments for human endeavours on the Moon and beyond.