Effective communication and teamwork are vital in high-stakes environments such as the operating room, where timely and accurate information exchange directly affects patient safety and surgical outcomes. Among intraoperative interactions, the collaboration between the surgeon and anesthesiologist is especially critical for maintaining smooth workflows and preventing adverse events. Despite its importance, little HCI research has explicitly examined the unique needs of this dyad or how AI-driven supportive systems might be designed to address them.
In this work, we present a qualitative study of surgeon–anesthesiologist collaboration, drawing on focus groups with both specialties and in-situ observations of 45 surgeries spanning open, laparoscopic, and robotic procedures. Our findings uncover key challenges, unmet needs, and coordination breakdowns that shape this relationship. Based on these insights, we conceptualize a systems design to better support intraoperative collaboration.
Medical 3D imaging allows surgeons to analyse liver anatomy through reconstructed models, supporting planning and decision-making. As these reconstructions enter operating room, it remains unclear how best to integrate them into surgical workflows, and which display modalities are most effective. After mapping current practices through user research, we investigated four approaches to presenting 3D reconstructions in simulated open-liver surgery. Pairs of surgeons diagnose cases and proposed hepatectomy incision lines under four conditions: (1) 3D reconstructions on a 2D monitor, (2) Augmented Reality (AR) overlay on a 2D monitor, (3) AR overlay via head-mounted display (HMD) for the main surgeon and 2D monitor for the assistant, and (4) AR overlay via HMDs for both surgeons. Results showed that the HMD+HMD condition was preferred for supporting anatomical understanding and collaboration. While accuracy was unaffected, findings emphasise the importance of shared perspective for effective teamwork using AR.
Interaction while performing physical tasks is inherently challenging, as both hands are fully engaged. In Minimally-Invasive Surgery (MIS), for instance, navigating images requires either delegation, causing frustration and delays, or hand de-sterilization, increasing risk. We introduce Interaction Through Instruments, an interaction paradigm in which task instruments become interaction devices. To design this technique in MIS, we first conduct a survey (N=23) identifying intraoperative needs, interaction strategies and workarounds, and persistent challenges. Then, through five participatory design workshops (N=10), we identify challenges in blending a user interface into views of a physical space, informing the design of InteractOR, a system that combines surgical instrument segmentation with pinch-gesture recognition to enable interaction within the surgical view. Finally, in a Comparative Structured Observation study (N=12) we compare two visualization strategies (side-by-side and overlay) against delegation, showing that interaction through instruments can reduce focus shifts, increase efficiency, and foster autonomy.
Uncertainty caused by instrument tracking errors affects critical tasks such as surgery assisted by Augmented Reality (AR) guidance. This work investigates whether visualizing such uncertainty can improve task performance, trust, and confidence. We present four visualization techniques: Cone, Circle, Gauge, and Color. A two-part study evaluated these techniques on a surgical drilling task, first with 24 non-professional participants and then with 4 professional surgeons. Results indicate that uncertainty visualization improved drilling accuracy by 24% but increased task time by 76%. It also enhanced user confidence and trust in the system, with Cone and Circle as the most preferred visualizations. Based on our findings, we discuss design recommendations for integrating uncertainty visualization into AR-based surgical systems. This work paves the way for a higher success rate in surgical procedures.
Robot-assisted surgery (RAS) has raised concerns within human-computer interaction, particularly regarding the socio-material configurations of robotic systems and their impact on surgical practices. To investigate these configurations in the context of the da Vinci robotic system, we used a research-through-design approach. Through this approach, we developed six Visual RAS (ViRAS) scenarios derived from RAS observations. These scenarios represent different configurations of interacting surgical team members and the robotic system in distinct adverse RAS events. In ViRAS-guided interviews with experienced RAS surgeons, we found that ViRAS scenarios help reflect on surgical practices and the specific material and spatial properties of RAS. Our findings indicated that the team’s cognitive engagement during surgery could be improved by providing sensory augmentation to facilitate task perception and individual skill development. Through our research, we show how ViRAS scenarios, as a tool for reflection, can reveal opportunities for designing socio-material configurations in RAS and beyond.
Clinical pre-consultation, where patients share health information prior to an appointment, offers a pathway to more patient-centered care by freeing time for meaningful patient–physician conversations. Conversational agents powered by large language models (LLMs) can automate this process to make it more scalable and consistent, but this risks producing information overload that exacerbates physicians’ workload as they spend time parsing through data. This paper examines the opportunities and challenges of using conversational agents to mediate the transfer of information between patients and physicians, with the aim of producing clinically useful, patient-driven pre-consultation summaries to capture their histories and concerns. Through sessions with both physicians and patients, we show that such a summary can increase patient confidence and sense of control over their health information while fostering a more collaborative dynamic. We conclude with design recommendations for integrating pre-consultation agents into clinical workflows.
Current hospital medication management systems generate fragmented clinical workflows,forcing clinicians to improvise repairs by creating external artifacts. We argue that instead of formalizing their workarounds into new electronic health record (EHR) features, future systems should explicitly support improvisation capabilities that help clinicians better handle unpredictable breakdowns. In a field study (144 hours) with 11 clinicians, we observed how resource constraints require local optimizations - "expert hacks'' by clinicians - at the expense of global consistency. Subsequent group interviews with 20 clinicians highlighted three distinct issues: one clinician's shortcut often becomes another clinician's roadblock; annotations fail to distinguish between norms and deviations; and clinicians often reify personal routines to translate information across systems. We argue that simply adding new EHR features will not suffice. Instead, we propose a novel design approach that focuses on helping clinicians create personal tools that let them successfully manage information breakdowns in their particular context.