Knowledge Work

会議の名前
CHI 2025
Talk to the Hand: an LLM-powered Chatbot with Visual Pointer as Proactive Companion for On-Screen Tasks
要旨

This paper presents Pointer Assistant, a novel human-AI interaction technique for on-screen tasks. The design features a chatbot displayed as an extra mouse pointer, alongside the user's, which proactively gives feedback on user actions while directing them to relevant areas on the screen and responding to the user's direct chat messages. The effectiveness of the design's key characteristics, pointer form and proactivity, was investigated in a study involving 220 participants in a financial budget planning task. Results demonstrated that the pointer design and interaction reduced task load while improving satisfaction with the experience, and increased the number of budget categories ideated during the task compared to the traditional passive chat log design. Participants viewed Pointer Assistant as a fun, innovative, and helpful visual guide while noting that its assertiveness can be improved. Future developments could offer even further enhancements to the user experience of human-AI collaboration and task outcomes.

著者
Thanawit Prasongpongchai
KASIKORN Business-Technology Group, Nonthaburi, Thailand
Pat Pataranutaporn
Massachusetts Institute of Technology, Cambridge, Massachusetts, United States
Monchai Lertsutthiwong
KASIKORN Business-Technology Group, Nonthaburi, Thailand
Pattie Maes
Massachusetts Institute of Technology, Cambridge, Massachusetts, United States
DOI

10.1145/3706598.3715579

論文URL

https://dl.acm.org/doi/10.1145/3706598.3715579

動画
OmniQuery: Contextually Augmenting Captured Multimodal Memories to Enable Personal Question Answering
要旨

People often capture memories through photos, screenshots, and videos. While existing AI-based tools enable querying this data using natural language, they only support retrieving individual pieces of information like certain objects in photos, and struggle with answering more complex queries that involve interpreting interconnected memories like sequential events. We conducted a one-month diary study to collect realistic user queries and generated a taxonomy of necessary contextual information for integrating with captured memories. We then introduce OmniQuery, a novel system that is able to answer complex personal memory-related questions that require extracting and inferring contextual information. OmniQuery augments individual captured memories through integrating scattered contextual information from multiple interconnected memories. Given a question, OmniQuery retrieves relevant augmented memories and uses a large language model (LLM) to generate answers with references. In human evaluations, we show the effectiveness of OmniQuery with an accuracy of 71.5%, outperforming a conventional RAG system by winning or tying for 74.5% of the time.

著者
Jiahao Nick. Li
UCLA, Los Angeles, California, United States
Zhuohao (Jerry) Zhang
University of Washington, Seattle, Washington, United States
Jiaju Ma
Stanford University, Stanford, California, United States
DOI

10.1145/3706598.3713448

論文URL

https://dl.acm.org/doi/10.1145/3706598.3713448

動画
AiGet: Transforming Everyday Moments into Hidden Knowledge Discovery with AI Assistance on Smart Glasses
要旨

Unlike the free exploration of childhood, the demands of daily life reduce our motivation to explore our surroundings, leading to missed opportunities for informal learning. Traditional tools for knowledge acquisition are reactive, relying on user initiative and limiting their ability to uncover hidden interests. Through formative studies, we introduce AiGet, a proactive AI assistant integrated with AR smart glasses, designed to seamlessly embed informal learning into low-demand daily activities (e.g., casual walking and shopping). AiGet analyzes real-time user gaze patterns, environmental context, and user profiles, leveraging large language models to deliver personalized, context-aware knowledge with low disruption to primary tasks. In-lab evaluations and real-world testing, including continued use over multiple days, demonstrate AiGet’s effectiveness in uncovering overlooked yet surprising interests, enhancing primary task enjoyment, reviving curiosity, and deepening connections with the environment. We further propose design guidelines for AI-assisted informal learning, focused on transforming everyday moments into enriching learning experiences.

著者
Runze Cai
National University of Singapore, Singapore, Singapore
Nuwan Janaka
National University of Singapore, Singapore, Singapore
Hyeongcheol Kim
National University of Singapore, Singapore , Singapore
Yang Chen
National University of Singapore, Singapore, Singapore
Shengdong Zhao
City University of Hong Kong, Hong Kong, China
Yun Huang
University of Illinois at Urbana-Champaign, Champaign, Illinois, United States
David Hsu
National University of Singapore, Singapore, Singapore
DOI

10.1145/3706598.3713953

論文URL

https://dl.acm.org/doi/10.1145/3706598.3713953

動画
CoKnowledge: Supporting Assimilation of Time-synced Collective Knowledge in Online Science Videos
要旨

Danmaku, a system of scene-aligned, time-synced, floating comments, can augment video content to create `collective knowledge'. However, its chaotic nature often hinders viewers from effectively assimilating the collective knowledge, especially in knowledge-intensive science videos. With a formative study, we examined viewers' practices for processing collective knowledge and the specific barriers they encountered. Building on these insights, we designed a processing pipeline to filter, classify, and cluster danmaku, leading to the development of CoKnowledge -- a tool incorporating a video abstract, knowledge graphs, and supplementary danmaku features to support viewers' assimilation of collective knowledge in science videos. A within-subject study (N=24) showed that CoKnowledge significantly enhanced participants’ comprehension and recall of collective knowledge compared to a baseline with unprocessed live comments. Based on our analysis of user interaction patterns and feedback on design features, we presented design considerations for developing similar support tools.

著者
Yuanhao Zhang
Hong Kong University of Science and Technology, Hong Kong, China
Yumeng Wang
the Hong Kong University of Science and Technology, Hong Kong, Hong Kong
Xiyuan Wang
ShanghaiTech University, Shanghai, China
Changyang He
Max Planck Institute for Security and Privacy, Bochum, Germany
Chenliang Huang
New York University, Brooklyn, New York, United States
Xiaojuan Ma
Hong Kong University of Science and Technology, Hong Kong, Hong Kong
DOI

10.1145/3706598.3713682

論文URL

https://dl.acm.org/doi/10.1145/3706598.3713682

動画
Generative AI in Knowledge Work: Design Implications for Data Navigation and Decision-Making
要旨

Our study of 20 knowledge workers revealed a common challenge: the difficulty of synthesizing unstructured information scattered across multiple platforms to make informed decisions. Drawing on their vision of an ideal knowledge synthesis tool, we developed Yodeai, an AI-enabled system, to explore both the opportunities and limitations of AI in knowledge work. Through a user study with 16 product managers, we identified three key requirements for Generative AI in knowledge work: adaptable user control, transparent collaboration mechanisms, and the ability to integrate background knowledge with external information. However, we also found significant limitations, including overreliance on AI, user isolation, and contextual factors outside the AI's reach. As AI tools become increasingly prevalent in professional settings, we propose design principles that emphasize adaptability to diverse workflows, accountability in personal and collaborative contexts, and context-aware interoperability to guide the development of human-centered AI systems for product managers and knowledge workers.

受賞
Honorable Mention
著者
Bhada Yun
University of California, Berkeley, Berkeley, California, United States
Dana Feng
University of California, Berkeley, Berkeley, California, United States
Ace S.. Chen
University of California Berkeley, Berkeley, California, United States
Afshin Nikzad
University of Southern California, Los Angeles, California, United States
Niloufar Salehi
UC, Berkeley, Berkeley, California, United States
DOI

10.1145/3706598.3713337

論文URL

https://dl.acm.org/doi/10.1145/3706598.3713337

動画
PlanTogether: Facilitating AI Application Planning Using Information Graphs and Large Language Models
要旨

In client-AI expert collaborations, the planning stage of AI application development begins from the client; a client outlines their needs and expectations while assessing available resources (pre-collaboration planning). Despite the importance of pre-collaboration plans for discussions with AI experts for iteration and development, the client often fails to reflect their needs and expectations into a concrete actionable plan. To facilitate pre-collaboration planning, we introduce PlanTogether, a system that generates tailored client support using large language models and a Planning Information Graph, whose nodes and edges represent information in the plan and the information dependencies. Using the graph, the system links and presents information that guides client's reasoning; it provides tips and suggestions based on relevant information and displays an overview to help understand the progression through the plan. A user study validates the effectiveness of PlanTogether in helping clients navigate information dependencies and write actionable plans reflecting their domain expertise.

著者
Dae Hyun Kim
Yonsei University, Seoul, Korea, Republic of
Daeheon Jeong
KAIST, Daejeon, Korea, Republic of
Shakhnozakhon Yadgarova
KAIST, Daejeon, Korea, Republic of
Hyungyu Shin
KAIST, Daejeon, Korea, Republic of
Jinho Son
Algorithm Labs, Seoul, Korea, Republic of
Hariharan Subramonyam
Stanford University, Stanford, California, United States
Juho Kim
KAIST, Daejeon, Korea, Republic of
DOI

10.1145/3706598.3714044

論文URL

https://dl.acm.org/doi/10.1145/3706598.3714044

動画