DroidRetriever: A Transparent and Steerable Automation System for Collaborative Mobile Information Seeking

要旨

Information seeking on mobile devices is often fragmented, trapping users in repetitive cycles of context switching and data re-entry, which increases cognitive load and disrupts workflow. Existing mobile agents provide limited cross-source integration and are largely opaque, presenting progress as a linear feed with few opportunities to intervene, steer, or take control. We present DroidRetriever, a transparent, steerable system for cross-source mobile information seeking. It accepts voice or typed input and the multi-LLM system decomposes the task, navigates to target pages, takes screenshots, and synthesizes a concise report with citation-linked screenshots. We make the process transparent through a progress dashboard combining sub-task progress and real-time exploration maps for seamless takeover. DroidRetriever also pauses on detected privacy or high-risk screens and prompts intervention. Across 35 tasks over 24 apps, experiments and user studies demonstrate improvements in coverage, transparency, and reduced workload. We release our code at https://github.com/AkimotoAyako/DroidRetriever.

著者
Yiheng Bian
Xi'an Jiaotong University, Xi'an, China
Yunpeng Song
Xi'an Jiaotong University, Xi'an, China
Guiyu Ma
Xi'an Jiaotong University, Xi’an, China
Rongrong Zhu
Xi'an Jiaotong University, Xi'an, Shaanxi, China
Zhongmin Cai
Xi'an Jiaotong University, Xi'an, China

会議: CHI 2026

ACM CHI Conference on Human Factors in Computing Systems

セッション: Human Steering and Interaction with AI

P1 - Room 111
7 件の発表
2026-04-16 20:15:00
2026-04-16 21:45:00