Mobile Augmented Reality (AR) offers a powerful way to provide spatially-aware guidance for real-world applications. In many cases, these applications involve the configuration of a camera or articulated subject, asking users to navigate several spatial degrees of freedom (DOF) at once. Most guidance for such tasks relies on decomposing available DOF into subspaces that can be more easily mapped to simple 1D or 2D visualizations. Unfortunately, different factorizations of the same motion often map to very different visual feedback, and finding the factorization that best matches a user’s intuition can be difficult. We propose an interactive approach that infers rotational degrees of freedom from short user demonstrations. Users select one or two DOFs at a time by demonstrating a small range of motion, which we use to learn a rotational frame that best aligns with user control of the object. We show that deriving visual feedback from this inferred learned rotational frame leads to improved task completion times on 6DOF guidance tasks compared to standard default reference frames used in most mixed reality applications.
Conceptual design is an important stage in industrial product development, influenced by the design space and materials available to designers. Advancements in human-computer interaction (HCI) and artificial intelligence (AI) technologies have broadened these aspects considerably. On the one hand, augmented reality (AR) technologies merge physical and virtual representations to enhance intuitive interaction and embodied cognition. On the other hand, generative artificial intelligence (GAI) serves as a novel design material, boosting creativity and productivity. Inspired by these technological strides, we proposed an Intelli-Embodied Design Space (IEDS), which integrates designers, AR, and GAI to support industrial conceptual design by combining embodied interaction with generative variability. Within IEDS, designers can interact with the physical prototypes intuitively, while GAI refines these into virtual forms that can be embedded in the physical world through AR technology. In this study, we established the theoretical framework and interaction modes of IEDS through literature reviews and expert interviews. Subsequently, we designed and implemented three GAI+AR tools, GAI + Head-mounted Display (HMD), GAI + Handheld Display (HHD), and GAI + Spatial Augmented Reality (SAR), based on three AR approaches in IEDS to practically examine the benefits and challenges of these interaction modes across industrial conceptual design tasks. We discussed IEDS's influence on industrial conceptual design and released its application guidelines to the HCI community.
Context-aware AR instruction enables adaptive and in-situ learning experiences. However, hardware limitations and expertise requirements constrain the creation of such instructions. With recent developments in Generative Artificial Intelligence (Gen-AI), current research tries to tackle these constraints by deploying AI-generated content (AIGC) in AR applications. However, our preliminary study with six AR practitioners revealed that the current AIGC lacks contextual information to adapt to varying application scenarios and is therefore limited in authoring. To utilize the strong generative power of GenAI to ease the authoring of AR instruction while capturing the context, we developed CARING-AI, an AR system to author context-aware humanoid-avatar-based instructions with GenAI. By navigating in the environment, users naturally provide contextual information to generate humanoid-avatar animation as AR instructions that blend in the context spatially and temporally. We showcased three application scenarios of CARING-AI: Asynchronous Instructions, Remote Instructions, and Ad Hoc Instructions based on a design space of AIGC in AR Instructions. With two user studies (N=12), we assessed the system usability of CARING-AI and demonstrated the easiness and effectiveness of authoring with Gen-AI.
Smartphones are integral to modern life, yet research highlights the cognitive drawbacks associated with their mere presence. While physically removing them can mitigate these effects, it is often inconvenient and may heighten anxiety due to prolonged separation. To address this, we use holographic augmented reality (AR) displays to visually diminish distractions with two interventions: 1) Visual Camouflage, which disguises the smartphone with a hologram that matches its size and blends with the background, making it less noticeable, and 2) Visual Substitution, which occludes the smartphone with a contextually relevant hologram, like books on a desk. In a study with 60 participants, we compared cognitive performance with the smartphone nearby, remote, and visually diminished by our AR interventions. Our findings show that the interventions significantly reduce cognitive impairment, with effects comparable to physically removing the smartphone. The adaptability of our approach opens new avenues to manage visual distractions in daily life.
Emerging AR applications require seamless integration of the virtual and physical worlds, which calls for tools that support both passive perception and active manipulation of the environment, enabling bidirectional interaction. We introduce EchoSight, a system for AR glasses that enables efficient look-and-control bidirectional interaction. EchoSight exploits optical wireless communication to instantaneously connect virtual data with its physical counterpart. EchoSight's unique dual-element optical design leverages beam directionality to automatically align the user's focus with target objects, reducing the overhead in both target identification and subsequent communication. This approach streamlines user interaction, reducing cognitive load and enhancing engagement. Our evaluations demonstrate EchoSight's effectiveness for room-scale communication, achieving distances up to 5 m and viewing angles up to 120 degrees. A study with 12 participants confirms EchoSight's improved efficiency and user experience over traditional methods, such as QR Code scanning and voice control, in AR IoT applications.
We present PerspectAR, a novel system addressing perspective distortion on displays caused by large size and wide viewing angles. PerspectAR has three components: a virtual AR screen that curves dynamically according to a user's position relative to the display, a sliding transparent window giving unobstructed access to the physical display in front of the user, and gaze indicators to assist collaborators when they are looking at different renderings. In a within-subjects study in a semi-controlled public environment with 12 pairs, we compared physical display-only and PerspectAR configurations for data analysis tasks. Participants reported less physical workload with PerspectAR and spent more time near the physical display without compromising task performance. Feedback indicates that PerspectAR addressed perspective distortion and provided a contextual view that was useful as a memory aid. Due to the virtual screen curvature, PerspectAR was seen as less effective for tasks involving distance estimates between objects.
Note-taking is critical during speeches and discussions, serving for later summarization and organization and for real-time question and opinion reminding in question-and-answer sessions or timely contributions in discussions. Manually typing on smartphones for note-taking could be distracting and increase cognitive load. While LLMs are used to automatically generate summaries and highlights, the content generated by AI may not match users’ intentions without user input. Therefore, we propose an AI-copiloted AR system, GazeNoter, to allow users to swiftly select diverse LLM-generated suggestions via gaze on an AR headset for real-time note-taking. GazeNoter leverages an AR headset as a medium for users to swiftly adjust the LLM output to match their intentions, forming a user-in-the-loop AI system for both within-context and beyond-context notes. We conducted two studies to verify the usability of GazeNoter in attending speeches in a static sitting condition and walking meetings and discussions in a mobile walking condition, respectively.