Vision-Based Multimodal Interfaces: A Survey and Taxonomy for Enhanced Context-Aware System Design

要旨

The recent surge in artificial intelligence, particularly in multimodal processing technology, has advanced human-computer interaction, by altering how intelligent systems perceive, understand, and respond to contextual information (i.e., context awareness). Despite such advancements, there is a significant gap in comprehensive reviews examining these advances, especially from a multimodal data perspective, which is crucial for refining system design. This paper addresses a key aspect of this gap by conducting a systematic survey of data modality-driven Vision-based Multimodal Interfaces (VMIs). VMIs are essential for integrating multimodal data, enabling more precise interpretation of user intentions and complex interactions across physical and digital environments. Unlike previous task- or scenario-driven surveys, this study highlights the critical role of the visual modality in processing contextual information and facilitating multimodal interaction. Adopting a design framework moving from the whole to the details and back, it classifies VMIs across dimensions, providing insights for developing effective, context-aware systems.

著者
Yongquan 'Owen' Hu
University of New South Wales, Sydney, NSW, Australia
Jingyu Tang
Huazhong University of Science and Technology, Wuhan, China
Xinya Gong
Southern University of Science and Technology, Shenzhen, China
Zhongyi Zhou
RIKEN AIP, Tokyo, Japan
Shuning Zhang
Tsinghua University, Beijing, China
Don Samitha Elvitigala
Monash University, Melbourne, Australia
Florian ‘Floyd’. Mueller
Monash University, Melbourne, VIC, Australia
Wen Hu
UNSW, Syndey, New South Wales, Australia
Aaron Quigley
CSIRO’s Data61 , Sydney, NSW, Australia
DOI

10.1145/3706598.3714161

論文URL

https://dl.acm.org/doi/10.1145/3706598.3714161

動画

会議: CHI 2025

The ACM CHI Conference on Human Factors in Computing Systems (https://chi2025.acm.org/)

セッション: Multimodal Interaction

G302
7 件の発表
2025-04-30 18:00:00
2025-04-30 19:30:00
日本語まとめ
読み込み中…