2. Vision-based UIs

会議の名前
UIST 2024
Vision-Based Hand Gesture Customization from a Single Demonstration
要旨

Hand gesture recognition is becoming a more prevalent mode of human-computer interaction, especially as cameras proliferate across everyday devices. Despite continued progress in this field, gesture customization is often underexplored. Customization is crucial since it enables users to define and demonstrate gestures that are more natural, memorable, and accessible. However, customization requires efficient usage of user-provided data. We introduce a method that enables users to easily design bespoke gestures with a monocular camera from one demonstration. We employ transformers and meta-learning techniques to address few-shot learning challenges. Unlike prior work, our method supports any combination of one-handed, two-handed, static, and dynamic gestures, including different viewpoints, and the ability to handle irrelevant hand movements. We implement three real-world applications using our customization method, conduct a user study, and achieve up to 94\% average recognition accuracy from one demonstration. Our work provides a viable path for vision-based gesture customization, laying the foundation for future advancements in this domain.

著者
Soroush Shahi
Apple Inc., Cupertino, California, United States
Vimal Mollyn
Apple Inc., Cupertino, California, United States
Cori Tymoszek Park
Apple Inc., Cupertino, California, United States
Runchang Kang
Apple lnc., Seattle, Washington, United States
Asaf Liberman
Apple Inc., Cupertino, California, United States
Oron Levy
Apple Inc., Cupertino, California, United States
Jun Gong
Apple Inc., Cupertino, California, United States
Abdelkareem Bedri
Apple Inc., Cupertino, California, United States
Gierad Laput
Apple Inc., Cupertino , California, United States
論文URL

https://doi.org/10.1145/3654777.3676378

動画
VirtualNexus: Enhancing 360-Degree Video AR/VR Collaboration with Environment Cutouts and Virtual Replicas
要旨

Asymmetric AR/VR collaboration systems bring a remote VR user to a local AR user’s physical environment, allowing them to communicate and work within a shared virtual/physical space. Such systems often display the remote environment through 3D reconstructions or 360° videos. While 360° cameras stream an environment in higher quality, they lack spatial information, making them less interactable. We present VirtualNexus, an AR/VR collaboration system that enhances 360° video AR/VR collaboration with environment cutouts and virtual replicas. VR users can define cutouts of the remote environment to interact with as a world-in-miniature, and their interactions are synchronized to the local AR perspective. Furthermore, AR users can rapidly scan and share 3D virtual replicas of physical objects using neural rendering. We demonstrated our system’s utility through 3 example applications and evaluated our system in a dyadic usability test. VirtualNexus extends the interaction space of 360° telepresence systems, offering improved physical presence, versatility, and clarity in interactions.

著者
Xincheng Huang
University of British Columbia, Vancouver, British Columbia, Canada
Michael Yin
University of British Columbia, Vancouver, British Columbia, Canada
Ziyi Xia
University of British Columbia, Vancouver, British Columbia, Canada
Robert Xiao
University of British Columbia, Vancouver, British Columbia, Canada
論文URL

https://doi.org/10.1145/3654777.3676377

動画
Personal Time-Lapse
要旨

Our bodies are constantly in motion—from the bending of arms and legs to the less conscious movement of breathing, our precise shape and location change constantly. This can make subtler developments (e.g., the growth of hair, or the healing of a wound) difficult to observe. Our work focuses on helping users record and visualize this type of subtle, longer-term change. We present a mobile tool that combines custom 3D tracking with interactive visual feedback and computational imaging to capture personal time-lapse, which approximates longer-term video of the subject (typically, part of the capturing user’s body) under a fixed viewpoint, body pose, and lighting condition. These personal time-lapses offer a powerful and detailed way to track visual changes of the subject over time. We begin with a formative study that examines what makes personal time-lapse so difficult to capture. Building on our findings, we motivate the design of our capture tool, evaluate this design with users, and demonstrate its effectiveness in a variety of challenging examples.

著者
Nhan Tran
Cornell University, Ithaca, New York, United States
Ethan Yang
Cornell University, Ithaca, New York, United States
Angelique Taylor
Cornell University, New York City, New York, United States
Abe Davis
Cornell Tech, Cornell University, New York, New York, United States
論文URL

https://doi.org/10.1145/3654777.3676383

動画
Chromaticity Gradient Mapping for Interactive Control of Color Contrast in Images and Video
要旨

We present a novel perceptually-motivated interactive tool for using color contrast to enhance details represented in the lightness channel of images and video. Our method lets users adjust the perceived contrast of different details by manipulating local chromaticity while preserving the original lightness of individual pixels. Inspired by the use of similar chromaticity mappings in painting, our tool effectively offers contrast along a user-selected gradient of chromaticities as additional bandwidth for representing and enhancing different details in an image. We provide an interface for our tool that closely resembles the familiar design of tonal contrast curve controls that are available in most professional image editing software. We show that our tool is effective for enhancing the perceived contrast of details without altering lightness in an image and present many examples of effects that can be achieved with our method on both images and video.

著者
Ruyu Yan
Princeton University, Princeton, New Jersey, United States
Jiatian Sun
Cornell University, Ithaca, New York, United States
Abe Davis
Cornell University, Ithaca, New York, United States
論文URL

https://doi.org/10.1145/3654777.3676340

動画