ReactGenie: A Development Framework for Complex Multimodal Interactions Using Large Language Models

要旨

By combining voice and touch interactions, multimodal interfaces can surpass the efficiency of either modality alone. Traditional multimodal frameworks require laborious developer work to support rich multimodal commands where the user’s multimodal command involves possibly exponential combinations of actions/function invocations. This paper presents ReactGenie, a programming framework that better separates multimodal input from the computational model to enable developers to create efficient and capable multimodal interfaces with ease. ReactGenie translates multimodal user commands into NLPL (Natural Language Programming Language), a programming language we created, using a neural semantic parser based on large-language models. The ReactGenie runtime interprets the parsed NLPL and composes primitives in the computational model to implement complex user commands. As a result, ReactGenie allows easy implementation and unprecedented richness in commands for end-users of multimodal apps. Our evaluation showed that 12 developers can learn and build a non-trivial ReactGenie application in under 2.5 hours on average. In addition, compared with a traditional GUI, end-users can complete tasks faster and with less task load using ReactGenie apps.

著者
Jackie (Junrui). Yang
Stanford University, Stanford, California, United States
Karina Li
Computer Science, Stanford, California, United States
Daniel Wan Rosli
Computer Science, Stanford, California, United States
Shuning Zhang
Tsinghua University, Beijing, China
Yuhan Zhang
Electrical Engineering, Stanford, California, United States
Yingtian Shi
Tsinghua University, Beijing, China
Anisha Jain
Independent Researcher, Stanford, California, United States
Tianshi Li
Northeastern University, Boston, Massachusetts, United States
Monica Lam
Stanford University, Stanford, California, United States
James A.. Landay
Stanford University, Stanford, California, United States
論文URL

doi.org/10.1145/3613904.3642517

動画

会議: CHI 2024

The ACM CHI Conference on Human Factors in Computing Systems (https://chi2024.acm.org/)

セッション: Emotions and User Experience

324
5 件の発表
2024-05-16 18:00:00
2024-05-16 19:20:00