English as a Second Language (ESL) learners often encounter unknown words that hinder their text comprehension. Automatically detecting these words as users read can enable computing systems to provide just-in-time definitions, synonyms, or contextual explanations, thereby helping users learn vocabulary in a natural and seamless manner. This paper presents EyeLingo, a transformer-based machine learning method that predicts the probability of unknown words based on text content and eye gaze trajectory in real time with high accuracy. A 20-participant user study revealed that our method can achieve an accuracy of 97.6%, and an F1-score of 71.1%. We implemented a real-time reading assistance prototype to show the effectiveness of EyeLingo. The user study shows improvement in willingness to use and usefulness compared to baseline methods.
https://dl.acm.org/doi/10.1145/3706598.3714181
While voice input offers a convenient alternative to traditional text editing on mobile devices, practical implementations face two key challenges: 1) reliably distinguishing between editing commands and content dictation, and 2) effortlessly pinpointing the intended edit location. We propose Tap&Say, a novel multimodal system that combines touch interactions with Large Language Models (LLMs) for accurate text correction. By tapping near an error, users signal their edit intent and location, addressing both challenges. Then, the user speaks the correction text. Tap&Say utilizes the touch location, voice input, and existing text to generate contextually relevant correction suggestions. We propose a novel touch location-informed attention layer that integrates the tap location into the LLM's attention mechanism, enabling it to utilize the tap location for text correction. We fine-tuned the touch location-informed LLM on synthetic touch locations and correction commands, achieving significantly higher correction accuracy than the state-of-the-art method VT. A 16-person user study demonstrated that Tap&Say outperforms VT with 16.4% shorter task completion time and 47.5% fewer keyboard clicks and is preferred by users.
https://dl.acm.org/doi/10.1145/3706598.3713376
Measuring preverbal vocabulary comprehension of young children is vital for early intervention and developmental evaluation, yet challenging due to their limited communication abilities. We introduce Lookee, an AI-powered vocabulary comprehension assessment tool through gaze tracking for toddlers in the preverbal stage. Lookee incorporates the Intermodal Preferential Looking Paradigm (IPLP), which is one of the prominent word comprehension measures for toddlers and estimates word comprehension through a random forest model analysis. We design and validate Lookee through user studies involving 19 toddlers and their parents. Then we identify necessary design requirements from potential stakeholders' perspectives through in-depth interviews including researchers, clinicians, and parents. As a result, Lookee achieves considerable estimation accuracy with sufficient system usability, and demonstrates key design requirements for each stakeholder group. From our study, we highlight necessary design implications in developing and validating AI-powered clinical tools for toddlers.
https://dl.acm.org/doi/10.1145/3706598.3713386
With the growing number of immigrants globally, language barriers have become a significant challenge, particularly for those entering English-speaking countries. Traditional language learning methods often fail to provide sufficient practical opportunities, especially for diverse accents. To address this, we introduce Language Urban Odyssey (LUO), a serious game that leverages large language models (LLMs) and game-based learning to offer a low-cost, accessible virtual environment for English learners. Built on the Minecraft platform, LUO offers real-time speech interaction with NPCs of various accents, supported by multi-modal feedback. A controlled study (N=30) showed improvements in speaking abilities, accent comprehension, and emotional confidence. Our findings suggest that LUO provides a scalable, immersive platform that bridges gaps in language learning for immigrants facing cultural and social challenges.
https://dl.acm.org/doi/10.1145/3706598.3713945
Teletherapy for speech-language therapy (SLT) has become essential for many families. Early intervention for young children is important to ensure that developmental milestones are met. In this study, from a corpus of 10 videos, we present three cases of online and in-person therapy sessions with children between the ages of 3 and 6. Our analysis shows how online and in-person SLT sessions use tools, how they are conscripted into social and transactional moments, and identifies features of tools that support or hinder therapists’ goals (see Figure 1). From our findings, we discuss in detail four overarching features of tools and implications for design. These features support engagement, space usage, child-centred play, and adaptability in therapy sessions. The paper outlines how these features are present in the tools used in SLT, and describes how they impact SLT activities, therapists’ and children’s goals, and the environment for social transactional activities.
https://dl.acm.org/doi/10.1145/3706598.3713394
Block-building activities are crucial for developing children's spatial reasoning and mathematical skills, yet parents often lack the expertise to guide these activities effectively. BrickSmart, a pioneering system, addresses this gap by providing spatial language guidance through a structured three-step process: Discovery & Design, Build & Learn, and Explore & Expand. This system uniquely supports parents in 1) generating personalized block-building instructions, 2) guiding parents to teach spatial language during building and interactive play, and 3) tracking children's learning progress, altogether enhancing children's engagement and cognitive development. In a comparative study involving 12 parent-child pairs children aged 6-8 years) for both experimental and control groups, BrickSmart demonstrated improvements in supportiveness, efficiency, and innovation, with a significant increase in children's use of spatial vocabularies during block play, thereby offering an effective framework for fostering spatial language skills in children.
https://dl.acm.org/doi/10.1145/3706598.3714212