Pedagogical questions are crucial for fostering student engagement and learning. In daily teaching, teachers pose hundreds of questions to assess understanding, enhance learning outcomes, and facilitate the transfer of theory-rich content. However, even experienced teachers often struggle to generate a large volume of effective pedagogical questions. To address this, we introduce TutorCraftEase, an interactive generation system that leverages large language models (LLMs) to assist teachers in creating pedagogical questions. TutorCraftEase enables the rapid generation of questions at varying difficulty levels with a single click, while also allowing for manual review and refinement. In a comparative user study with 39 participants, we evaluated TutorCraftEase against a traditional manual authoring tool and a basic LLM tool. The results show that TutorCraftEase can generate pedagogical questions comparable in quality to those created by experienced teachers, while significantly reducing their workload and time.
https://dl.acm.org/doi/10.1145/3706598.3713731
The video gaming industry offers richer experiences through increasingly complex game mechanics, often hindering learnability. This research explores integrating an LLM-based chatbot, “Daisy,” in Stardew Valley, a narrative-rich role-playing game where learnability is critical. Over three weeks, 24 participants—14 new and 10 experienced players—engaged in a diary study and post-interviews. Analysis of diaries, chat logs, gameplay videos, and interviews revealed three themes: seeking information support, playing with chatbots, and addressing practical challenges. Findings show Daisy’s potential to enhance learnability through natural conversations, fostering immersion and emotional engagement, though issues like hallucinations and context awareness require improvement. This work highlights preliminary insights for integrating LLMs into narrative-rich games.
https://dl.acm.org/doi/10.1145/3706598.3713310
Large language models (LLMs) can empower teachers to build pedagogical conversational agents (PCAs) customized for their students. As students have different prior knowledge and motivation levels, teachers must review the adaptivity of their PCAs to diverse students. Existing chatbot reviewing methods (e.g., direct chat and benchmarks) are either manually intensive for multiple iterations or limited to testing only single-turn interactions. We present TeachTune, where teachers can create simulated students and review PCAs by observing automated chats between PCAs and simulated students. Our technical pipeline instructs an LLM-based student to simulate prescribed knowledge levels and traits, helping teachers explore diverse conversation patterns. Our pipeline could produce simulated students whose behaviors correlate highly to their input knowledge and motivation levels within 5% and 10% accuracy gaps. Thirty science teachers designed PCAs in a between-subjects study, and using TeachTune resulted in a lower task load and higher student profile coverage over a baseline.
https://dl.acm.org/doi/10.1145/3706598.3714054
Despite the growing prominence of Artificial Intelligence (AI) chatbots used in education, there remains a significant gap in our understanding of how interface design elements, particularly avatar representations, influence learning experiences. This paper explores the impact of different AI chatbot avatar representations on students' learning experiences through a mixed-methods within-subjects study, where participants interacted with three distinct types of AI chatbot interfaces with a common large language model (LLM) over a 14-week university course. Our findings reveal that preferences vary according to factors such as learning habits and learning activities. Avatar design also exhibits affordances for specific prompting behaviors, while the perceived human touch influenced learning experiences in nuanced ways. Additionally, real-world relationships with the individuals behind deepfakes influence these experiences. These insights suggest that the thoughtful integration of diverse avatar representations in AI chatbot systems for different learners and settings can greatly enhance learning experiences.
https://dl.acm.org/doi/10.1145/3706598.3713456
Surveying prior literature to establish a foundation for new knowledge is essential for scholarly progress. However, survey articles are resource-intensive and challenging to create, and can quickly become outdated as new research is published, risking information staleness and inaccuracy. Keeping survey articles current with the latest evidence is therefore desirable, though there is a limited understanding of why, when, and how these surveys should be updated. Toward this end, through a series of in-depth retrospective interviews with 11 researchers, we present an empirical examination of the work practices in authoring and updating survey articles in computing research. We find that while computing researchers acknowledge the value in maintaining an updated survey, continuous updating remains unmanageable and misaligned with academic incentives. Our findings suggest key leverage points within current workflows that present opportunities for enabling technologies to facilitate more efficient and effective updates.
https://dl.acm.org/doi/10.1145/3706598.3714047