Maximizing system scalability and quality are sometimes at odds. This work provides an example showing scalability and quality can be achieved at the same time in instructional design, contrary to what instructors may believe or expect. We situate our study in the education of HCI methods, and provide suggestions to improve active learning within the HCI education community. While designing learning and assessment activities, many instructors face the choice of using open-ended or close-ended activities. Close-ended activities such as multiple-choice questions (MCQs) enable automated feedback to students. However, a survey with 22 HCI professors revealed a belief that MCQs are less valuable than open-ended questions, and thus, using them entails making a quality sacrifice in order to achieve scalability. A study with 178 students produced no evidence to support the teacher belief. This paper indicates more promise than concern in using MCQs for scalable instruction and assessment in at least some HCI domains.
https://doi.org/10.1145/3411764.3445045
We designed and tested an attention-aware learning technology (AALT) that detects and responds to mind wandering (MW), a shift in attention from task-related to task-unrelated thoughts, that is negatively associated with learning. We leveraged an existing gaze-based mind wandering detector that uses commercial off the shelf eye tracking to inform real-time interventions during learning with an Intelligent Tutoring System in real-world classrooms. The intervention strategies, co-designed with students and teachers, consisted of using student names, reiterating content, and asking questions, with the aim to reengage wandering minds and improve learning. After several rounds of iterative refinement, we tested our AALT in two classroom studies with 287 high-school students. We found that interventions successfully reoriented attention and, compared to two control conditions, reduced mind wandering and improved retention (measured via a delayed assessment) for students with low prior-knowledge who occasionally (but not excessively) mind wandered. We discuss implications for developing gaze-based AALTs for real-world contexts.
https://doi.org/10.1145/3411764.3445269
Learning to recognize and apply programming patterns --- reusable abstractions of code --- is critical to becoming a proficient computer scientist. However, many introductory Computer Science courses do not teach patterns, in part because teaching these concepts requires significant curriculum changes. As an alternative, we explore how a novel user interface for practicing coding --- Faded Parsons Problems --- can support introductory Computer Science students in learning to apply programming patterns. We ran a classroom-based study with 237 students which found that Faded Parsons Problems, or rearranging and completing partially blank lines of code into a valid program, are an effective exercise interface for teaching programming patterns, significantly surpassing the performance of the more standard approaches of code writing and code tracing exercises. Faded Parsons Problems also improve overall code writing ability at a comparable level to code writing exercises, but are preferred by students.
https://doi.org/10.1145/3411764.3445228
Computational thinking (CT) education reaches only a fraction of young children, in part because CT learning tools often require expensive hardware or fluent literacy. Informed by needfinding interviews, we developed a voice-guided smartphone application leveraging storytelling as a creative activity by which to teach CT concepts to 5- to 8-year-old children. The app includes two storytelling games where users create and listen to stories as well as four CT games where users then modify those stories to learn about sequences, loops, events, and variables. We improved upon the app design through wizard-of-oz testing (N=28) and iterative design testing (N=22) before conducting an evaluation study (N=22). Children were successfully able to navigate the app, effectively learn about the target computing concepts, and, after using the app, children demonstrated above-chance performance on a near transfer CT concept recognition task.
https://doi.org/10.1145/3411764.3445039
Programming error messages play an important role in learning to program. The cycle of program input and error message response completes a loop between the programmer and the compiler/interpreter and is a fundamental interaction between human and computer. However, error messages are notoriously problematic, especially for novices. Despite numerous guidelines citing the importance of message readability, there is little empirical research dedicated to understanding and assessing it. We report three related experiments investigating factors that influence programming error message readability. In the first two experiments we identify possible factors, and in the third we ask novice programmers to rate messages using scales derived from these factors. We find evidence that several key factors significantly affect message readability: message length, jargon use, sentence structure, and vocabulary. This provides novel empirical support for previously untested long-standing guidelines on message design, and informs future efforts to create readability metrics for programming error messages.
https://doi.org/10.1145/3411764.3445696
Program tracing, or mentally simulating a program on concrete inputs, is an important part of general program comprehension. Programs involve many kinds of virtual state that must be held in memory, such as variable/value pairs and a call stack. In this work, we examine the influence of short-term working memory (WM) on a person's ability to remember program state during tracing. We first confirm that previous findings in cognitive psychology transfer to the programming domain: people can keep about 7 variable/value pairs in WM, and people will accidentally swap associations between variables due to WM load. We use a restricted focus viewing interface to further analyze the strategies people use to trace through programs, and the relationship of tracing strategy to WM. Given a straight-line program, we find half of our participants traced a program from the top-down line-by-line (linearly), and the other half start at the bottom and trace upward based on data dependencies (on-demand). Participants with an on-demand strategy made more WM errors while tracing straight-line code than with a linear strategy, but the two strategies contained an equal number of WM errors when tracing code with functions. We conclude with the implications of these findings for the design of programming tools: first, programs should be analyzed to identify and refactor human-memory-intensive sections of code. Second, programming environments should interactively visualize variable metadata to reduce WM load in accordance with a person's tracing strategy. Third, tools for program comprehension should enable externalizing program state while tracing.
https://doi.org/10.1145/3411764.3445257
HCI has historically provided little support for moving from fieldwork insights or theories to design outcomes. Having witnessed many students struggle and then justify their designs with a form of marketing hype, we developed a supporting approach of “field theories”. A field theory is a working theory about salient interactions in a particular domain and sensitizing concepts in order to frame design investigations. It is presented visually in a field theory diagram to support succinct communication and critique. Studying use of design prototypes that have been informed by a field theory helps to reflect upon and refine the theory. In this paper we present examples from our HCI classes and reflections based on interviews with students. We discuss how field theories offer an orientation in the spirit of a ‘bricoleur’ who harnesses elements of theory and practice to produce deeper understandings and more fitting outcomes for the task at hand.
https://doi.org/10.1145/3411764.3445414
We present the design and evaluation of a web-based intelligent writing assistant that helps students recognize their revisions of argumentative essays. To understand how our revision assistant can best support students, we have implemented four versions of our system with differences in the unit span (sentence versus sub-sentence) of revision analysis and the level of feedback provided (none, binary, or detailed revision purpose categorization). We first discuss the design decisions behind relevant components of the system, then analyze the efficacy of the different versions through a Wizard of Oz study with university students. Our results show that while a simple interface with no revision feedback is easier to use, an interface that provides a detailed categorization of sentence-level revisions is the most helpful based on user survey data, as well as the most effective based on improvement in writing outcomes.
https://doi.org/10.1145/3411764.3445683
Tangible interfaces have much potential for engendering shared interaction and reflection, as well as for promoting playful experiences. How can their properties be capitalised on to enable students to reflect on their learning, both individually and together, throughout learning sessions? This Research through Design paper describes our development of EvalMe, a flexible, tangible tool aimed at being playful, enjoyable to use and enabling children to reflect on their learning, both in the moment and after a learning session has ended. We discuss the insights gained through the process of designing EvalMe, co-defining its functionality with two groups of collaborators and deploying it in two workshop settings. Through this process, we map key contextual considerations for the design of technologies for in situ evaluation of learning experiences. Finally, we discuss how tangible evaluation technologies deployed throughout a learning session, can positively contribute to students’ reflection about their learning.
https://doi.org/10.1145/3411764.3445749
Novice programmers need differentiated assessments (such as adaptive Parsons problems) to maximize their ability to learn how to program. Parsons problems require learners to place mixed-up code blocks in the correct order to solve a problem. We conducted a within-subjects experiment to compare the efficiency and cognitive load of solving adaptive Parsons problems versus writing the equivalent (isomorphic) code. Undergraduates were usually more significantly efficient at solving a Parsons problem than writing the equivalent code, but not when the solution to the Parsons problem was unusual. This has implications for problem creators. This paper also reports on the mean cognitive load ratings of the two problem types and the relationship between efficiency and cognitive load ratings. Lastly, it reports on think-aloud observations of 11 students solving both adaptive Parsons problems and write-code problems and the results from an end-of-course student survey.
https://doi.org/10.1145/3411764.3445292
Conversational programmers want to learn about code primarily to communicate with technical co-workers, not to develop software. However, existing instructional materials don't meet the needs of conversational programmers because they prioritize syntax and semantics over concepts and applications. This mismatch results in feelings of failure and low self-efficacy. To motivate conversational programmers, we propose purpose-first programming, a new approach that focuses on learning a handful of domain-specific code patterns and assembling them to create authentic and useful programs. We report on the development of a purpose-first programming prototype that teaches five patterns in the domain of web scraping. We show that learning with purpose-first programming is motivating for conversational programmers because it engenders a feeling of success and aligns with these learners' goals. Purpose-first programming learning enabled novice conversational programmers to complete scaffolded code writing, debugging, and explaining activities after only 30 minutes of instruction.
https://doi.org/10.1145/3411764.3445571