Users are increasingly empowered to personalize natural language interfaces (NLIs) by teaching how to handle new natural language (NL) inputs. However, our formative study found that when teaching new NL inputs, users require assistance in clarifying ambiguities that arise and want insight into which parts of the input the NLI understands. In this paper we introduce ONYX, an intelligent agent that interactively learns new NL inputs by combining NL programming and programming-by-demonstration, also known as multi-modal interactive task learning. To address the aforementioned challenges, ONYX provides suggestions on how ONYX could handle new NL inputs based on previously learned concepts or user-defined procedures, and poses follow-up questions to clarify ambiguities in user demonstrations, using visual and textual aids to clarify the connections. Our evaluation shows that users provided with ONYX’s new features achieved significantly higher accuracy in teaching new NL inputs (median: 93.3%) in contrast to those without (median: 73.3%).
https://doi.org/10.1145/3544548.3580964
The launch of open governmental data portals (OGDPs) has popularized the open data movement of last decade. Although the amount of data in OGDPs is increasing, their functionalities are limited to finding datasets with titles/descriptions and downloading the actual files. This hinders the end users, especially those without technical skills, to find the open data tables and make use of them. We present Governor, an open-sourced web application developed to make OGDPs more accessible to end users by facilitating searching actual records in the tables, previewing them directly without downloading, and suggesting joinable and unionable tables to users based on their latest working tables. Governor also manages the provenance of integrated tables allowing users and their collaborators to easily trace back to the original tables in OGDP. We evaluate Governor with a two-part user study and the results demonstrate its value and effectiveness in finding and integrating tables in OGDP.
https://doi.org/10.1145/3544548.3580868
Machine learning models with high accuracy on test data can still produce systematic failures, such as harmful biases and safety issues, when deployed in the real world. To detect and mitigate such failures, practitioners run behavioral evaluation of their models, checking model outputs for specific types of inputs. Behavioral evaluation is important but challenging, requiring that practitioners discover real-world patterns and validate systematic failures. We conducted 18 semi-structured interviews with ML practitioners to better understand the challenges of behavioral evaluation and found that it is a collaborative, use-case-first process that is not adequately supported by existing task- and domain-specific tools. Using these findings, we designed Zeno, a general-purpose framework for visualizing and testing AI systems across diverse use cases. In four case studies with participants using Zeno on real-world models, we found that practitioners were able to reproduce previous manual analyses and discover new systematic failures.
https://doi.org/10.1145/3544548.3581268
Despite decades long establishment of effective tutoring principles, no adaptive tutoring system has been developed and open-sourced to the research community. The absence of such a system inhibits researchers from replicating adaptive learning studies and extending and experimenting with various tutoring system design directions. For this reason, adaptive learning research is primarily conducted on a small number of proprietary platforms. In this work, we aim to democratize adaptive learning research with the introduction of the first open-source adaptive tutoring system based on Intelligent Tutoring System principles. The system, we call Open Adaptive Tutor (OATutor), has been iteratively developed over three years with field trials in classrooms drawing feedback from students, teachers, and researchers. The MIT-licensed source code includes three creative commons (CC BY) textbooks worth of algebra problems, with tutoring supports authored by the OATutor project. Knowledge Tracing, an A/B testing framework, and LTI support are included.
https://doi.org/10.1145/3544548.3581574
Ill-structured problems demand that people adopt sophisticated strategies for planning, seeking support, and using available resources along their work process. These practices involve a challenging monitoring and strategizing process that existing tools cannot support since they largely lack an understanding of an organization's processes, social structures, venues, and tools. We introduce workplace programming for situationally-aware systems--an approach for encoding work situations using computational abstractions of an organization's ways of working and surfacing support strategies at appropriate times and settings. With this approach, we implement Orchestration Scripts, a system that supports various situated work activities in a socio-technical organization. Through a case study and field study, we show how our approach encodes different aspects of working effectively and helps people identify situations to enact effective strategies using the available support opportunities. Our results show how a programmable technology can provide situated support in today's workplaces.
https://doi.org/10.1145/3544548.3581456
Home care workers (HCWs) deliver essential health services within patients’ homes and are an important part of the US healthcare system. Yet, they are a marginalized workforce, whose physical isolation and lack of access to support structures make them vulnerable to exploitation. Computer-mediated support programs may help bridge this gap and, through critical and liberatory pedagogies, foster material social change. However, such pedagogies typically assume the involvement of a professional facilitator when, in practice, support programs are often led by peers with little to no facilitation training. Based on a three-month study with HCWs, this paper explores how peers can perform critical and liberatory facilitation practice in an online support program. We illustrate the challenges peers faced learning this practice and performing this role in an online environment. Our findings can improve the design of computer-mediated support programs and how to prepare peer leadership, particularly for addressing the needs of marginalized populations.
https://doi.org/10.1145/3544548.3580881