From wearable health tracking to sensor-laden cities, AI-enhanced pervasive sensing platforms promise far-reaching benefits yet also introduce societal risks. How might designers of these platforms effectively navigate their complex ecology and sociotechnical dynamics? To explore this question, we interviewed designers building mental health technologies who undertook this challenge. They are hospital chief medical information officers and startup founders together striving to create new sensors/AI platforms and integrate them into the healthcare ecosystem. We found that, while all designers aspired to build comprehensive care platforms, their efforts focused on serving either consumers or physicians, delivering a subset of healthcare interventions, and demonstrating system effectiveness one metric at a time. Consequently, breakdowns in patient journeys are emerging; societal risks loom large. We describe how the data economy, designers' mindsets, and evaluation challenges led to these unintended design consequences. We discuss implications for designing pervasive sensing and AI platforms for social good.
https://doi.org/10.1145/3613904.3642793
The ACM CHI Conference on Human Factors in Computing Systems (https://chi2024.acm.org/)