Calls for participatory AI development often assume that stakeholders can and should substantially shape a system's design. However, this agency may be constrained by competing demands, e.g. those safety-related. We explore this tension through a case study in Air Traffic Control (ATC) system development. Interviews with ATC operators and a focus group including R&D staff uncovered that operators’ input was confined to small changes, with major decisions made through opaque processes. Safety-related considerations often limited how operator input could be incorporated. Importantly, operators acknowledged that safety should take priority but called for more transparency over decision-making processes and the factors considered thereby. Our findings highlight how general calls for stakeholder empowerment can contradict safety-critical (and other) requirements. We show the importance of engaging broad perspectives to explore conflicting demands before aligning/prioritising these in the application context. We further outline implications for participatory practice relevant for responsible AI and HCI communities.
ACM CHI Conference on Human Factors in Computing Systems