In response to calls for open data and growing privacy threats, organizations are increasingly adopting privacy-preserving techniques that add noise to published datasets. These techniques seek to protect privacy of data subjects while enabling useful analyses. With expert feedback, we developed empirically-driven documentation explaining the noise characteristics of two Wikipedia pageview datasets: one using rounding (heuristic privacy) and another using differential privacy (DP, formal privacy). We then used these documents to conduct a task-based contextual inquiry (n=15) exploring how data users—largely unfamiliar with these methods—perceive, interact with, and interpret privacy-preserving noise during data analysis. Participants readily used simple uncertainty metrics from the documentation, but struggled when computing confidence intervals across multiple noisy estimates. They better devised simulation-based approaches for computing uncertainty with DP-noised vs. rounded data. Surprisingly, several participants incorrectly believed DP's stronger utility implied weaker privacy protections. We offer design recommendations for documentation and tools to better support data users working with privacy-noised data.
ACM CHI Conference on Human Factors in Computing Systems