Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study


Collecting data is one of the bottlenecks of Human-Computer Interaction (HCI) research. Motivated by this, we explore the potential of large language models (LLMs) in generating synthetic user research data. We use OpenAI’s GPT-3 model to generate open-ended questionnaire responses about experiencing video games as art, a topic not tractable with traditional computational user models. We test whether synthetic responses can be distinguished from real responses, analyze errors of synthetic data, and investigate content similarities between synthetic and real data. We conclude that GPT-3 can, in this context, yield believable accounts of HCI experiences. Given the low cost and high speed of LLM data generation, synthetic data should be useful in ideating and piloting new experiments, although any findings must obviously always be validated with real data. The results also raise concerns: if employed by malicious users of crowdsourcing services, LLMs may make crowdsourcing of self-report data fundamentally unreliable.

Best Paper
Perttu Hämäläinen
Aalto University, Espoo, Finland
Mikke Tavast
Aalto University, Espoo, Finland
Anton Kunnari
University of Helsinki, Helsinki, Finland



会議: CHI 2023

The ACM CHI Conference on Human Factors in Computing Systems (https://chi2023.acm.org/)

セッション: Large Language Models

Hall C
6 件の発表
2023-04-25 23:30:00
2023-04-26 00:55:00