Crowdsourcing & the value of discussion

Paper session

会議の名前
CHI 2020
Crowdsourcing in China: Exploring the Work Experiences of Solo Crowdworkers and Crowdfarm Workers
要旨

Recent research highlights the potential of crowdsourcing in China. Yet very few studies explore the workplace context and experiences of Chinese crowdworkers. Those that do, focus mainly on the work experiences of solo crowdworkers but do not deal with issues pertaining to the substantial amount of people working in 'crowdfarms'. This article addresses this gap as one of its primary concerns. Drawing on a study that involves 48 participants, our research explores, compares and contrasts the work experiences of solo crowdworkers to those of crowdfarm workers. Our findings illustrate that the work experiences and context of the solo workers and crowdfarm workers are substantially different, with regards to their motivations, the ways they engage with crowdsourcing, the tasks they work on, and the crowdsourcing platforms they utilize. Overall, our study contributes to furthering the understandings on the work experiences of crowdworkers in China.

キーワード
Crowdsourcing
Work Experience
Crowdworkers
Crowdfarms
Reputation management
Work Life Balance
Tasks
Platform Satisfaction
Motivations and Attitudes
著者
Yihong Wang
Xi'an Jiaotong-Liverpool University, Suzhou, China
Konstantinos Papangelis
Rochester Institute of Technology, Rochester, NY, USA
Michael Saker
City University London, London, United Kingdom
Ioanna Lykourentzou
Utrecht University, Utrecht, Netherlands
Alan Chamberlain
University of Nottingham, Nottingham, Nottinghamshire, United Kingdom
Vassilis-Javed Khan
Eindhoven University of Technology, Eindhoven, Noord Brabant, Netherlands
DOI

10.1145/3313831.3376473

論文URL

https://doi.org/10.1145/3313831.3376473

Improving Worker Engagement Through Conversational Microtask Crowdsourcing
要旨

The rise in popularity of conversational agents has enabled humans to interact with machines more naturally. Recent work has shown that crowd workers in microtask marketplaces can complete a variety of human intelligence tasks (HITs) using conversational interfaces with similar output quality compared to the traditional Web interfaces. In this paper, we investigate the effectiveness of using conversational interfaces to improve worker engagement in microtask crowdsourcing. We designed a text-based conversational agent that assists workers in task execution, and tested the performance of workers when interacting with agents having different conversational styles. We conducted a rigorous experimental study on Amazon Mechanical Turk with 800 unique workers, to explore whether the output quality, worker engagement and the perceived cognitive load of workers can be affected by the conversational agent and its conversational styles. Our results show that conversational interfaces can be effective in engaging workers, and a suitable conversational style has potential to improve worker engagement.

キーワード
Microtask Crowdsourcing
Conversational Interface
Conversational Style
User Engagement
Cognitive Task Load
著者
Sihang Qiu
Delft University of Technology, Delft, Netherlands
Ujwal Gadiraju
Leibniz Universität Hannover, Hannover, Germany
Alessandro Bozzon
Delft University of Technology, Delft, Netherlands
DOI

10.1145/3313831.3376403

論文URL

https://doi.org/10.1145/3313831.3376403

動画
Crowdsourcing the Perception of Machine Teaching
要旨

Teachable interfaces can empower end-users to attune machine learning systems to their idiosyncratic characteristics and environment by explicitly providing pertinent training examples. While facilitating control, their effectiveness can be hindered by the lack of expertise or misconceptions. We investigate how users may conceptualize, experience, and reflect on their engagement in machine teaching by deploying a mobile teachable testbed in Amazon Mechanical Turk. Using a performance-based payment scheme, Mechanical Turkers (N=100) are called to train, test, and re-train a robust recognition model in real-time with a few snapshots taken in their environment. We find that participants incorporate diversity in their examples drawing from parallels to how humans recognize objects independent of size, viewpoint, location, and illumination. Many of their misconceptions relate to consistency and model capabilities for reasoning. With limited variation and edge cases in testing, the majority of them do not change strategies on a second training attempt.

キーワード
teachable interfaces
interactive machine learning
object recognition
crowdsourcing
personalization
著者
Jonggi Hong
University of Maryland, College Park, MD, USA
Kyungjun Lee
University of Maryland, College Park, MD, USA
June Xu
University of Maryland, College Park, MD, USA
Hernisa Kacorri
University of Maryland, College Park, MD, USA
DOI

10.1145/3313831.3376428

論文URL

https://doi.org/10.1145/3313831.3376428

Peer-to-Peer Energy Markets: Understanding the Values of Collective and Community Trading
要旨

Peer-to-peer energy-trading platforms (P2P) have the potential to transform the current energy system. However, research is presently scarce on how people would like to participate in, and what would they expect to gain from, such platforms. We address this gap by exploring these questions in the context of the UK energy market. Using a qualitative interview study, we examine how 45 people with an interest in renewable energy understand P2P. We find that the prospective users value the collective benefits of P2P, and understand participation as a mechanism to support social, ecological and economic benefits for communities and larger groups. Drawing on the findings from the interview analysis, we explore broad design characteristics that a prospective P2P energy trading platform should provide to meet the expectations and concerns voiced by our study participants.

受賞
Honorable Mention
キーワード
peer to peer energy trading platforms
semi-structured interview
sustainability
thematic analysis
著者
Denise J. Wilkins
Microsoft Research & University of Exeter, Cambridge & Exeter, United Kingdom
Ruzanna Chitchyan
University of Bristol, Bristol, United Kingdom
Mark Levine
Lancaster University & University of Exeter, Lancaster & Exeter, United Kingdom
DOI

10.1145/3313831.3376135

論文URL

https://doi.org/10.1145/3313831.3376135

Expert Discussions Improve Comprehension of Difficult Cases in Medical Image Assessment
要旨

Medical data labeling workflows critically depend on accurate assessments from human experts. Yet human assessments can vary markedly, even among medical experts. Prior research has demonstrated benefits of labeler training on performance. Here we utilized two types of labeler training feedback: highlighting incorrect labels for difficult cases ("individual performance" feedback), and expert discussions from adjudication of these cases. We presented ten generalist eye care professionals with either individual performance alone, or individual performance and expert discussions from specialists. Compared to performance feedback alone, seeing expert discussions significantly improved generalists' understanding of the rationale behind the correct diagnosis while motivating changes in their own labeling approach; and also significantly improved average accuracy on one of four pathologies in a held-out test set. This work suggests that image adjudication may provide benefits beyond developing trusted consensus labels, and that exposure to specialist discussions can be an effective training intervention for medical diagnosis.

キーワード
Medical Images
Diagnosis
Adjudication
Labeler Training
著者
Mike Schaekermann
Google & University of Waterloo, Mountain View, CA, USA
Carrie J. Cai
Google, Mountain View, CA, USA
Abigail E. Huang
Google, Mountain View, CA, USA
Rory Sayres
Google, Mountain View, CA, USA
DOI

10.1145/3313831.3376290

論文URL

https://doi.org/10.1145/3313831.3376290