Social media sites use content moderation to attempt to cultivate safe spaces with accurate information for their users. However, content moderation decisions may not be applied equally for all types of users, and may lead to disproportionate censorship related to people's genders, races, or political orientations. We conducted a mixed methods study involving qualitative and quantitative analysis of survey data to understand which types of social media users have content and accounts removed more frequently than others, what types of content and accounts are removed, and how content removed may differ between groups. We found that three groups of social media users in our dataset experienced content and account removals more often than others: political conservatives, transgender people, and Black people. However, the types of content removed from each group varied substantially. Conservative participants' removed content included content that was offensive or allegedly so, misinformation, Covid-related, adult, or hate speech. Transgender participants' content was often removed as adult despite following site guidelines, critical of a dominant group (e.g., men, white people), or specifically related to transgender or queer issues. Black participants' removed content was frequently related to racial justice or racism. More broadly, conservative participants' removals often involved harmful content removed according to site guidelines to create safe spaces with accurate information, while transgender and Black participants' removals often involved content related to expressing their marginalized identities that was removed despite following site policies or fell into content moderation gray areas. We discuss potential ways forward to make content moderation more equitable for marginalized social media users, such as embracing and designing specifically for content moderation gray areas.
Content moderation is an essential part of online community health and governance. While much of extant research is centered on what happens to the content, moderation also involves the management of violators. This study focuses on how moderators (mods) make decisions about their actions after the violation takes place but before the sanction by examining how they ``profile'' the violators. Through observations and interviews with volunteer mods on Twitch, we found that mods engage in a complex process of collaborative evidence collection and profile violators into different categories to decide the type and extent of punishment. Mods consider violators' characteristics as well as behavioral history and violation context before taking moderation action. The main purpose of the profiling was to avoid excessive punishment and aim to integrate violators more into the community. We discuss the contributions of profiling to moderation practice and suggest design mechanisms to facilitate mods' profiling processes.
Governance in online communities is an increasingly high-stakes challenge, and yet many basic features of offline governance legacies—juries, political parties, term limits, and formal debates, to name a few—are not in the feature-sets of the software most community platforms use. Drawing on the paradigm of Institutional Analysis and Development, this paper proposes a strategy for addressing this lapse by specifying basic features of a generalizable paradigm for online governance called Modular Politics. Whereas classical governance typologies tend to present a choice among wholesale ideologies, such as democracy or oligarchy, Modular Politics would enable platform operators and their users to build bottom-up governance processes from computational components that are modular and composable, highly versatile in their expressiveness, portable from one context to another, and interoperable across platforms. This kind of approach could implement pre-digital governance systems as well as accelerate innovation in uniquely digital techniques. As diverse communities share and connect their components and data, governance could occur through a ubiquitous network layer. To that end, this paper proposes the development of an open standard for networked governance.
https://doi.org/10.1145/3449090
Content moderation systems for social media have had numerous issues of bias, in terms of race, gender, and ability among many others. One proposal for addressing such issues in automated decision making is by designing for contestability, whereby users can shape and influence how decisions are made. In this study, we conduct a series of participatory design workshops with participants from communities that have experienced problems with social media content moderation in the past. Together with participants, we explore the idea of designing for contestability in content moderation and find that users' designs suggest three fruitful, practical avenues: adding representation, improving communication, and designing with compassion. We conclude with design recommendations drawn from participants' proposals, and reflect on the challenges that remain.
https://doi.org/10.1145/3476059
To manage user-generated harmful video content, YouTube relies on AI algorithms (e.g., machine learning) in content moderation and follows a retributive justice logic to punish convicted YouTubers through demonetization, a penalty that limits or deprives them of advertisements (ads), reducing their future ad income. Moderation research is burgeoning in CSCW, but relatively little attention has been paid to the socioeconomic implications of YouTube’s algorithmic moderation. Drawing from the lens of algorithmic labor, we describe how algorithmic moderation shapes YouTubers’ labor conditions through algorithmic opacity and precarity. YouTubers coped with such challenges from algorithmic moderation by sharing and applying practical knowledge they learned about algorithms. By analyzing video content creation as algorithmic labor, we unpack the socioeconomic implications of algorithmic moderation and point to necessary post-punishment support as a form of restorative justice. Lastly, we put forward design considerations for algorithmic moderation systems.
https://doi.org/10.1145/3479573
Due to challenges around low-quality comments and misinformation, many news outlets have opted to turn off commenting features on their websites. The New York Times (NYT), on the other hand, has continued to scale up its online discussion resources to reach large audiences. Through interviews with the NYT moderation team, we present examples of how moderators manage the first ~24-hours of online discussion after a story breaks, while balancing concerns about reporting credibility. We discuss how managing comments at the NYT is not merely a matter of content regulation, but can involve reporting from the "community beat" to recognize emerging topics and synthesize the multiple perspectives in a discussion to promote community. We discuss how other news organizations, including those lacking moderation resources, might appropriate the strategies and decisions offered by the NYT. Future research might investigate strategies to share and update moderation resources around articles and topics.
https://doi.org/10.1145/3476074
The rapid growth of open source software necessitates a deeper understanding of moderation and governance methods used within these projects. The code of conduct, a set of rules articulating standard behavior and responsibilities for participation within a community, is becoming an increasingly common policy document in open source software projects for setting project norms of behavior and discourage negative or harassing comments and conversation. This study describes the conversations around adopting and crafting a code of conduct as well as those surrounding community governance. We conduct a qualitative analysis of a random sample of GitHub issues that involve the code of conduct and identify different categories of surrounding conversation. We find that codes of conduct are used both proactively and reactively to govern community behavior in project issues. Oftentimes, the initial addition of a code of conduct does not involve much community participation and input. However, a controversial moderation act is capable of inciting mass community feedback and backlash. Project maintainers balance the tension between disciplining potentially offensive forms of speech and encouraging broad and inclusive participation. These results have implications for the design of inclusive and effective governance practices for open source software communities.
https://doi.org/10.1145/3449093
In January 2019, YouTube announced its platform would exclude potentially harmful content from video recommendations while allowing such videos to remain on the platform. While this step is intended to reduce YouTube's role in propagating such content, the continued availability of links to these videos in other online spaces makes it unclear whether this compromise actually reduces their spread. To assess this impact, we apply interrupted time series models to measure whether different types of YouTube sharing in Twitter and Reddit changed significantly in the eight months around YouTube's announcement. We evaluate video sharing across three curated sets of potentially harmful, anti-social content: a set of conspiracy videos that have been shown to experience reduced recommendations in YouTube, a larger set of videos posted by conspiracy-oriented channels, and a set of videos posted by alternative influence network (AIN) channels. As a control, we also evaluate effects on video sharing in a dataset of videos from mainstream news channels. Results show conspiracy-labeled and AIN videos that have evidence of YouTube's de-recommendation experience a significant decreasing trend in sharing on both Twitter and Reddit. For videos from conspiracy-oriented channels, however, we see no significant effect in Twitter but find a significant increase in the level of conspiracy-channel sharing in Reddit. For mainstream news sharing, we actually see an increase in trend on both platforms, suggesting YouTube's suppression particular content types has a targeted effect. This work therefore finds evidence that reducing exposure to anti-social videos within YouTube, without deleting these videos, has potential pro-social, cross-platform effects in sharing across the information ecosystem. At the same time, increases in the level of conspiracy-channel sharing raise concerns about how content producers are responding to YouTube's changes, and transparency from the platform is needed to evaluate these effects further.
https://doi.org/10.1145/3449085