update i can’t get him off my fucking canvas. i zoomed out my canvas and there was a duplicate of him in the top corner of my canvas. Ive hidden all my layers, he’s on NONE of them, i’ve tried drawing over him, i cant, WHAT THE FUCK DO I DO.
For those who might happen across this,Iâm an administrator for the forum âSufficient Velocityâ, a large old-school forum oriented around Creative Writing. I originally posted this on there (and any reference to âhereâ will mean the forum), but I felt I might as well throw it up here, as well, even if I donât actually have any followers.
This week, Iâve been reading fanfiction on Archive of Our Own (AO3), a site run by the Organisation for Transformative Works (OTW), a non-profit. This isnât particularly exceptional, in and of itself â like many others on the site, I read a lot of fanfiction, both on Sufficient Velocity (SV) and elsewhere â however what was bizarre to me was encountering a new prefix on certain works, that of âEnd OTW Racismâ. While Iâm sure a number of people were already familiar with this, I was not, so I looked into it.
What I found⌠wasnât great. And I donât think anyone involved realises that.
To summarise the details, the #EndOTWRacism campaign, of which you may find their manifesto here, is a campaign oriented towards seeing hateful or discriminatory works removed from AO3 â and believe me, there is a lot of it. To whit, they want the OTW to moderate them. A laudable goal, on the face of it â certainly, we do something similar on Sufficient Velocity with Rule 2 and, to be clear, nothing I say here is a critique of Rule 2 (or, indeed, Rule 6) on SV.
But itâs not that simple, not when youâre the size of Archive of Our Own. So, letâs talk about the vagaries and little-known pitfalls of content moderation, particularly as it applies to digital fiction and at scale. Letâs dig into some of the details â as far as credentials go, I have, unfortunately, been in moderation and/or administration on SV for about six years and this is something we have to grapple with regularly, so I would like to say I can speak with some degree of expertise on the subject.
So, what are the problems with moderating bad works from a site? Letâs start with discoveryâ that is to say, how you find rule-breaching works in the first place. There are more-or-less two different ways to approach manual content moderation of open submissions on a digital platform: review-based and report-based (you could also call them curation-based and flag-based), with various combinations of the two. Automated content moderation isnât something Iâm going to cover here â I feel I can safely assume Iâm preaching to the choir when I say itâs a bad idea, and if Iâm not, Iâll just note that the least absurd outcome we had when simulating AI moderation (mostly for the sake of an academic exercise) on SV was banning all the staff.
In a review-based system, you check someoneâs work and approve it to the site upon verifying that it doesnât breach your content rules. Generally pretty simple, we used to do something like it on request. Unfortunately, if you do that, it can void your safe harbour protections in the US per Myeress vs. Buzzfeed Inc. This case, if you werenât aware, is why we stopped offering content review on SV. Suffice to say, itâs not really a realistic option for anyone large enough for the courts to notice, and extremely clunky and unpleasant for the users, to boot.
Report-based systems, on the other hand, are something we use today â users find works they think are in breach and alert the moderation team to their presence with a report. On SV, this works pretty well â a user or users flag a work as potentially troublesome, moderation investigate it and either action it or reject the report. Unfortunately, AO3 is not SV. Iâll get into the details of that dreadful beast known as scaling later, but thankfully we do have a much better comparison point â fanfiction.net (FFN).
FFN has had two great purges over the years, with a⌠mixed amount of content moderation applied in between: one in 2002 when the NC-17 rating was removed, and one in 2012. Both, ostensibly, were targeted at adult content. In practice, many fics that wouldnât raise an eye on Spacebattles today or Sufficient Velocity prior to 2018 were also removed; a number of reports suggest that something as simple as having a swearword in your title or summary was enough to get you hit, even if you were a âTâ rated work. Most disturbingly of all, there are a number of â impossible to substantiate â accounts of groups such as the infamous Critics United âmass reportingâ works to trigger a strike to get them removed. I would suggest reading further on places like Fanlore if you are unfamiliar and want to know more.
Despite its flaws however, report-based moderation is more-or-less the only option, and this segues neatly into the next piece of the puzzle that is content moderation, that is to say, the rubric. How do you decide what is, and what isnât against the rules of your site?
Anyone whoâs complained to the staff about how vague the rules are on SV may have had this explained to them, but as that is likely not many of you, Iâll summarise: the more precise and clear-cut your chosen rubric is, the more it will inevitably need to resemble a legal document â and the less readable it is to the layman. Weâll return to SV for an example here: many newer users will not be aware of this, but SV used to have a much more âline by line, clearly delineatedâ set of rules and⌠people kind of hated it! An infraction would reference âCommunity Compact III.15.5â rather than Rule 3, because it was more or less written in the same manner as the Terms of Service (sans the legal terms of art). While it was a more legible rubric from a certain perspective, from the perspective of communicating expectations to the users it was inferior to our current set of rules â even less of them read it, and we donât have great uptake right now.
And it still wasnât really an improvement over our current set-up when it comes to âmoderation consistencyâ. Even without getting into the nuts and bolts of âhow do you define a racist work in a way that does not, at any point, say words to the effect of ‘I know it when I see itââ â which is itself very, very difficult donât get me wrong Iâm not dismissing this â you are stuck with finding an appropriate footing between a spectrum of ‘the US penal codeâ and ‘donât be a dickâ as your rubric. Going for the penal code side doesnât help nearly as much as you might expect with moderation consistency, either â no matter what, you will never have a 100% correct call rate. You have the impossible task of writing a rubric that is easy for users to comprehend, extremely clear for moderation and capable of cleanly defining what is and what isnât racist without relying on moderator judgement, something which you cannot trust when operating at scale.
Speaking of scale, itâs time to move on to the third prong â and the last covered in this ramble, which is more of a brief overview than anything truly in-depth â which is resources. Moderation is not a magic wand, you canât conjure it out of nowhere: you need to spend an enormous amount of time, effort and money on building, training and equipping a moderation staff, even a volunteer one, and it is far, far from an instant process. Our most recent tranche of moderators spent several months in training and it will likely be some months more before theyâre fully comfortable in the role â and thatâs with a relatively robust bureaucracy and a number of highly experienced mentors supporting them, something that is not going to be available to a new moderation branch with little to no experience. Beyond that, thereâs the matter of sheer numbers.
Combining both moderation and arbitration â because for volunteer staff, pure moderation is in actuality less efficient in my eyes, for a variety of reasons beyond the scope of this post, but weâll treat it as if theyâre both just ‘moderatorsâ â SV presently has 34 dedicated moderation volunteers. SV hosts ~785 million words of creative writing.
AO3 hosts ~32 billion.
These are some very rough and simplified figures, but if you completely ignore all the usual problems of scaling manpower in a business (or pseudo-business), such as (but not limited to) geometrically increasing bureaucratic complexity and administrative burden, along with all the particular issues of volunteer moderation⌠AO3 would still need well over one thousand volunteer moderators to be able to match SVâs moderator-to-creative-wordcount ratio.
Paid moderation, of course, you can get away with less â my estimate is that you could fully moderate SV with, at best, ~8 full-time moderators, still ignoring administrative burden above the level of team leader. This leaves AO3 only needing a much more modest ~350 moderators. At the US minimum wage of ~$15k p.a. â which is, in my eyes, deeply unethical to pay moderators as full-time moderation is an intensely gruelling role with extremely high rates of PTSD and other stress-related conditions â that is approximately ~$5.25m p.a. costs on moderator wages. Their average annual budget is a bit over $500k.
So, thatâs obviously not on the table, and we return to volunteer staffing. Which⌠letâs examine that scenario and the questions it leaves us with, as our conclusion.
Letâs say, through some miracle, AO3 succeeds in finding those hundreds and hundreds and hundreds of volunteer moderators. Weâll even say none of them are malicious actors or sufficiently incompetent as to be indistinguishable, and that they manage to replicate something on the level of or superior to our moderation tooling near-instantly at no cost. We still have several questions to be answered:
How are you maintaining consistency? Have you managed to define racism to the point that moderator judgment no longer enters the equation? And to be clear, you cannot allow moderator judgment to be a significant decision maker at this scale, or you will end with absurd results.
How are you handling staff mental health? Some reading on the matter, to save me a lengthy and unrelated explanation of some of the steps involved in ensuring mental health for commercial-scale content moderators.
How are you handling your failures? No moderation in the world has ever succeeded in a 100% accuracy rate, what are you doing about that?
Using report-based discovery, how are you preventing ‘report brigadingâ, such as the theories surrounding Critics United mentioned above? It is a natural human response to take into account the amount and severity of feedback. While SV moderators are well trained on the matter, the rare times something is receiving enough reports to potentially be classified as a ‘brigadeâ on that scale will nearly always be escalated to administration, something completely infeasible at (youâre learning to hate this word, Iâm sure) scale.
How are you communicating expectations to your user base? If youâre relying on a flag-based system, your usersâ understanding of the rules is a critical facet of your moderation system â how have you managed to make them legible to a layman while still managing to somehow ‘trulyâ define racism?
How are you managing over one thousand moderators? Like even beyond all the concerns with consistency, how are you keeping track of that many moving parts as a volunteer organisation without dozens or even hundreds of professional managers? Iâve ignored the scaling administrative burden up until now, but it has to be addressed in reality.
What are you doing to sweep through your archives? SV is more-or-less on-top of ‘oldâ works as far as rule-breaking goes, with the occasional forgotten tidbit popping up every 18 months or so â and thatâs what weâre extrapolating from. These thousand-plus moderators are mostly going to be addressing current or near-current content, are you going to spin up that many again to comb through the 32 billion words already posted?
I could go on for a fair bit here, but this has already stretched out to over two thousand words.
I think the people behind this movement have their hearts in the right place and the sentiment is laudable, but in practice it is simply ‘wonât someone think of the childrenâ in a funny hat. It cannot be done.
Even if you could somehow meet the bare minimum thresholds, you are simply not going to manage a ruleset of sufficient clarity so as to prevent a much-worse repeat of the 2012 FF.net massacre, you are not going to be able to manage a moderation staff of that size and you are not going to be able to ensure a coherent understanding among all your users (we havenât managed that after nearly ten years and a much smaller and more engaged userbase). Thereâs a serious number of other issues I havenât covered here as well, as this really is just an attempt at giving some insight into the sheer number of moving parts behind content moderation: the movement wants off-site content to be policed which isnât so much its own barrel of fish as it is its own barrel of Cthulhu; AO3 is far from English-only and would in actuality need moderators for almost every language it supports â and most damning of all, if Section 230 is wiped out by the Supreme Court it is not unlikely that engaging in content moderation at all could simply see AO3 shut down.
As sucky as it seems, the current status quo really is the best situation possible. Sorry about that.
I have largely been not engaging with the latest go-around of end racism in the OTW, in part because I donât feel itâs really my lane, but mostly because I do not think the things they are asking for are going to solve the problem they want to solve, and has great potential to cause more problems.
The above post has a lot of very detailed breakdown on why content moderation on the scale of the Archive is not feasible with their current budget and infrastructure, but I really want to highlight and delve more into the issue they briefly touched on of malicious reporting.
I have been in fandom in one form or another since the turn of the century, and the one thing I have seen repeatedly â time and time again, with evolving vocabulary falling into the exact same structures â is that fans will weaponize any available tool to attack other fans. They have done, they still do, they will continue to do this.
I can see a position that says, well thatâs an acceptable collateral for the greater good of reducing racial harm. But I also believe, pretty firmly, that POC creators are going to be the ones hurt most by this. This will happen for three reasons:
POC creators are more likely to be targeted for harassment in the first place, because racism (because thereâs just ‘something they canât put their finger onâ about this person that they donât like, that drives people to dig through their work and biography for excuses;)
Non-American creators are more likely to be POC creators, and non-American fans are the most likely to be not English speakers, not steeped in American purity culture, and not up on the shibboleths of whatever is considered Right And Good on (american) social media this week;
POC creators are more likely to be engaging with POC characters, works, OCs and themes in the first place, and are already being policed for performing their ethnicity to the audienceâs satisfaction
Here are a few things I have seen happen in the past few years, and these are only things that I have directly seen happen in real time, not getting into stuff that may happen in other channels Iâm not witness to:
Asian artists being targeted and harassed for âwhitewashingâ Asian characters (eg, drawing them Not Asian Enough; thereâs an entire conversation to be had on marked vs unmarked styles of depiction that I donât have space for here;)
SEA artists being targeted and harassed for drawing characters at the same level of color as the characterâs official art, when the community has decided that they should be darker (see also the American hangup of judging a characterâs ethnic representation according to color of skin and color of skin only;)
Black creators being targeted and harassed for making âfetishâ content of OCs and characters based on their own selves.
In some of these cases, that things got so out of hand could be attributed to well-meant but misguided zealotry. But in the majority of harassment cases Iâve seen over two decades in this fandom, the single driving root of every conflict is that a fan decided they did not like the way another creator shipped, drew, or wrote their favorite character, so they looked around for a suitably hefty stick to whack them with; and that stick might be wrapped in a concealing cellophane of social justice but itâs still going to be used as a weapon.
Fans are not going to stop doing this. POC creators are going to continue to be in the cross-sights. I do not see any measurable benefit in giving them bigger and heftier sticks, or trying to pull AO3 moderators into these arguments as referees.