Earlier this year, Twitch replaced its all-or-nothing temporary suspensions with separate penalties for streaming and chatting. Christine Weber, the company’s Chief Technology Officer and global Head of Trust & Safety, says the system is meant to match enforcement to the impact of what a user did. Suspensions now target where a violation occurred.

“People make mistakes. It happens. We just tell them they can’t chat for a while,” says Christine, who has held both roles for about five years. Twitch’s old approach, she says, was more binary, and users were removed from the service for a set time regardless of the impact of their offense.

The change applies to a service that has five million streamers live each month and a global audience of more than 100 million viewers a month, according to Christine. Twitch, founded in 2011 and based in San Francisco, serves streamers building careers on live video. Creators come to Twitch for two-way interactions with their communities, and many cite its safety tools as a reason to come and stay. The new suspension system is where that approach reaches penalties.

Severe Violations Still Draw Both Suspensions

Severe violations that pose the greatest risk to community safety still receive both a streaming and a chatting suspension in order to prevent harm. Some offenses have no place on the service, she says, and can still end in a full suspension. A day-long streaming suspension is also possible for a streamer whose broadcast crossed a line, she says.

Twitch pairs the penalty with an explanation of what the user did wrong. Educating streamers and viewers on that is a priority for the company, Christine says. “We always get that information out to folks. We help them understand.” The system has allowed Twitch to “give that feedback loop to people,” she says.

90% of Suspended Users Never Reoffend, Repeat Violations Draw Escalating Penalties

Twitch’s internal data show that 2% of active users have received a suspension and that 90% of them never reoffend. For Christine, those figures demonstrate that “many users are committed to learning and doing better when they make mistakes.”

Repeat violations carry escalating consequences. “If you continue to do it again and again, well, the consequences escalate. Kind of like getting a speeding ticket,” Christine says. She credits streamers and their communities for the numbers and says she is proud of them. She also acknowledges that users who simply want to cause trouble still show up on Twitch, and says the company moves to identify and remove them quickly.

Christine says Twitch views streamers as “entrepreneurs,” and that view led it to match the penalties to the offense. “They are business people, they are growing their career and their business, and we want Twitch to be a central part of that journey with them,” she says.

Twitch Asks Chatters to Reconsider Messages as Part of Its Education Effort

Twitch runs a system that goes back to the chatter and asks whether they really want to say something. “A lot of people don’t realize how their messaging may be taken,” Christine says. The prompt gives users a second chance, she says, and Twitch has found it effective. She describes it as part of Twitch’s education of users.

The platform also holds messages that look questionable so a moderator can approve or reject them, according to Christine. Streamers who have not yet recruited a human moderator can set Twitch’s tools to be very conservative about what reaches chat, she adds.

Humans in the Loop on Live Content, AI for High-Confidence Chat Blocks

Twitch keeps humans in the loop when it makes moderation and safety decisions. “Our content is live, and determining whether something from a stream is violative involves nuance that AI just can’t account for consistently,” Christine says.

The platform still uses AI for some safety products, she notes. AutoMod scans chat messages in real time and blocks those the company can say with a high degree of confidence are harmful.

Twitch’s systems also weigh context. Content that might not look harmful in isolation can be harmful once the full picture is considered, and the systems look at the streamer’s video and the chat together.

Viewer reports cover what automated processes miss. Christine says a 24/7 moderation staff triages them, handles the most harmful first, and deals with more than 80% in under 10 minutes. She says the reports help Twitch move toward a more proactive approach to safety.

Streamers Are Responsible for Their Chats

“A streamer is responsible for what happens in their chat,” Christine says, and chatters share that responsibility. Twitch’s community guidelines are “essentially the floor,” and streamers can build on top of them. A creator can tell the community that vulgarity is off limits, and a tool helps moderators flag it and publishes that rule to chatters before they ever start chatting.

When trolling starts, a streamer can turn on “Shield Mode” with one click. The feature can slow chat or limit chatting to subscribers, among other settings. Streamers can also apply content classification labels, such as marking a stream as inappropriate for youth or as political speech, so viewers know what a channel is about.

Twitch has also seen an influx of creators who started live streaming on other services. Christine describes the trend as new within the last couple of years. “We really focus on making sure that people understand our rules, especially if they are coming from other services that might have other rules,” she says. Twitch reports back to those creators after their streams to show what the broadcast looked like in terms of safety posture.

Christine says the tools, education and outreach together give streamers confidence in how they build a community. “They can be safe, and they can build their community with confidence, and they don’t have to worry as much about negativity coming into their community,” she says of streamers.

Safety Work Has to Keep Adapting as Memes and Tech Change

The point Christine wants other platforms and brands to understand is that safety is hard and nuanced. It requires a dedicated team of professionals across product development, brand safety, and operations who understand context, she says. “Memes will develop, and you have to quickly and efficiently adapt to those to prevent anything untoward from showing up on your service,” she says.

Twitch views its own safety work as always evolving because its community and technology are, Christine says. Its near-term focus is making it as easy and safe as possible to go live and build the right community. 

“It’s a huge priority for us to actively gather feedback from our community and use their insights to shape what we do next,” she says.

Subscribe to Our Newsletter

Check Out Our Podcast