Microsoft is testing a new anti-toxicity program with custom, multi-level user controls that automatically filter Xbox Live inbox content to reduce the visibility of abusive text messages (or not, if you’re into that kind of thing).
The feature includes four different filter levels – Friendly, Medium, Mature, and Unfiltered – and will be available on Xbox consoles, and Xbox apps on PC and mobile. As you’d expect, “Friendly” filters out everything the algorithm determines to be potentially offensive, and “Unfiltered”, much as it is now, lets it through anyway. “Medium” and “Mature” are in between. For adult accounts, filtered messages show up with a warning but can be opened at the user’s discretion, but blocked by default on kid accounts.
So that’s fun but what’s more interesting, perhaps, is how Xbox decides if a message is problematic when a lot of its users are pretending to murder each other in games but not in reality.
“Context is a really tricky thing in the gaming space,” Xbox ops boss Dave McCarthy tells The Verge. “It’s one thing to say you’re going to go on a killing spree when you’re getting ready for a multiplayer mission in Halo, and it’s another when that’s uttered in another setting. Finding ways for us to understand context and nuance is a never-ending battle.”
To solve this, Xbox uses a mix of artificial intelligence and regular no-no-word text filters, with a database of prohibited phrases that can be instantly updated as users inevitably find exploits to break the system.
And in future, Microsoft intends to deploy the same filters to live voice chat, and bleep out that guy making implausible claims about your mom.
“What we’ve started to experiment with is ‘Hey, if we’re real-time translating speech to text, and we’ve got these text filtering capabilities, what can we do in terms of blocking possible communications in a voice setting?’” McCarthy explains. “It’s early days there, and there are a myriad of other AI and technology that we’re looking to stack around the voice problem, things like emotion detection and context detection that we can apply there. I think we’re learning overall… we’re taking our time with this to do it right.”
The new feature is available now to Xbox Insider subscribers, and is scheduled for a public launch by the end of 2019.