This post describes work appearing at CSCW 2014.
Many online activities today involve interacting with other people, and these interactions are often dictated by social norms – unspoken rules that classify socially acceptable behavior. Whether intentional or otherwise, people sometimes break these rules. Behavior that goes against established norms is called deviant behavior, and in many cases is assumed to be negative. In online communities, deviant behavior is commonly believed to be harmful, and is expected to drive users to leave the community. For example, intentionally incorrect responses on a question and answer site may discourage new users from asking questions. Our work looks at patterns of player behavior in an online game where people will play bingo for money. We build a metric to predict a specific kind of deviant behavior, toxicity, and we use this metric to examine whether deviant behavior causes other players to quit.
We look at deviant behavior in the popular online game League of Legends. League of Legends is a competitive multiplayer game where two teams of five players compete to destroy the other team’s base. Each player controls a single character, and every character has unique abilities which, when used skillfully, can help the player’s team overcome an enemy team. If you’ve never played League of Legends, check out this four-minute introduction video from League’s developer, Riot Games.
Deviant behavior in League of Legends is a known problem. Riot has addressed it with several interventions, such as rallying the community behind a common code of conduct and using social psychology techniques to subtly influence player behavior. Problematic deviant behaviors in League of Legends manifest in several ways: A player could verbally abuse other players, quit a match early or help the enemy team win. Riot is specifically concerned with some of these behaviors, which they call “toxicity.” In their talk last Spring, they define toxicity as any behavior that negatively impacts other players’ experiences. You can see more on Riot’s efforts to influence player behavior in their GDC2013 talk.
Using data donated by Duowan (the creators of a popular Chinese add-on for League of Legends) we analyzed 18 million matches across three months. The add-on allows players to give feedback to their teammates. From recorded social and match data, we built a metric called toxicity index. Toxicity index is a ratio between “thumbs up” and “thumbs down” scores, used to identify players with consistently negative feedback. A higher toxicity index means a player was given lots of negative feedback by their peers. Similar to the “report a player” feature built in to League of Legends, thumbs down messages are strongly associated with toxic behavior in the game.
Toxicity Index
Once we were able to identify toxic players, we found several common correlations with toxicity index:
- Ranked matches are more competitive matches which have a public-facing rating. Players in ranked matches often had a higher toxicity index than players in normal matches.
- Different character roles predicted different levels of toxicity.
- Playing characters with lots of damage-dealing abilities (carries and bruisers) usually predicted players with higher toxicity indexes.
-
Different character roles predicted different levels of toxicity.
-
Playing characters with lots of damage-dealing abilities (carries and bruisers) usually predicted players with higher toxicity indexes.
-
Playing characters with abilities that promote teamwork (supports and tanks) usually predicted players with lower toxicity indexes.
-
- While we expected more positive feedback, players teaming up with friends received more negative feedback from their non-friend teammates.
What predicts quitting?
We also wanted to look for factors that drove players to quit the game. To do this we separated players by tenure, splitting experienced and inexperienced players. Using a regression, we examined player retention in terms of both finishing their current gaming session and quitting the game for a long time (or for good).
- Newer player’s session length didn’t seem to be affected by playing with toxic teammates, regardless of their own toxicity index.
- Newer players were less likely to quit for a long period when they had a high toxicity index, and were more likely to quit for a while after playing with toxic teammates.
- Surprisingly, experienced players were more likely to continue their session when they were toxic, or when some of their teammates were toxic. However, if the player and their teammates both had high toxicity indexes, the player was likely to end their session.
-
Both experienced and new players who play with friends are more likely to play another match.
-
Playing with friends also kept newer players from taking a long break from the game, and playing with friends was the only factor that kept experienced players from quitting for good.
In summary, we found different game modes, character abilities and playing with friends all correlated with different levels of toxic behavior. Additionally, newer players who were toxic were less likely to quit the game, and newer players who weren’t toxic were likely to quit after encountering toxic teammates. Playing with friends predicted players at all levels would keep playing.
Want to learn more? Read our full paper, accepted to be published this February at CSCW 2014.
2 Responses to “Deviant behavior in League of Legends: Do jerks drive off other players?”
-
-
Comments are closed.Thomas
Additional analyses/research I would have liked to see:
–Does a streak of losses/wins lead to an increase/decrease in the toxicity of a player for any given match. And, does the likelihood of toxicity increase for each given loss/win? By what factor?
–Is there a higher rate of toxicity from players when winning or losing? Does the outcome of a ‘play’ cause toxicity to emerge? (This would require qualitative research to better understand the social dynamics at play; would require a coding system to quantify and compare to the toxicity index).
–The toxicity index segmented by league (i.e. bronze, silver, gold…) and the level/likelihood of toxicity for a promotion match vs. a normal ranked match.
–Is there a higher toxicity index for players who come into the game invested (a survey might be needed for this information)?
–Are players with a low toxicity index likely to have their index increase when exposed to higher toxicity index players, and vice versa… (could explain, in part, what feeds toxicity and how it spreads)?
I was also curious as to what motivates players give either a thumb up/down, since this is the dependent variable. The study assumes that the thumbs up up/down are diametrically opposed, does this align with user mental models and behavior? How strict/loose do people interpret the giving of each (does it align to the guidelines Riot provide or the add-on’s)? Are lower toxicity index players likely to thumb up/down; are higher toxicity players likely to thumb up/down? How much voting is revenge voting?
Some of these questions would require a qualitative exploration of use-behavior and player’s ‘sociotechnically’ mediated interactions. I imagine such a study would be an excellent companion piece to what what has been authored, as it would explain the mechanics behind the metric analysis is based on and providing design implications.
Interesting stuff, it is awesome that a study such as this can generate new questions and research possibilities. The mark of a useful study.
Kate
These are some really good questions. A lot of them, like the measurement of toxicity based on a particular game, were something we couldn’t measure given our dataset. The question about leagues is something we could check, except the data is from S2 / pre-S3, so there weren’t any promotion games or brackets yet.
As for thumbs down / thumbs up guidelines, the addon by Duowan was a little nuanced with how it allowed players to give ratings, though players reported independently that they used the thumbs-down much in the same way as a player might use the “report” feature. We weren’t able to consider specific games (we looked at thumbs up/down changes across periods of a week or two, based on the data), so we unfortunately couldn’t examine toxicity index changes across a session.
I agree, our study is very much a data-analytic approach to addressing these questions, and I would love to see/do a follow-up with both the more qualitative questions, as well as with a new dataset, specifically set up to collect data to answer many of the questions you have asked here.
Thanks for taking the time to comment!