A new study linking driving habits and gaming habits has been frequenting the internet lately, but just what are the researchers trying to convey? I happen to be a research student in a Rutgers PhD program, so here’s what I think.

The study, put forth by Continental Tires, attempts to demonstrate a link between people who play racing games and people who are bad drivers. Basically what they did was ask 2,000 people about their gaming habits and their driving history, including questions about whether or not they’ve ever been stopped by police, if they’ve hit stationary objects while parking, or if they self-assess risk-taking driving behavior. They also asked a few questions about how many attempts it took to pass their driving exam and how many dents they’ve had on their vehicle in the last year.

At a glance, the study indicates that people who self-assessed as people who play racing games also indicated more frequent negative driving incidents like road rage, running red lights, and getting pulled over by the police. At that point, our brains draw the all-too-natural conclusion that playing racing games causes you to be a bad driver. Besides some anecdotal video evidence that might show how gaming skills could save your life, we typically don’t have any immediate counter arguments.

However, as a research student I have plenty.

First and foremost, this is not a “study.” It’s a survey sponsored by a tire company. It isn’t published in any sort of academic journal and it isn’t peer reviewed, so there’s no way to tell how valid or reliable it really is. I can’t stand it when news media calls these things “studies.”

Secondly, correlation is not causality. In simpler terms, just because two things are linked does not mean that one causes the other. While a statistic link is necessary for causality, it is not sufficient to assert it. While playing racing games could make you a worse driver, it could be that being a bad driver causes you to play more racing games. It could also be that some unknown factor causes you to both be a bad driver and to play more racing games, a phenomenon that researchers call spuriousness, or a spurious link.

Correlation doesn't imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing, "Look over there."

Image courtesy of XKCD.com

A perfectly reasonable explanation for spuriousness here could be age. Young people are more likely to play video games and young people are also more likely to be bad drivers. It could be that being young explains all of the data, yet due to the limitations of the survey we have no idea if that’s true or not.

Beyond that, I take strong issue with their sampling method, or the way in which they chose people to participate in their survey. More specifically, their complete lack of sampling method disclosure: We have no idea if they randomly selected participants (a randomized sample), if they just took whoever was available (an availability sample), or if they got a small handful of people to do it who then enlisted their friends to do it (a snowball sample). Without any clues about their sampling technique, we don’t know if they’ve introduced a sampling bias, or the idea that the way in which a researcher selects participants can accidentally select participants who all share certain characteristics that skew the results. Maybe playing racing games does make you a bad driver, but only if some additional factor is true, but the surveyors happened to get a bunch of people who all had that additional factor going for them because of their sampling method.

An instance of snowball sampling gone wrong, courtesy of Calvin & Hobbes

Furthermore, there’s no evidence of any sort of statistical testing. All we have is a table, the results of the survey, and the perceived notion that these results mean something. There’s no talk about margins of error, outliers, or significance, meaning that we know nothing about how wrong the data could be, what the exceptional answers were like, or if the numbers given to us actually mean anything. While it might seem significant that race gamers took, on average, 1 attempt to pass their driving exam but non-gamers took, on average, 2, using the mean and truncating or rounding it for those groups could be a terrible choice to suggest significance. If, for instance, the gaming average was 1.4 and the non-gamer average was 2.0 and the margin of error was 0.35, it’s very possible that there is no difference at all between how many times it takes a gamer to pass the driving test and how many times it takes a non-gamer to pass it.

The bottom line is that these results aren’t really results at all, but just some numbers that researchers at Continental Tires wrote down after they gave out a survey. The results are absolutely worthless and don’t indicate anything meaningful at all and we shouldn’t be swayed by the allure of badly presented numbers.

The sad part is that this kind of thing still makes the news, getting at least some people to believe statements that are patently false.

Advertisements