Rocking the boat: The riddle of writing reviews

its a trap

Reviews are hard. I know, because I’m currently working on three of them. It’s been a while since I’ve done a review, and the older and wiser me is second-guessing a lot of previously held assumptions.

The value of reviews, their complexities and challenges, has been on my mind lately so I decided to use my column space this week to challenge the taken-for-granted knowledge around them, and discuss what their value actually is – and how best we can write and consume them.

Objectivity vs Subjectivity

Answer quick – do you think reviews should be objective or subjective?

You sure?

I don’t really know what the right answer to this question is. Reviews are of course, by nature, entirely subjective. But what is a review trying to be? Is it simply our own experience of something, or is it an objective analysis by a subjective party?

There’s layers of complication to this. If I’m part of the 0,1% of people who invoke a very rare bug that doesn’t allow me to play the game properly, I would be subjectively justified in giving the game an awful rating.

Of course, objectively, giving a game a 2/10 due to an issue that would never affect the overwhelming majority of people is nonsensical. Clearly these are extremes, and you may be justified in declaring me an idiot and saying that it doesn’t have to be one or the other, I can simply use my own discretion.

For the most part, I agree with that. But that kind of objectivity involves a certain level of conformity to others. If you want to discount your own bad experience since it was applicable only to you, it means incorporating experience that wasn’t there in the first place. Which leads to my next issue:

The pull of the group

Do you think that, as a reviewer, you should read other people’s reviews?

Let’s say you’re reviewing a mouse, and in your opinion it feels cheap. A bit too plastic, a bit too flimsy. Now what if you read ten other reviews of the same mouse, and every review praises its sturdiness and high build quality? Would you still be comfortable with including your own experience?

Comfortable, no. Absolutely not. Whether or not that means you revise what you said is a different story, but you’ll certainly be feeling a little more anxious about it now. See, people have an innate fear of nonconformity. Social psychologists have done plenty of research to show that when someone is placed in a group where everybody agrees on the same thing, that person is overwhelmingly likely to go with the group’s opinion even if they were previously quite certain about something.

Except being a Belieber. Nobody can force you into that, that's all on you.
Except being a Belieber. Nobody can force you into that, that’s all on you.

The classic example of this experiment is showing people three straight lines and asking which one is the longest. Now line B might be quite clearly longer than line C or line A, but when twelve other people in the group (actors, obviously) all say line C is the longest, the last person will often agree.

The catch here is that it’s not merely about peer pressure or being shy, it’s that that person will genuinely start to believe that they have it wrong, that their eyes are deceiving them and line C really is the longest.

Let’s go back to our mouse example. If I read ten reviews that say a mouse has amazing build quality, but I think it’s shit, I’m going to start doubting whether or not I know what the hell a well-built mouse actually feels like.

Of course, you can also not read any other reviews or opinions at all. But then we fall into the same dangers of extreme subjectivity described previously. If something goes horribly wrong, you’d want to know if other people had experienced the same thing as well.

Of course, there’s another glaring social psychology issue that hasn’t been addressed yet.

The impact of hype

Hype has a very real, and very unfortunate, influence on our opinions of things. Quite simply when we expect something to be good, or even when we really want it to be good, we tend to form our own subjective confirmation bias around it – we emphasise the good, and gloss over the bad.

This is most apparent for me in the Call of Duty series. When the original Modern Warfare came out, people were in love with the franchise. Everybody wanted more, and the game had yet to turn into the repetitive, annualised-release golden goose that it is today.

The original Modern Warfare was incredible, Modern Warfare 2 and 3 were really, really not. With both games getting review scores in the 90s, you know something is up. What’s interesting to note is that once Call of Duty hype wore off, once public opinion turned against it and it was labelled as the poster child for repetitive franchise whoring, the reviews went with it.

Ghosts, Advanced Warfare and Black Ops III all rest comfortably in the low 80s, despite the latter two almost certainly being better games than the likes of Modern Warfare 3 (Ghosts was pretty bad).

Ah man, remember when they put it on Wii U? *shudder*
Ah man, remember when they put it on Wii U? *shudder*

Hype may not be all-defining, a terrible game will remain a terrible game, but it does have an impact. People are simply too scared to give big AAA scores that are too low. Which brings us to my final issue.

What’s in a number?

When I was at University, there was a friendly rivalry between the BCom, BA and BSc students. One thing the BSc folk used to like to laud over us BA peasants was that the top academic performers were almost always BSc students.

What I always had to point out was that they weren’t playing with the same set of rules. You see, in a Physics exam getting everything right gives you 100%, that’s simply all there is to it.

If you’re writing a Psychology essay, however, marks are assigned based on the quality of the essay – and there is no 100. A really good essay would net you a 75, and an outstanding one would get something in the low 80s. The rating scale may technically go to 100, but everyone knows there’s a soft cap at around 85.

Game reviews have trended upwards over the years, to the point where the actual scores have become meaningless in a vacuum. The only way to judge scores now is not based on the number, but on the comparison.

A score of 80 is now considered pretty crappy. It’s a game that’s decent, but not really all that great. How the hell does “competent, but not anything special” fit in with a score of 80? Most people agree that Call of Duty: Ghosts is the worst one ever made, and yet somehow that manages an 80, the percentage equivalent of an A?

To really illustrate the bloat of review scores in the last decade, consider this: Half-Life, Half-Life 2 and Modern Warfare 2 are within two percentage points of each other. Two FPSes that are considered some of the most revolutionary and sophisticated of all time occupy the same review space as a mediocre sequel. If we’ve used up our “perfect” score on MW2, there’s very little space to work with when a game comes along that truly blows us away.

Reviews need to return to a place of honesty. Honesty in terms of what you really thought of the thing you’re reviewing, not what other people expect you to think. Honesty in terms of giving it a score that matches how you actually feel about it.

I’m curious, NAG Online readers, on what your thoughts are. Do you still read reviews? What value do you gain from them? And what would you like to see change about the status quo?

Project Porg just wants you to live your best life