When I reviewed Star Wars Jedi: Survivor back in April, I gave it a 9 after playing through its campaign on PlayStation 5 without major issues… only to wake up the next day to hear horror stories of the PC version performing terribly. (This gave me flashbacks to when the same thing happened with Batman: Arkham Knight in 2015). The same month, Luke Reilly had to hit Redfall hard in his review with a score of 4 after numerous bugs drained the lifeblood out of what was already not a great time. At a glance, this certainly could give the appearance of reviewers being inconsistent: why does one game with a reputation for bugs get raked over the coals while another “gets a pass?” Like so many other things it’s not nearly that simple, and to a large extent there is no one-size-fits-all solution for game critics to employ when it comes to something as notoriously slippery as bugs – but we do our best anyway.
There’s never been a game that doesn’t have bugs of some sort, but they’ve become an even hotter topic of discussion in the extraordinary gaming year of 2023. We’ve seen troubled launches of the likes of Redfall, Jedi: Survivor, and Baldur’s Gate 3, which – underneath the near-universal praise – has sparked a conversation about how many bugs are acceptable in highly ambitious games.
Completely understandably, people don’t like bugs (except when they’re hilarious and happen to other people in montages on YouTube) and want fewer of them in the highly anticipated games they lined up to spend at least $70 on. That’s something everyone – consumers, critics, and developers – can agree on. But when bugs inevitably rear their ugly heads, conflicts arise. Gamers rightfully want reviewers to do their jobs as consumer advocates by holding developers’ feet to the fire for every single issue… even if the reviewer didn’t personally experience those problems and had no way to know if someone else would. At the same time, many gamers are also fiercely protective of games that they love, warts and all, and if a reviewer holds a game accountable for a problem – especially one that a fan doesn’t see during their own playthrough, that’s seen as unfair. I’ve even seen it argued that if the reviewer experiences a major bug that completely breaks their game, if it doesn’t affect everybody who plays then it shouldn’t affect the review.
Bugs are exceptionally tricky because their very nature makes them inconsistent.
That’s a difficult if not impossible circle to square, which means bugs are an exceptionally tricky topic to address in any game review because their very nature makes them inconsistent. In the vast majority of cases a reviewer only has time to play through a game once before writing their opinion, which makes it something of a crapshoot, especially in the biggest and most complex games. Look at something like Baldur’s Gate 3, where we have enormous freedom to determine how we play, and making a choice that the developers might not have had time to test can be disastrous. Picking up a sword before talking to an NPC who gives you a quest while playing as a mage might be game-breaking, while doing the same thing as any other class is totally fine. Maybe it’s not even that simple – it’s something you did hours before, or an item that’s equipped on a party member sitting back at camp that does you in. If you’re playing in co-op you double your trouble, because who knows what your partner has been up to while you weren’t looking? When there are nearly infinite permutations that must be accounted for, bugs will definitely happen to some people that don’t happen to all.
We also saw this kind of thing in play with Redfall’s enemy AI, which in some cases failed to activate at all, and there was no way to know why. In the case of Jedi: Survivor, consoles ran decently at launch, but your results when running the PC version depend heavily on which combination of CPU, GPU, and OS you’re using – and there are countless configurations to choose from. In short, there is no way for a reviewer to account for any of this when they’re playing for review. They only know what they see when they play.
What’s more, reviewers are typically playing – and landing on their final score – before the general public even begins. So when they encounter a bug it’s all but impossible to sort out why it’s happening or roughly how many people might experience the same thing. (Similarly and even more paranoia-inducing, if you don’t encounter any bugs at all, you can’t be certain that no one else will.) When we hit roadblocks we can’t get around by reloading a previous save we contact the developer; if they can fix it before we publish, great – the public never experiences that problem, so we pretend it never happened and everybody wins (which is reward enough for critics’ unpaid QA work). But if they can’t, the only honest and responsible thing to do is to convey to our readers and viewers what bugs we ran into and how they affected our experience, and our final recommendation.
The only honest and responsible thing to do is to convey what bugs we ran into.
But how do you factor in bugs in a way that’s fair, exactly? I often see people in the community suggest an automatic score deduction for bugs, but in my experience mandatory adjustments to a score never work as well as they sound like they should on paper. For one thing, not all bugs are created equal. Some block progress – or reverse it – and make you want to throw things at your screen; others are annoyances you have to work around; precious few rare gems are outright hilarious and memorable, and might even make a game better than the developers ever intended, with the classic example being skiing in StarSiege Tribes. (I will always remember a mammoth in Skyrim that, while charging me, stepped on a rock in just the wrong way and was flung into the stratosphere.) It wouldn’t make sense to treat all of these the same way when scoring because they do not all have the same impact. You could do a detailed list of what types of bugs result in precisely how steep of a penalty, but that becomes so complex that it’s meaningless to just about everyone.
Another question I see a lot is whether or not it’s fair to criticize a game for a bug that will probably be fixed in an upcoming patch. It’s another tricky question, because it’s completely true that most major bugs are addressed. But “probably” and “most” are doing a lot of work here, and to ignore them on the grounds that maybe, at some point in the future, they might not be relevant would neglect a reviewer’s responsibility to the great majority of their audience: people who are reading within a week or so of it being published. If a bug exists at launch, people who take a reviewer’s advice and buy and play that game are the most likely to experience that issue.
With all of that in mind, the best approach we’ve landed on is to keep it simple: a reviewer tells you what happened to them when they played, what lows they were willing to forgive in order to experience the highs, and what ruined the mood for them. That isn’t ever going to perfectly reflect the experience everyone else will have when they play, and it’s not comprehensive across all platforms because no reviewer can make sure a game works the same way on Switch, Xbox One, Xbox Series S, Xbox Series X, PlayStation 4, PlayStation 5, and PC. Also, it might be different in the future, but at the time of writing it is a genuine, honest reflection of what we played and how we felt about it.
We don’t want to just leave it there though, because we still want to do the best we can to answer the question of how performance differs across platforms, and this is what we hope to accomplish with our Performance Reviews. In addition to our full reviews of major games, our extremely knowledgeable expert, NX-Gamer, drills down into extremely technical detail – far better than any of our gameplay-focused reviewers could – and uses a custom-made software setup to compare and evaluate performance on all platforms. Due to day-one patches it’s generally not possible to publish these on release day (since benchmarking results can be significantly different) but we try to have them ready to inform you as soon as possible. And the moment we come up with a way to do that even better, we’ll be all over it.
Dan Stapleton is IGN's Director of Reviews. You can follow him on Bluesky @danstapleton.bsky.social or Twitter (which he still insists on calling it) @danstapleton.
Continue reading...
There’s never been a game that doesn’t have bugs of some sort, but they’ve become an even hotter topic of discussion in the extraordinary gaming year of 2023. We’ve seen troubled launches of the likes of Redfall, Jedi: Survivor, and Baldur’s Gate 3, which – underneath the near-universal praise – has sparked a conversation about how many bugs are acceptable in highly ambitious games.
Completely understandably, people don’t like bugs (except when they’re hilarious and happen to other people in montages on YouTube) and want fewer of them in the highly anticipated games they lined up to spend at least $70 on. That’s something everyone – consumers, critics, and developers – can agree on. But when bugs inevitably rear their ugly heads, conflicts arise. Gamers rightfully want reviewers to do their jobs as consumer advocates by holding developers’ feet to the fire for every single issue… even if the reviewer didn’t personally experience those problems and had no way to know if someone else would. At the same time, many gamers are also fiercely protective of games that they love, warts and all, and if a reviewer holds a game accountable for a problem – especially one that a fan doesn’t see during their own playthrough, that’s seen as unfair. I’ve even seen it argued that if the reviewer experiences a major bug that completely breaks their game, if it doesn’t affect everybody who plays then it shouldn’t affect the review.
Bugs are exceptionally tricky because their very nature makes them inconsistent.
That’s a difficult if not impossible circle to square, which means bugs are an exceptionally tricky topic to address in any game review because their very nature makes them inconsistent. In the vast majority of cases a reviewer only has time to play through a game once before writing their opinion, which makes it something of a crapshoot, especially in the biggest and most complex games. Look at something like Baldur’s Gate 3, where we have enormous freedom to determine how we play, and making a choice that the developers might not have had time to test can be disastrous. Picking up a sword before talking to an NPC who gives you a quest while playing as a mage might be game-breaking, while doing the same thing as any other class is totally fine. Maybe it’s not even that simple – it’s something you did hours before, or an item that’s equipped on a party member sitting back at camp that does you in. If you’re playing in co-op you double your trouble, because who knows what your partner has been up to while you weren’t looking? When there are nearly infinite permutations that must be accounted for, bugs will definitely happen to some people that don’t happen to all.
We also saw this kind of thing in play with Redfall’s enemy AI, which in some cases failed to activate at all, and there was no way to know why. In the case of Jedi: Survivor, consoles ran decently at launch, but your results when running the PC version depend heavily on which combination of CPU, GPU, and OS you’re using – and there are countless configurations to choose from. In short, there is no way for a reviewer to account for any of this when they’re playing for review. They only know what they see when they play.
What’s more, reviewers are typically playing – and landing on their final score – before the general public even begins. So when they encounter a bug it’s all but impossible to sort out why it’s happening or roughly how many people might experience the same thing. (Similarly and even more paranoia-inducing, if you don’t encounter any bugs at all, you can’t be certain that no one else will.) When we hit roadblocks we can’t get around by reloading a previous save we contact the developer; if they can fix it before we publish, great – the public never experiences that problem, so we pretend it never happened and everybody wins (which is reward enough for critics’ unpaid QA work). But if they can’t, the only honest and responsible thing to do is to convey to our readers and viewers what bugs we ran into and how they affected our experience, and our final recommendation.
The only honest and responsible thing to do is to convey what bugs we ran into.
But how do you factor in bugs in a way that’s fair, exactly? I often see people in the community suggest an automatic score deduction for bugs, but in my experience mandatory adjustments to a score never work as well as they sound like they should on paper. For one thing, not all bugs are created equal. Some block progress – or reverse it – and make you want to throw things at your screen; others are annoyances you have to work around; precious few rare gems are outright hilarious and memorable, and might even make a game better than the developers ever intended, with the classic example being skiing in StarSiege Tribes. (I will always remember a mammoth in Skyrim that, while charging me, stepped on a rock in just the wrong way and was flung into the stratosphere.) It wouldn’t make sense to treat all of these the same way when scoring because they do not all have the same impact. You could do a detailed list of what types of bugs result in precisely how steep of a penalty, but that becomes so complex that it’s meaningless to just about everyone.
Another question I see a lot is whether or not it’s fair to criticize a game for a bug that will probably be fixed in an upcoming patch. It’s another tricky question, because it’s completely true that most major bugs are addressed. But “probably” and “most” are doing a lot of work here, and to ignore them on the grounds that maybe, at some point in the future, they might not be relevant would neglect a reviewer’s responsibility to the great majority of their audience: people who are reading within a week or so of it being published. If a bug exists at launch, people who take a reviewer’s advice and buy and play that game are the most likely to experience that issue.
With all of that in mind, the best approach we’ve landed on is to keep it simple: a reviewer tells you what happened to them when they played, what lows they were willing to forgive in order to experience the highs, and what ruined the mood for them. That isn’t ever going to perfectly reflect the experience everyone else will have when they play, and it’s not comprehensive across all platforms because no reviewer can make sure a game works the same way on Switch, Xbox One, Xbox Series S, Xbox Series X, PlayStation 4, PlayStation 5, and PC. Also, it might be different in the future, but at the time of writing it is a genuine, honest reflection of what we played and how we felt about it.
We don’t want to just leave it there though, because we still want to do the best we can to answer the question of how performance differs across platforms, and this is what we hope to accomplish with our Performance Reviews. In addition to our full reviews of major games, our extremely knowledgeable expert, NX-Gamer, drills down into extremely technical detail – far better than any of our gameplay-focused reviewers could – and uses a custom-made software setup to compare and evaluate performance on all platforms. Due to day-one patches it’s generally not possible to publish these on release day (since benchmarking results can be significantly different) but we try to have them ready to inform you as soon as possible. And the moment we come up with a way to do that even better, we’ll be all over it.
Dan Stapleton is IGN's Director of Reviews. You can follow him on Bluesky @danstapleton.bsky.social or Twitter (which he still insists on calling it) @danstapleton.
Continue reading...
