AnandTech discovered that Huawei programmed its flagship P20 phone to artificially inflate its score in 3DMark, a popular benchmark program designed to measure graphics performance.
A custom cheating code detected the 3DMark app and put the phone in full throttle to get the best results possible. This is not allowed per 3DMark rules. When the company learned about AnandTech’s findings, it ran its own internal version, which resulted in a significantly lower score.
As a result, 3DMark has delisted the Huawei P20 as well as the following cheating Huawei phones: the P20 Pro, the Nova 3, and the Honor Play. But this is hardly the first time a smartphone vendor has been caught cheating, and it won’t be the last.
MORE: Smartphones with the Longest Battery Life
Liar liar, chips on fire
Nothing of this is new: Android phones benchmark cheating has been running rampant for a while, with 3DMark (at the time it was called Futuremark) delisting phones like the Galaxy Note or the HTC One in 2013.
Samsung was also caught cheating on phone benchmarks with the Galaxy S4, And OnePlus, too, with the OnePlus 5.
The fact is that synthetic benchmark cheating has been happening ever since benchmarks came to be in the computing world. I remember it happening regularly in the early 1990s, before the internet came in. In the aughts, Nvidia was caught multiple times fine tuning code for benchmark performance, just like its archrival ATI (which later was adquired by AMD) — back in 2003. Intel cheated, too, back in 2000.
A bizarre response
The whole situation is a real shame, because it detracts from the Huawei P20 Pro's amazing performance as a triple camera phone. In fact, the P20 Pro beat the iPhone X and Pixel 2 XL in our camera face-off.
Huawei's response to all this? It's a bit strange.
In a statement to Android Authority, the Chinese phone manufacturer declared that it has a special “performance mode” that is available when its “intelligent software” detects certain programs, like a benchmarking app. However, this mode is not available for users, the company says: “Huawei is planning to provide users with access to ‘Performance Mode’ so they can use maximum power of their device when they need to.”
So the so-called performance mode right now is only available to benchmark software and not for end users? Why is that? It all sounds like corporate baloney trying to justify the facts: Huawei cheated.
What can you trust?
The message here: don’t trust any synthetic benchmarks at face value. Just wait for the reviews and see if the phone or PC you want to buy runs what you want to run as fast as you want. Apply the same to any other features, from camera quality to display performance.
Tom’s Guide does post synthetic benchmark numbers, but it also runs its own real-world performance benchmarks, such as a video transcoding test, and it measures battery life via its own web surfing test.
Experience is the only benchmark that, in the end, really counts.
