Chinese antivirus vendor Qihoo 360 cheated on antivirus evaluation tests, three independent testing labs charged yesterday (April 30).
"Qihoo 360 ... submitted products for comparative and certification testing which behaved significantly differently from those made available to its users and customers," said a joint statement issued by German lab AV-TEST, Austrian lab AV-Comparatives and British testing and certification body Virus Bulletin. "The three testing bodies will revoke all certifications and rankings awarded to the company's products so far this year."
In a statement posted on Facebook today (May 1), Qihoo 360 countered that all Chinese antivirus firms tweak their software for testing by Western labs because Westerners' online behaviors are "significantly different from those of Chinese Internet users."
Later yesterday, AV-TEST posted an update to its Facebook page that said a second, unnamed, antivirus vendor was under investigation for manipulating results.
MORE: Best Antivirus Software
"We believe the accusation and subsequent action by AV-C[omparatives] is without merit," Qihoo 360 said on its official Facebook page.
Qihoo 360's American representative repeated the statement that was posted on the company's Facebook page. Qihoo, which until this year had been stuck in the middle ranks of antivirus products, has done remarkably well in independent evaluations recently, taking first place in AV-TEST's latest round of Windows 8.1 tests, tied with perennial leaders Bitdefender and Kaspersky.
AV-Comparatives gave Qihoo 360 fifth place on its latest Windows 7 tests; had Qihoo not registered two false positives, it would have tied for third with Bitdefender.
Swapping in a ringer?
If you're seeing an unusual pattern here, so did the labs. They said Qihoo swapped out its own malware-detection engine and replaced it with Bitdefender's before it submitted the Qihoo 360 software to the labs for evaluation. Essentially, the accusation is that Qihoo rode someone else's horse to the finish line.
"If the products being tested aren't those being used in the real world, nobody's getting any useful information," said John Hawes, chief of operations at Virus Bulletin.
"Users rely on independent results to make an educated decision regarding their protection software," said Maik Morgenstern, CEO of AV-TEST. "If vendors start to manipulate the testing process, they are hurting everyone involved."
Tom's Guide uses AV-TEST's malware-detection scores in its own antivirus product reviews, but has never reviewed Qihoo 360.
Qihoo is in fact entitled to use Bitdefender's anti-malware engine; it openly licenses the Romanian company's software for use in Qihoo 360 Internet Security, its free antivirus product for the global market. AV-TEST evaluates both that product and its domestic Chinese equivalent, which uses Qihoo's own QVM anti-malware engine, which is called Qihoo 360 and is also free.
It was the Qihoo 360 domestic product that topped AV-TEST's latest evaluations and did well in AV-Comparatives' tests, and which the labs said was unfairly tweaked. (Other Chinese antivirus vendors also split their products between domestic and export versions.)
"As far as can be determined, all versions made generally available to users in Qihoo's main market regions had the Bitdefender engine disabled and the QVM engine active," the labs' statement said. "This would provide a considerably lower level of protection and a higher likelihood of false positives."
The labs said Qihoo told them that its domestic Chinese rivals Baidu and Tencent also tweaked their products before submitting them for testing. The labs countered that although Baidu and Tencent's products did behave strangely, "both firms were able to provide good reasons."
Cultural differences
Qihoo's explanation on its Facebook page was perhaps even more intriguing.
"AV-C[omparative]'s lab testing system is mostly based on the behaviors of European/Western Internet users, which may be significantly different from those of Chinese Internet users," the Qihoo page said. "Many popular software add-ons in China that are flagged as malware by the AV-C definition are in fact performing proper functions and not malicious."
The page did not specify what those add-ons might be.
"Qihoo 360 and other domestic vendors' security products in China treat such add-ons as legitimate and non-threatening," it added. "However, under AV-C testing logic, all domestic versions of security products in China are judged to be less effective since they fail to detect such 'threats.' This is certainly not fair."
"All Chinese security product vendors make modifications to standard domestic versions for foreign lab testing to showcase the effectiveness of the basic protection capabilities of these products," the page continued.
That might provide a clue to AV-TEST's own cryptic Facebook post about a second company cheating on the evaluations, apparently with somewhat different methods than those Qihoo 360 is accused of using.
"We have found strong evidence that another company, not Qihoo, is optimizing their product to do well in our performance test by excluding certain files and processes from checking," the AV-TEST Facebook page said. "This is based on filenames and process names and can pose a security risk as well!"
But another line in Qihoo's Facebook missive may only bolster suspicions that Chinese antivirus makers are forced to weaken their domestic products to accommodate their government's intelligence efforts.
"A security product that strictly follows AV-C's testing environment rule could be rendered useless in China due to the significantly different real-world environment," the Qihoo page said.
Lest American readers feel smug about this, this week U.S. prosecutors and government officials told Congress that Apple and Google need to weaken their mobile-device encryption standards so that police and other authorities can easily read private citizens' communications.
- 7 Ways to NSA-Proof Your Smartphone
- 10 Simple Tips to Avoid Identity Theft
- Best Wireless Home Security Cameras
Paul Wagenseil is a senior editor at Tom's Guide focused on security and gaming. Follow him at @snd_wagenseil. Follow Tom's Guide at @tomsguide, on Facebook and on Google+.
