Plausible Deniability

Virus Bulletin, one of the antivirus/antimalware industry’s trusted sources on product testing, warned today that a number of leading AV vendors failed to detect known threats in their latest round of tests on the new Vista SP1 platform. Their labs report is fairly damning: McAfee, Trend Micro and Sophos all fail to get the VB100 seal of approval on the Vista SP1 platform, as did products from Alwil, BitDefender, Norman, PC Tools and VirusBuster.

They aren’t alone. Out of 37 AV products tested, we learn, 17 failed to obtain VB100 certification, which requires that they detect 100% of the malicious programs listed as “in the wild” by the WildList Organization. Companies must also scan a list of known clean files maintained by VB without generating any false positives.

Finally, the products that are tested have to do all this in their default configuration — no “optimizing” allowed. There are other tests as well, which you can read about here. They include performance tests and scans of other malware lists, etc. etc.

Of course, when you read behind the press release about popular AV programs falling flat on their face, there are a couple asterisks worth noting. First of all: the platform in question — Vista SP1 — was released shortly after the deadline for product submissions to VB. VB reviewer John Hawes - -a very smart guy — is up front about that fact that not every antimalware vendor was even able to get a copy of SP1 for testing before submitting their wares to VB for certification.

Also, given the large number of submissions (40), VB tester Hawes shelved those that couldn’t “provide usable results after the standard three install (attempts).”

Still, that didn’t stop VB from raising the alarm in its press release, with Hawes quoted saying that “Threats several vendors failed to detect in this test have been circulating in the real world for some months now,” and “it’s disappointing to see so many products tripping up over threats that are not even new–computer users should be getting a better service from their AV vendors than this.”

What’s going on? A couple things. With all respect to the folks at VB, who provide a valuable service to the antimalware industry, part of this is marketing. Certifications like the VB100 add prestige and visibility to the magazine and help with subscriptions, and nothing drives attention in the press like a little controversy. Ordinarilly, VB100 ratings wouldn’t get a mention. With some big vendors falling down, these just might.

Second, vendors chase after VB100 certifications and others like it because they’re something tangible they can point to in their marketing and product promotions, and because they seem to make something very fishy and subjective (the quality of antimalware products) seem concrete.

Is that really a bad thing? Well, frankly, yes. It is. Certifications like VB100, which are based largely on static file analysis have gone a long way towards sustaining the signature based detection model when others might serve consumers and enterprises better. True, most companies already do blend behavior and signature based detection methods, but companies that rely heavily on the former (like BitDefender) tend to do worse on tests like the VB100. Does that mean BitDefender provides inferior protection to a company like, say KingSoft, which did receive the award? Hardly, but the lack of certification still becomes a hook on which to hang competitive claims. Bottom line: you get punished for not using signatures, even if that’s the right or most effective thing to do.

It’s not as if nobody noticed these things before. I’ve been attending the annual Virus Bulletin Conference for years, and there’s always been heated discussion about testing methodology. As far back as 2003, in fact,the VB Conference had a panel discussion about anti-virus testing with West Coast Labs, ICSA Labs and VB. The conclusion: Wild List detection rates are only one measure of an effective antimalware product, but that time and resource restrictions limited broader testing.

However, as the malware problem has exploded in recent years, the problems with the industry’s narrow focus on signature matching became too obvious to ignore. At long last, changes are coming to VB and other testing organizations. For one thing, the industry-sponsored Anti-malware Testing Standards Organization (AMTSO) will soon release new guidelines for evaluating and certifying anti-malware products. Those standards are expected to radically reduce the importance of static file analysis (i.e. signature matching) in determining the overall effectiveness of antimalware products — pretty much the only thing keeping the signature game going. We also note closer cooperation between previously disparate testing labs such as AV-Comparatives, AV-Test.org and the Russian Anti- Malware Test Lab (AMTL) will also result in more uniform (and hopefully thorough) vetting of antimalware suites.

In short, new testing methods will end the illusion of competence that current testing models perpetuate. They’ll also raise the bar on malware detection for established vendors, and may very possibly reshuffle the ranks within the antimalware business. Stay tuned!