As Peter Scholtes once said: "In a world without data, opinion prevails." In IT, as is the case elsewhere, this may become a dangerous adage. Especially when it comes to comparing products with characteristics that can be measured in some way, for which benchmarks may vary radically when they are ill adapted.
Take the antivirus for example.
Between one antivirus and another, there may be common, known and easily comparable characteristics: operability of the solution, vendor (or integrator) client support and even the sustainability of the company manufacturing the product. As for the unique characteristics of an antivirus, they are much harder to gauge - they relate to the product's core business, meaning the service it will provide to the client. In information security, service is most often defined by the antivirus's ability to block threats. But how do you evaluate this aptitude in relatable terms? Through benchmarks?
Benchmarks for antiviruses are designed according to the properties of conventional antiviruses - meaning that they compare the ability of the antivirus's signature bases to block threats, an honorable approach, but... reactive. Signature bases tally threats that have already been identified. So if a threat shape-shifts and strikes, it will no longer correspond to any entry in the signature base, which will not alert the antivirus which in turn will be completely helpless in preventing any damage.
It is exactly for this reason that today's malware programs constantly mutate: as long as their perpetual and myriad variants cannot be identified by conventional antiviruses, they will keep getting bolder. In this case, how can benchmarks be trusted if they don't take into account the reality of attack methods? Who can be trusted when current tests keep applauding antivirus vendors for their performance while the ease with which malware and ransomware break into systems continues to make headlines?
Let's shift the focus: if a benchmark relies on reactive methods, how do you evaluate proactive security solutions?
Because proactive approaches exist to protect endpoints. Their technology strives to identify a threat not based on its form, but rather on its behavior. For several years now, antiviruses have gradually been integrating behavioral mechanisms. This is indicated in most vendor roadmaps, bearers of promises to come. But the future is of little comfort (can the client wait?) when threats of all shapes and size continue to be newsworthy. Comparing to-be-available product offerings becomes mission practically impossible. And if you had to give an opinion on how well they work, well...
While waiting for benchmarks to evolve, while waiting for antiviruses to go back to being an unbreakable barrier, while waiting... Caution is the mother of safety, so let's not base our purchasing decisions on a roadmap or someone's opinion. Let's face the facts and what today's trusted, proven solutions have to offer.
