New Anti-Spam Benchmark

We talked to the testing outfit that just released an anti-spam benchmark about how the tests work and how often the software will be tested in the future.

by Alex Goldman ISP-Planet Managing Editor [May 2, 2005]

VeriTest, a subsidiary of Waltham, Mass.-based Lionbridge, released an anti-spam benchmark last week. The company tested a large variety of anti-spam products (not all, though), and provided the results to its paying subscribers and to the companies tested. The results will be made available for free later this year. The practice of releasing data first to paying subscribers is common in the research industry.

Steve Nemzer, vice president of certification and market testing programs at VeriTest, told us that the company has been actively tracking the ISP industry for a while. "We've had an ISP benchmark for about ten years," he said. "We have a background of providing qualified data to the ISP community."

He emphasized that the anti-spam test put the various solutions in a real world situation, not just a lab, with obvious benefits. "We had real time DNS lookup and the RBLs also were tested in real time."

The company did not test open source solutions. Nemzer says that VeriTest runs the solution in two configurations. The first configuration is "out of the box" and the second is the configuration that results from spending time with an analyst to tune and configure the software. Open source solutions are not designed to run out of the box and have no single recognized analyst to fine tune their performance. "If we just go with the out of the box performance of SpamAssassin, for example, that won't fairly reflect the performance of SpamAssassin because SpamAssassin needs tuning," he noted.

The company ran comprehensive tests, checking to see whether spam was categorized correctly. Using a lower cost labor force based in China and India for the grunt work, it was able to deploy human resources to check every decision the software made.

Scoring
The company awarded points to each solution based on its effectiveness, according to this table, with a 40 percent weighting to blocking spam and a 60 percent weighting to avoiding false positives, as most enterprises are more interested in avoiding false positives than in blocking spam:

Points	Blocking Rate	False Positive Rate
5	95-100%	0-0.5%
4	90-95%	0.5-1.0%
3	85-90%	1.0-1.5%
2	< 85%	> 1.5%

The chart shows the company expects an ideal solution to fail to block up to 5 percent of spam but to miscategorize as spam no more than 0.5 percent of e-mail.

The company translated the points into "stars" and a company can receive up to five stars in each category: out of the box performance and tuned performance. MessageLabs received five stars in each (however, as it is a hosted solution, there is, technically, no such thing as a mere "out of the box" MessageLabs offering).

The benchmark test will be run quarterly.

Nemzer says he's also working on a test of qualitative user experience for ISPs. "We would set up accounts at the ISPs and turn them into honeypots, doing everything wrong such as posting in usenet, posting the e-mail to web pages that can be crawled by spiders, and evaluate how well the ISPs screen out the unwanted spam that people have had to modify their behavior to avoid."

Pricing and availability
The anti-spam benchmark is available now for an annual subscription of $75,000 (for four quarterly reports) or $22,500 for a single report. Full subscribers receive additional benefits including access to sample e-mail streams, membership in an Anti-Spam Test Methodology Advisory Council, advance access to results, and even custom technical support from VeriTest analysts (up to 40 hours annually).

— End