The Technology to Run a Massive Mail Operation

In a chat with the CTO, we learned about the technology used to manage 30 million mailboxes.

by Alex Goldman ISP-Planet Managing Editor [January 16, 2007]

There are very few companies with e-mail operations as large as San Jose, Calif.-based Everyone.net. Founded in 1998, the company claims to run over 30 million mailboxes for about 300,000 domains. That's fewer mailboxes than Hotmail, and fewer domains than, perhaps, the top 10 webhosts worldwide. Perhaps the most similar competitor is Hong Kong-based Outblaze. Outblaze and Everyone.net specialize in one thing: managing mailboxes. Outblaze claims 40 million mailboxes and 400,000 domains.

Everyone.net's not alone, but it's got a very impressive peer group.

So, given an open-ended conversation with Wayne Lewis, CTO of Everyone.net, we jump at the chance.

Technology
Our first question is broad. We ask what technology the company used, expecting Lewis to talk about hardware. He doesn't.

"One thing we did really well early on," Lewis says, "is that we embraced Web Services, as early as 1999, before there was any standardization. There was no SOAP, for example."

As a result, much of the company's back end software is its own. "We have home grown stuff, built to solve common problems. For example, for our customers, messaging is just one piece of an overall solution. Our customer is probably not just a pure e-mail provider, and our solution is probably part of a bundle. The customer may provide connectivity with e-mail and a calendar, or may provide e-mail with webhosting."

The solution uses APIs to work well with whatever the customer needs. "We have to be able to play with billing systems, messaging systems, portals. Although we have some direct sales, our volume comes from our private label solution, so we're constantly improving it. In a typical month, we'll add two or three new capabilities while maintaining 100 percent backward compatibility."

Currently, the company is improving its AJAX-based interface.

Hardware is irrelevant
So do you use a SAN? What hardware do you use? "We are storage agnostic. We use multiple storage vendors. Our application is specifically designed not to care. We can work with NAS at the file level and with SAN on the block level."

The software has to be flexible because it's deployed in multiple locations. "It's hard to become large in the e-mail space with a single instance, a single deployment. The vendor will always pitch the scalability of their product, but you have to solve scalability at the application level, not on hardware."

Instead, the company focuses on using hardware efficiently. "At the end of the day, it's about how many I/Os you can do on any system. About two years ago, we embarked on significant changes to how we do inbound mail. One goal was to absolutely positively eliminate as many I/Os as possible. We developed what we call our Single Path Pipeline. Now we perform one write of an e-mail over its entire lifetime."

So you're like Google? "Google, and Amazon with their S3 service, have learned this lesson. The larger you get, the more you recognize the value in taking commodity storage and putting intelligence into your app to achieve five nines reliability, even though the hardware components will never achieve five nines reliability."

Lewis takes the opportunity to explain why ISPs should use his service. "Economies of scale kick in when you get big like we are. The typical service provider will not have the expertise to put together a solution like this. You need in-depth knowledge of how disks and storage work. Service providers want an organization with expertise in e-mail to ride the wave of innovation instead of working on plugging in the innovation themselves. Five years ago, you could get by with a basic Postfix server, but the bar has since been raised."

Security
Another challenge must be security? "Security is the part of e-mail that's difficult to predict. How much more spam will you get next month versus the amount you received this month? The other parts of the solution are more predictable. This is where a good architecture comes into play. We have multiple tiers of filters, each more CPU intensive. The top tier does a quick and early detection, and feeds the results into the more intelligent filters. The result is that we provide consistency on how much mail hits your backend. For a service provider going it alone, if you receive double the spam, you probably have to double the cost of your perimeter defense. We're charging on a per mailbox basis, so if spam doubles next month, that doesn't change what you pay us."

So what methods do you use? "We do a number of things. We bring in third party solutions. For example, we use a third party solution for content analysis. But 80 plus percent of the spam we stop is the result of algorithms we developed and implemented into our pipeline, like spamvertized URLs, IP volumes. The pipeline is designed to be flexible, even on a per customer and per domain basis."

So, in the event of a flood, there are software knobs you can tweak to increase the level of protection? "First, some automatic things kick in. The front end is always looking at the underlying performance of the back end, and it can take action in an automatic fashion. The operations team does have knobs for manual improvements. For example, we can do greylisting. When we know there's an attack, we can treat some mail differently. In the worst case scenario, some legitimate mail will be delayed, but never blocked."

So if you already have these capabilities in your system, and a full time operations staff, you don't need a company like everyone.net. But if the floods are overwhelming, it's time to outsource or upgrade.

— End