ISPs of all sizes are starting to confront a problem that used to only affect the largest: e-mail servers need to scale with demand.

[September 13, 2005]

On the ISP-Tech list in August, BC asked:

"I was just curious: from a technical standpoint, what are the reasons why you would want to have separate SMTP and POP3/IMAP servers?"

[JC replied] "We separate our incoming and outgoing primarily for virus and spam scanning. We have our domain names mx'd off to our spam/virus scanner, then it delivers to our incoming server via IP address. This way, even if one of our customers is virus'd up, they will be caught by our filters before they deliver to the incoming. Our incoming server does not accept any SMTP connections unless it is from our spam/virus server.

Another reason could be load issues. If you have 1000's of messages waiting in your queue, that could cause load problems if you are running a single incoming/outgoing server."

[AR disagreed] "I've never been able to figure out why, at least in the scale that most independent ISPs survive in, having separate hostnames and/or hardware was necessary."

[DB replied to AR] "Having separate domain names lets you scale up without having to require that users rename their POP3/SMTP/IMAP hosts, and makes it easier to scale.

E-mail systems scale both vertically (larger systems) and horizontally (more systems). Vertical scaling is expensive, which horizontal scaling is easier. Each of the above services can reside on a host of its own. To make it scale even more, there can be multiple hosts for each service.

SMTP servers are primarily limited by disk I/O and the number of messages flowing through the system. IMAP is primarily limited by the number of concurrent connections the system can handle. A/V and spam scanning gateways are CPU bound.

Splitting each service on its own hardware lets you tune the system optimally. With appropriate DNS records and proxies, you can scale horizontally without having to reconfigure mail clients (webmail or otherwise) for appropriate retrieval and submission hosts.

While this type of splitting is not important for small systems with a few thousand users, it is essential for larger systems."

BC liked JC's suggestion and requested more information:

"Would you mind briefly explaining how you guys set this spam/virus scanner solution up? I'd probably like to do the same here. We are getting bombarded with about 20,000 spam and viruses per day. We currently have everything running on one box, but I'd like to do something similar to how you guys did it."

[JC explained] "No problem.

First, we set up separate DNS entries for our incoming and outgoing mail servers. For incoming, it is just domain.com, for outgoing it is mail.domain.com. We mapped the domain.com to 1.2.3.4 and mail.domain.com to 4.3.2.1 for IP addressing.

Our Spam/Virus has a completely different domain, spam.com mapped to 5.5.5.5. For our main domain (domain.com), we setup our MX records in DNS to point to spam.com. The spam box accepts incoming connections for domain.com. The spambox (spam.com) scans the messages, quarantines accordingly and then delivers good messages directly to 1.2.3.4. It does not do an MX look to prevent MX looping problems (MX loops back to me).

On mail.domain.com, it does not have any hosts set up in the local-host-names file (sendmail). Also, it does not have any local mail boxes set up (actually, we don't even have a local delivery agent installed). When a client sends a message to bob@domain.com, the server mail.domain.com accepts it (based upon the access file rules for relaying), it then does an MX lookup for domain.com which returns the spam.com server and it delivers accordingly. The spam.com server processes the mail and delivers accordingly. Our setup does not scan outbound mail not destined for domain.com.

We have firewall blocks in place in front of our domain.com server to prevent any host(s) other than 5.5.5.5 contacting it via port 25. This way, when the evil spammer does an A lookup for domain.com and ignores the MX records, they get timeouts when attempting to connect.

The other item we set up was SMTP auth on our going server using SASLAuthd. To prevent duplication of the password files, we used SASL2 which allows for remote IMAP authentication for relaying (via sendmail). On domain.com, we have our IMAP daemon to authenticate locally.

Now, with all that said, the next question you will probably ask is what we are using for our Spam and Virus scanning. We outsource to Postini, but that part is not important and I am not including it to stir a debate about the best/worst/evilest spam/virus scanning companies out there."

JH concluded with some practical suggestions:

"I worked for an ISP that had started out with one hostname for both SMTP and POP3 (mail.domain.dom). Eventually, they grew to the point where they wanted to separate our outbound and inbound. For one thing, we had set up an external spam/virus filtering server to handle all their inbound e-mail, and we wanted to make sure that no one could connect directly to their inbound server and bypass that spam/virus filtering. Doing so required a separate server for outbound e-mail. The problem was that there were over 30,000 customers with one hostname for both servers—telling them to change would be a nightmare.

We eventually came up with a NAT solution. We set up an old spare Cisco router with our mail server's IP address and put the inbound and outbound mail servers on a private network behind it. Any connection coming in on port 25 went to one server, and connections coming in on port 110/143 went to the other. We also gave the Cisco a second external IP for the spam/virus scanner to use (inbound.domain.dom). We could have just made the spam/virus scanner use an alternative port (like 26), but decided it would be cleaner to use the second IP address. Additionally, we put an access list in the Cisco to allow the spam/virus scanner in, but not allow anyone else to reach the inbound mail server on 25. This allowed us to take the IP matching/filtering off of the Linux box."

—End