Best of the ISP-Lists

Looking Upstream

You want to know how good the bandwidth you're depending on will be before you buy it.

[January 8, 2007]

On the ISP-Bandwidth list in December, MH wrote:

Without knowing the ins and outs of each carrier (which I would certainly like to work towards), what is a good measure of value (not price, not performance)?

To help compare providers, I took the number of peers on fixedorbit.com and divided that by $/mbit. I know its not perfect, but would someone offer me some constructive criticism?

I am referring to large providers that have a single AS spanning multiple continents with many POPs on each. I know that some smaller providers may provide a service with a great value.

One respondent used fixedorbit.com but noted that the site is often out of date.

[PT warned] "'peers' on fixedorbit.com, aside from being outdated and incapable of dealing with ANY ASN over 32767, is only a count of the externally visible neighbor ASNs. Could be a customer, could be a peer, could be a transit connection."

[DW advised] "You can use the route-views routers to determine peering if you know the command syntax."

CM recommended the following:

http://www.netconfigs.com/general/ranks.htm

[MH replied] "It'd be nice if they provided points they computed for that AS. Then one could use (Netconfigs points)/($/mbit) to determine a value."

CK provided an amusing equation for value:

The equation that we use for value is:

V = mx1*px3 / ( nx2 + rx4 + sx5 + tx6 + vx7 )

where

V = relative value

mx1 = cost per megabit based on actual commit

nx2 = average network latency in ms * 1000, based on www.internethealthreport.com

px3 = (quantity of upstream public peers + quantity of upstream private peers * 10 + quantity Tier1 Upstream carriers * 2 )

rx4 = quantity of spammers known to be on the network*1000

sx5 = (ratio of clueful network engineers to NOC staff) / average NOC response time

tx6 = (quantity of times Carrier has filed bankruptcy in last 8 years ) squared * quantity of money I have lost due to their BK

vx7 = quantity of incorrect bills * average discrepany/month * hours/month resolving billing issues

Then of course there is always the modification factor for the Richard Cranium level of the sales rep.

[AR replied] "I can think of a number of providers with open peering policies, and thus many AS adjacencies, who aren't particularly worth doing business with. Conversely, I can think of many vendors with selective peering policies (and thus relatively few peers) which I wouldn't hesitate to purchase from. When basing your decision, it is important to carry out qualitative—and not quantitative—assessments.

A good first step would be to perform a netflow analysis. Identify your network's "top n" source and destination ASNs, as well as any additional ASNs you might have a strategic need to access, such as perhaps inter-provider VPN or SIP termination.

Then, come up with a means of determining the quality of the prospective provider's connectivity to these networks, as either a whole or by cherry-picking individual peer routes (when doing the latter, it's important to also ensure that the provider has a sufficiently robust set of BGP communities at your disposal for the purpose of inbound traffic engineering). Active probes (a la smokeping and iperf) from key vantage points work to a point, as does experimentation with looking glasses and route servers, though neither approach paints a full picture. Increasingly common is SLA and/or RFP verbiage concerning inter-provider connectivity/capacity, though the examples I've seen were all fundamentally flawed, and generally weren't worth the paper they were printed on...

At the end of the day, nothing can top hands-on experimentation. If you're in a facility where you and the provider share a presence, go order up a cross-connect, and try the them out on monthly contract with little or no commit. Thirty (30) days is sufficient time to formulate some solid opinions on the quality of the provider's infrastructure, provisioning process, support, and overall whether or not you'd want to enter into a longer-term business relationship.

Equally important is chatting with a prospective provider's customers to determine how positive the overall experience has been, and if there are any particular "gotchas" to beware of. If you don't know any, this list might prove to be a good resource to tap.

Vendor evaluation is not for the faint of heart. Depending on the size and scope of your potential deployment, you might find it worth your while to hire a qualified professional.

It is true that some smaller providers may provide a service with a great value.

Lastly, while there's clearly no exact science or formula for this all, though I greatly enjoyed CK's attempt at creating one, which I've printed out and hung for future reference."

[SG added] "I just might add to this that there are many factors other than technical to consider also. While a carrier might have great interconnectivity or very uncongested loops, if your service is disrupted by a failure on their side and nobody can be reached to assist that's a problem! The asking other customers part can help get a bead on the customer service issues if any exist. Also things like time to deliver, billing accuracy, and responsiveness. When you add all these factors up it is very clear why you would never base a serious business on one carrier only and put all your eggs in one basket. Bringing as much diversity as you can technically manage (with in reason) helps off set any one bad carrier choice as well."

[HL noted] "One can also find and contact downstream ASNs. If downstream ASN count is too small for a valid sampling, look through netblocks whose origin ASN is the provider of interest."

[JJ concluded] "The best way to evaluate a carrier or colo is the search function on www.webhostingtalk.com where anyone and everyone is allowed to post the dirty laundry."

— End