DSL Prime: Bell Canada's Claims

Canada's ILEC said that it was forced to throttle competition because of network congestion, but the CRTC now has the data to show that's not true. Of course, ILECs can break any law with impunity, so there will not be consequences. The company should pay for a network upgrade that it can clearly afford.

by Dave Burstein of DSL Prime and Future of TV [July 2, 2008]

Bell Canada: Minimal Congestion, Easily solved
Finding and fixing the problem
Conclusion first: Bell Canada's data show there are almost no problematic points of congestion, and those few are diminishing. There is a minor issue in about 5 percent of DSLAMs because they haven't upgraded the connections out the back for several years. The necessary upgrades should cost $4 million to $40 million, or $2 to $20 per customer. The bottleneck (and solution) is obvious when anyone who knows broadband networks takes a look at the data requested by CRTC Chairman von Finckenstein. That's really good news, confirming all the reports that any carrier large enough to have their own fiber backbone can inexpensively avoid congestion problems (except cable upstream before 3.0.)

The expertise of the Bell engineers is obvious from the presentation. They set the standards very high. If five out of 1300 + readings taken every 15 minutes are within 10-30 percent of peak traffic loads, the circuit is called "congested." That would be unnoticeable to the user, but is a sign trouble might be coming. That's the kind of reliability testing that limits downtime to 20 minutes a year, a good thing.

The CRTC asked for the right data, over time and broken down by the different parts of the network. Even at the standard of less than 1 in 200 readings coming even close to congestion, less than 1 percent of all backbone, BAS, or aggregation links had a problem in the last 12 months. In May '08, 0.2 percent of the backbone and BAS links reached the threshold. 0.9 percent of aggregation links triggered an alert. This is insignificant, probably less than 1 in 20,000 packets affected. The trend is down.

A possibly meaningful 5.2 percent of DSLAM links hit that 1 in 200 level, identifying a possible but almost unnoticeable problem. This startled me, because I hadn't heard of a congested DSLAM problem in years. Since about 2002, DSLAMs like the Stingers Bell Canada uses have been designed to be "non-blocking", with a backplane and network processor faster than any likely traffic from the homes connected. Bell clarified this for me. The problem wasn't the DSLAM, but the connection out the back of the DSLAM. Some of these ran at speeds of 45 and 153 megabits, which easily could explain this occasional problem. Since 2002, most carriers have put in Gig-E connections as standard, 5 to 15 times faster. Masayoshi Son told me in 2002 the Japanese standard was 4 gig-e's per exchange. Much of the BellSouth network was upgraded to Gig-E in 2004, per the CTO.

Lucent hasn't gotten back to me with what it would cost Bell to upgrade 400 exchanges to Gig-E. In quantity, similar units can be upgraded, including installation, for less than $10,000. That would be $4 million, or about $2 for each Bell Canada DSL subscriber. Alternately, simply upgrading the 45 and 153 Mbps ATM links to OC 12, four times the capacity, would probably be easy and doesn't require any conversion to IP. It's possible some of the exchanges do not have fiber available, and the increased traffic would require modest upgrades in other parts of the network. I can't imagine that costing more than $100,000 per exchange, all included, or a total of $40M. I've asked Bell for more detailed information to pin things down, or a CRTC engineer should be able to cost things out pretty closely in a few days. The work itself might take 6 months, because there are sure to be details to determine from each exchange and engineering diagrams to check, but is very standard stuff done thousands of times a year.

If that $4 million to $40 million seems surprisingly low, you're forgetting how far the cost of DSL equipment has dropped. Brand new DSLAMs for Bell's 2 million customers would cost about $40 per customer, or about $80 million total—to replace the entire network! They would run on average twice as fast, use much less power and space, and bring operating costs down. France Telecom converted nearly all the exchanges to the more recent ADSL2+ models about three years ago without difficulty. That's the upper limit on likely costs, and the upgrade should be closer to $4 million to $12 million than $40 million to $80 million.

For a service with an annual revenue around $400 per customer, and a margin upward of $200/per year per customer, $2 to $20 per is an insignificant amount to make the system work as advertised. In fact, nearly every other major carrier makes these upgrades as part of their routine maintenance. Bell had a logical reason to stop upgrading, however. Their plan three years ago was to move most customers to terminals closer to their homes (FTTN) which would allow selling a video package. That now delayed plan would have shifted much of the traffic to the new equipment. allowing the old gear to serve the residual customers. It is also probably significant that Bell cut capex in Q1 by 25 percent.

It has not escaped our notice that this is substantially different from statements in Ottawa.

Bell Canada Lesson: Find Bottleneck. Fix
Congestion can be avoided
Bell Canada data make clear congestion (except at 9/11 style peaks) is not a natural process in today's networks and can typically be solved once the cause is identified. This is not news; Comcast, AT&T, and Verizon have gone on the record in multiple official FCC filings they do not have any congestion problems behind the DSLAM/CMTS, and SBC CEO Ed Whitacre testified to that effect in the U.S. Senate. Let's apply that lesson to different real world scenarios.

U.S. cable often does not have enough upstream capacity, and the effort to solve that through degrading p2p is a major political struggle Comcast is losing. I've seen data that confirms to me the problem is real although overstated. The natural solution is to increase upstream speed as quickly as possible. Fortunately, DOCSIS 3.0 is coming quickly, with at least a 12x increase in speed, and a 30x increase at many nodes. Comcast is doing 20 percent in 2008, and essentially all their 20M homes by 2010. There's no economic or technical reason the other cablecos can't do similar a little later. Takeaway: DOCSIS 3.0 offers an easy technical fix to raise capacity enough for the nest 5-10 years. Make it so.

Small British ISPs are getting clobbered by what they have to pay BT for backhaul. Larger ISPs like Carphone and Sky are solving the problem by buying their own backhaul network and cutting costs 80 to 90 percent. The dozen smaller ISPs competing energetically for the U.K. retail market could easily be squeezed out of business. Britain's most likely future is dominated by 4 carriers (BT, Virgin, Carphone, and Sky) unless the cost squeeze on the others is resolved. For BT itself, traffic rising at a normal 30 percent per year per customer should be no problem, because the actual cost of moving bits is going down with Moore's Law at a similar 30 percent per year. ISPs paying BT for backhaul are seeing little or none of that 30 percent per year drop, as OFCOM has allowed BT wholesale to set prices unrelated to incremental costs. One natural solution is to make sure BT's price to independents stays in line with incremental costs, which would drop it 20 to 40 percent each year. Carphone and Sky could spread their costs over a larger volume by reselling to the ISPs, and be profitable at half BT's price. Newspaper reports had Vodafone cutting a deal like that. The alternate providers of fiber between offices could become more efficient and also offer a service. Takeaway: Incumbents (and others controlling the backhaul) can distort markets and create an unnatural bottleneck.

NTT Japan recognizes a theoretical challenge if users demand extraordinary bandwidth. They have just imposed an upstream cap of about 900 gig per month. Customers may still draw terabytes on the downstream. That's perfectly fair bandwidth is cheap, but not free. That NTT sees no problem up to 900 gig and Comcast up to 250 gig implies that companies like Rogers with 40 gig limits are not doing it because the traffic is unsustainable. FCC Commissioner Rob McDowell speaks about carriers trying to protect their own video sales. Takeaway: Caps half or a tenth of what other companies set are likely a cover for a price increase or directly anti-competitive.

Copyright 2008 Dave Burstein.
The DSL Prime Newsletter is reprinted with permission.

"The power of the printing press belongs solely to those who own the presses"
—A.J. Leibling

The Internet is the cheapest printing press ever invented.