DSL

DSL Prime News Weekly: The Inside Source

ISP failures hit CLECs and equipment makers, ISPs for sale, the rest of America is not easy to reach, thoughts on security, and more.

by Dave Burstein
DSL Prime
[October 19, 2000]

"Due to extraordinarily poor customer service on the part of ATT Wireless, involving intentionally hanging up on me, lack of any understanding of their services and endless failures to call as promised, I have decided to change my service to Verizon. My new number is ..." No surprise there — except the signature on the email was Dave Farber, Chief Technical Officer of the FCC. When your customer service department fouls up, you never know who they might be burning. Poor service is not just AT&T, of course.

We're staying by the phone (212-678-4788) and email today (October 17), as the the market reacts to Covad's startling announcement. "Demand has never been greater," enthused Bob Knowling of Covad, "the outlook for our company has never been better." But during the conference call, analysts broke away to tell investors "Sell! Sell! Sell! and drove Covad's price down 10%. By 3 a.m., Covad was down 30%.

The panic spread to Rhythms, NorthPoint, and Globespan in minutes, as the Internet gets bad news out fast. It could turn around by morning, but as we write the madness of crowds is turning against the industry.

The fundamentals of DSL are unchanged since before April — the biggest real problem is just serving all the customers looking to buy. Money was pouring in to internet stocks, and only conservative old fogies like us cared that profits were years away. Bill Schrader of PSInet reported "money is always there when we want it." DSL was hot, so every company in it would get rich — regardless of the number of players and how hard the competition looked to get.

Now, as funders panic, cash money isn't there to pay the bills.

$11.4M scary shortfall in Covad payments
Demand has never been greater
"Why didn't you say something earlier?'" they asked Bob Knowling in Covad's conference call. "How could you not know three weeks ago you had credit problems?" as a $25M increase in Covad receivables led to a $11M hit to revenue. The auditors apparently convinced Covad Monday not to recognize $11M in revenue, although Knowling is actively involved in each account and hopes to collect most or all. The reaction blew away his other news:

• Q3 accounts were at 205,000, and installing over 1,000 most days means they should easily pass the 270,000 (reduced) yearend guidance
• revenues per customer were holding up, as Bluestar direct sales kick in
• line-sharing is finally coming out of trials, with equipment in hundreds of offices and a good proportion of new orders going to the (much less expensive) shared lines, especially in 2001
• delays are down in Pac Bell territory, although the average customer stills waits a totally unacceptable 27 days
• only 30% of orders require a second truck roll. But that number may be distorted by a policy of simply refusing service in the first place to many customers not close to the COs, as some companies are doing.
• entire cities are turning cashflow positive, and most COs will turn cashflow positive in 18 months after install

Covad's $500M convertible offering, combined with Nokia vendor funding, will carry them into 2002. Nokia has apparently persuaded Covad not to switch to Lucent's Stinger, used by Bluestar. But equipment financing is getting risky.

Two-thirds of new orders are coming from a few key large accounts - Earthlink, UUnet, AT&T among them. This confirms Covad's previously reported retreat from all but a handful of ISPs, and they have been cutting off smaller ISPs even without credit problems.

Mindspring will aggressively promote a Covad consumer self-install program

Covad is not announcing office closing and layoffs, but some cutbacks are apparent to us. Wanna buy an ISP?
9 must be sold or financed
9 ISPs are in risk, and Covad yesterday decided to drop their revenues from sales, until they raise money or are bought out. Knowling said "I'm reviewing the offer sheets for two of the ISPs — we expect to get our money when the deals go through in the next month. Most of the nine will be funded or sold very quickly."

No names were given, but rumors swirl around every ISP on Covad's list — essentially, every ISP in the business. None have yet been making money on DSL, of course — business plans typically didn't call for EBITDA positive for several years. Covad's key security is control of the line to the customer; ISPs cannot switch to another DLEC without the prohibitive expense of running a new line. In one case, enduser's payments are going directly to a Covad lockbox system. "I control the customer. I will roll business accounts over to Bluestar."

One piece of good news we're personally glad of: "Flashcom pays their bills on time, and is not one of the 9." We were called during the due diligence of the last funding round. We included our warning about competition, but no one listens to that.

Knowling said the larger, faster-paying ISPs were getting priority in installs, improving but still badly behind. If that's enforced, they will have a killing advantage over the ones in trouble.

End of the CLEC war for residential?
Covad and NorthPoint had been backing away
Not discussed by Covad was that part of the financial problem of the ISPs was the elimination of the large hidden subsidy the DLECs had been providing the ISPs. As part of their goal of making numbers to satisfy Wall Street, DLECs had been spending almost $1,000 for each customer, partly in the form of market development funds to the ISP.

When Rhythms moved from a direct to a wholesale model, they cut some deals that were very attractive to lure ISPs. NorthPoint chose to be competitive, while Covad moved more to a direct sales strategy, exemplified by the Bluestar buy (which also, of course, bailed out their controlling venture capitalists.) That spending meant the residential customer could not possibly be profitable for years, but made sense because before April, Wall Street was valuing DLECs at over $5,000 per customer, a distortion unsustainable after April.

So Rhythms and NorthPoint signaled each other in the press to back off, and ISP payments were reduced by all. Covad and NorthPoint have been de-emphasizing residential (they officially deny it), at least until 2001. On the other hand, Rhythms' Catherine Hapka wrote us "We continue to see strong demand in the consumer market, especially with the advent of line sharing, which we have led the charge for." As Sprint launches 2100 COs and residential $45 service
ION gateways for the millions?
Although we broke the story in August that Lucent was heavily shipping Stingers to Sprint, we were startled by how wide they are going. Opening quickly will be Austin, Dallas, Denver, Houston, Kansas City, Los Angeles, Phoenix, San Antonio, San Diego, and Seattle. Sprint also offers broadband wireless service in Colorado Springs, Detroit, Houston, Phoenix, and Tucson. By yearend 2001, they expect their own DSL equipment to be in place in one-third of the US, with fixed wireless supplementing the rollout.

Installation and equipment is free with a contract. Complete packages, with 2 or 4 voice lines, cost over $100. Simply by converting a fraction of their LD customers, Sprint can become one of the largest DSL providers as fast as they can do installations. Like partially owned Earthlink (one of the fastest-growing DSL ISPs), they can convert existing customers at low cost, hence find profit easier to reach. Mindspring/Earthlink for now is a key Covad and telco ISP; no comment available on whether they will switch to Sprint going forward.

While Copper Mountain shows the impact on equipment suppliers
Good quarter, but sales headed down
Mike Kelly is retiring as chief of sales (Charles Niemann moves up) as Copper Mountain was pummeled after forecasting a sales drop because key customers will slow down equipment sales in the next few quarters. With Lucent's revenue shortfall (more in the switch business — DSL is a key growth area for them), and the CLEC crisis of the week, the financial problems of providers are clearly moving back the food chain. The market turned on Globespan overnight, the largest independent DSL chip vendor, although they are not the key supplier to either Copper Mountain or Covad.

Explaining it all: 45% of US has 3 DSL competitors in their CO
65% of homes passed by DSL — balance much harder to serve
Some remarkable research by Ed Pinkham gives the finest picture of the market we have seen. 3,700+ COs have DSLAMs, including nearly all the large ones. Rhythms, NorthPoint, and Covad have very similar 2,000 CO deployments, covering 35-45% of the country each, while the telcos have targeted the same COs. Only 10% of 16,000 unserved COs have over 5,000 homes, meaning further deployment will require more DSLAMs and expensive backbone connections. This explains why Knowling said Covad will be dramatically lowering capital expenditures, as they do not choose to install their own equipment. New Edge Networks, exclusive provider in many COs, has a key strategic position; it will be much cheaper to resell their service than installing your own equipment. We've previously made the economic case for telcos to serve every CO won't; competitors would probably slow down even without financial problems. More of Pinkham's research to come.

Where's the security?
Accelerated: put it at the customer
Whoever says DSL is secure is uninformed; a good hacker can crack most existing networks. Most of the skilled crackers have more interesting things to do than disturb your users, so most firewall alarms are amateur "script kiddies" and merely annoying. But the easiest way to bring a major website to its knees is a "distributed denial of service attack", and that software has been found on many DSL-connected computers. Corporations, in particular, fear a security breach, and are calling for protection. Shasta/Nortel offers a low-cost firewall in the system management unit, CheckPoint and Ramp recommend a sophisticated network system, Zone offers software free to the user, but Accelerated's Joachim Wallach thinks that a customer premise box, running at full line speed, and supporting IPSEC, the corporate standard, is the most reliable way to go. CPE security prevents certain kinds of attacks that a centralized system management device cannot, and with today's technology, can be affordably delivered by hardware that doesn't slow down the network or connection. His article is at

http://www.dslprime.com/a/dslsecurity_from_accelerated.html

Accelerated is rumored to have good news from Winstar in their quarterly call, and their security initiative will give them support in several key accounts.

WINfirst: Billion $ of Lucent fiber to the home
Competition you can't match
With $850M from Morgan Stanley, First Union, Madison Dearborn, and founder Denver's Jim Vaughn, WINFirst will be stringing fiber along side coax in Dallas, Austin and Sacramento. Initially, they are running 100 megabit Fast Ethernet to each home, but could easily go faster. High-speed data and voice over IP (2-4 phone lines) will be the first use of fiber, with switched video to come. Lucent's John Slevin described a completely Lucent network, from Nexabit terabit routers to a new IAD with voice, and a cost breakthrough based upon new equipment and use of a single fiber. Running a new network, fiber cost little more than copper; the hardware, like VDSL adds $400-800 to the cost today, but allows superior offerings and a path to the future.

Autoconfig — The Forum's on it
Forum looks for a working model for December
Standards bodies throughout the world have a reputation for moving slowly, but the DSL Forum is looking to change that by calling two special working group meetings to outline a standard for autoconfiguration. John Stephens is leading the committee, which is tasked to resolve some difficult questions.

Copyright 2000 Dave Burstein.
The DSL Prime Newsletter is reprinted with permission.

"The power of the printing press belongs solely to those who own the presses"
—A.J. Leibling

The Internet is the cheapest printing press ever invented.

—End