WilTel/Level3 problems may cause of outage

Posts: 448

WilTel/Level3 are having big problems this afternoon, and last I knew, Vonage was a WilTel customer. Here are a few excerpts from the NANOG list.

"Anyone experiencing problems with Wiltel Backbone, or know of any issues with the Wiltel Backbone? I called their NOC and was told they are experiencing a nationwide routing problem that they are working on but couldn’t get any further details?"

"Told me it was related to the L3/Wiltel integration. Most of the breaks I've been seeing seem to be at the point where most of my traffic has be going from wiltel to L3 in DC or so. Oh, and that the "former wiltel" tier 1 guys had 40+ calls in the hold queue..."

"I can confirm this as well, although have no proof (at this point)
that Layer3 is necessarily to blame. We (or rather the company I
work for) are seeing similar between MCI/UU/Verizon and WCG when reaching some of our clients"

Posts: 25 · Vonage Forum Junior Joined: Jun 26, 2006 Posts: 25

thanks for the update but I personally have no clue what your talking about, maybe not that technically advanced yet.

Posts: 448 · Posted: Mon Jun 26, 2006 5:37 pm Post subject:

In layman's terms, if WilTel (which was recently absorbed by Level3) is having backbone routing problems... and if Vonage is a customer of WilTel... then it follows that Vonage may be experiencing network issues related to the WilTel issue.

Posts: 25 · Vonage Forum Junior Joined: Jun 26, 2006 Posts: 25

rebus wrote:

In layman's terms, if WilTel (which was recently absorbed by Level3) is having backbone routing problems... and if Vonage is a customer of WilTel... then it follows that Vonage may be experiencing network issues related to the WilTel issue.

ahhh... ok thanks!

Posts: 497 · Vonage Forum Evangelist Joined: Apr 29, 2005 Posts: 497

Why does the outage go in waves? Some people have service, others don't, then it switches, then back. It seems to me that everybody would be down if Vonage's connection to the internet was down.

I don't know how this all works, so if anybody is willing to educate me I'd appreciate it.

Posts: 318 · Posted: Mon Jun 26, 2006 6:23 pm Post subject:

rebus,

Time Warner/Brighthouse are also customers of Level3. This had the positive effect of my Vonage service being uninterrupted but the negative effect of most sites being inaccessible to me all day today.

Luckily, this problem appears to have been corrected as of right now...

Thanks for the info.

Posts: 448 · Posted: Tue Jun 27, 2006 11:10 am Post subject:

arcking wrote:

Why does the outage go in waves? Some people have service, others don't, then it switches, then back. It seems to me that everybody would be down if Vonage's connection to the internet was down.

I don't know how this all works, so if anybody is willing to educate me I'd appreciate it.

Vonage is tight-lipped about its connectivity, but I'm certain they have a lot more than a single circuit to a single carrier. When a carrier has routing issues, it's not always as simple as black/white, up/down. Routing problems can be transient (they come and go), or can manifest themselves in ways that degrade the delivery of data without a complete outage.

Routers exchange path information with each other, which is how they know which circuit to send data down, to reach a specific destination. One such protocol used for this type of communication is BGP (border gateway protocol). BGP does not tell its peers (other routers it speaks with) about every path to a destination, but merely the best path. When that "best path" goes down, a router will withdraw that route and announce a different route to its peers. When the downed circuit comes back up, the alternate route is withdrawn and the original "best path" is once again announced. When a circuit goes down and up, it is said to have "flapped", and if it does this more than a time or two within a specified interval of time, that "flapping" draws a penalty in that other routers will ignore ("dampen") that path for a while. Once that circuit has remained online without further flapping, other routers will once again pickup the route. Meanwhile, traffic normally carried by those downed circuits must be shifted to other links. Depending on how much traffic is shifted, the remaining links can become saturated, causing some regional packet loss along those routes. (Generally, major carriers have enough cross-country capacity such that the whole country isn't seeing widespread packet loss.)

That is an overly simplified explanation, but it illustrates the point of how a few circuits bouncing can cause wide-spread disruption. Add to that any problems caused by fiber cuts, hardware failures, etc. While I don't have confirmation on this, the rumor is there was a major outage on the legacy WilTel network. WilTel was acquired by Level3, and Level3 has been in the process of migrating those old legacy WilTel circuits into the Level3 network.

Posts: 497 · Vonage Forum Evangelist Joined: Apr 29, 2005 Posts: 497

Thanks rebus, that's what I was needing. I know some of how it works from a Security Now! episode on how the internet works, but didn't have the specifics.