How to open a can of Whoop-A** on Vonage problems

Posts: 8 · New Forum Member Joined: Apr 08, 2006 Posts: 8

I am a long-time Vonage proponent..have used it as my main business line for nearly 3 years...and I use it a LOT...at least 2 hours/day from my office here in Atlanta calling all over the US and Canada.

Since about Jan 2006 my service took a sudden turn for the worse...about 25% of my calls experience "drop-outs" lasting from just an annoying fraction of a second to sometimes 5 and 10 seconds. With all the infrastructure changes that Comcast is making (e.g. adding digital simulcast) and the resulting cable TV issues (massive pixelization of 10 channels for 6-8 WEEKS), I had presumed that these problems were due to a Comcast plant issue.

By Mid-March I was able to get Comcast to (largely) resolve their issues, but despite this my Vonage problems remained the same...as if I don't have my own business to run I decided that was time to open a can of whoop-a** and get down to business on resolving the problem...there is no way I'm going back to Bellsouth Wink

.

As it happens my busines is doing network and security forensics...I've basically run around with packet analyzers for almost 20 years...now mind you I'm not much more than a novice troubleshooting Voip (SIP, RTP, etc..) protocols, but I figured that it's the only kind of systematic and detailed process that will unearth the root cause of complex problems.

Not to be too degrading to some of the troubleshooting data was bantered around in the long-running "Comcast v. Vonage" thread, but most of the testing techniques have almost NOTHING to do with the things that a Vonage call depends on:

http://www.vonage-forum.com/ftopic11377.html

For example:

* Ever single Vonage phone call likely involves a *different* Vonage server...if you are having a problem you need to know which server you call was being handled by *at that time*

* The Vonage web site (www.vonage.com) and the IP it hosted on has NOTHING to do with call setup and audio streams. OK, maybe there were some problem reaching their web site at one point, but I expect those problems were completely independent from other voice quality problems that anyone is having

* Issuing Pings and Tracerts from a Windows system generates ICMP packets that are generally 64 bytes in size...this traffic is VERY different than the 250 byte UDP packets used for voice streams (aka: RTP - Real Time Protocol)...also many devices specially will not respond to pings (e.g.: Vonage RTP relays)..therefore it is NOT a good tool to use when trying to troubleshoot Vonage issues

* Yes, upstream throughput is critical to good Voip quality...depending on your bandwidth settings a Vonage call requires about 80Kbps. Doing a speedtest (e.g. DSlreports and the like) are typically do simple *TCP* throughput tests..presuming that Comcast was playing some games with QoS of RTP packets (UDP!!) doing a TCP throughput test probably wouldn't detect it.

Anyway, here's an example of utilizing packet analysis (Ethereal) and UDP Ping tools (PingPlotter) to REALLY diagnose Vonage problems:

http://www.mynetwatchman.com/KB/NetKb/Vonage/AtlantaRTP.htm

My conclusion from the above analysis is that 30% (3 of 10) of Vonages Atlanta RTP Relays have some kind of problem that results an average of 10% packet loss *on ALL calls routed through them*.

This is really a guess, but I bet that someone forgot to hard-code the Ethernet speed and duplex settings on their network adapters and switch ports they are connected to...this often results in a duplex-mismatch where the server negoitiates to half-duplex mode and the switch to full-duplex...depending on network load this will results in extremly high packet corruption and loss.

For those wishing to replicate this level of analysis, I suggest you read through my "Packet Analysis for Idiots" guide here:

http://www.mynetwatchman.com/pckidiot/

...pay particular attention to Ch. 4 as most folks do not currently have their network physically configured to enable promiscuous monitoring of your Vonage traffic...you'll probably need to inject an Ethernet hub to act as a monitoring injection point.

As an aside... I see absolutely no evidence of Comcast de-prioritization of Vonage voice traffic...I DO see some occassional loss across their local backbone and ATT's backbone, but (at least here in Atlanta) it's negliable compared to the Vonage-specific infrastructure problem I've detailed here.

Posts: 2 · New Forum Member Joined: Mar 21, 2006 Posts: 2

Did you give your analysis to Comcast? What did they say?

My ISP was interested when I reported problems with a Qwest router based my results from the Broadband Reports packet loss tool. They even called me back three times to see if the problem was resolved.

They contacted Qwest and were told that the router in question was an "edge" router and woundn't respond to pings. Well somestimes it does, and sometimes it doesn't. I don't know what they did (if anything) but the problem (echo and dropouts) have pretty much gone away.

Posts: 8 · New Forum Member Joined: Apr 08, 2006 Posts: 8

inginear wrote:

Did you give your analysis to Comcast? What did they say?

Why would I contact Comcast? The whole basis of my analysis is that the problem is a local Vonage infrastructure problem. I *have* contacted Vonage and their engineering group is looking into it.

Posts: 6 · New Forum Member Joined: Apr 09, 2006 Posts: 6

Do any of you know a direct way of getting in touch with someone from Vonage tech support that is within 10 time zones of the USA? They all seem like nice people, but the thick accents make the language barrier steeper than the technology barrier.

After the "switch" to the 2nd phone port the VT2442 became totally unstable dropping internet connectivity every 10 minutes. The entire unit is packaged back in the original box waiting return to Vonage.

It's a shame, I believe their service could be decent, but you get what you pay for. I dialed the Verizon Voicewing tech support line on a whim twice today and got through in under 2 minutes each time. That's a far cry from the 20+ minutes to connect to India with Vonage.

Posts: 2 · New Forum Member Joined: Mar 21, 2006 Posts: 2

"I DO see some occassional loss across their local backbone and ATT's backbone...."

Posts: 1715 · Posted: Tue Apr 11, 2006 2:55 am Post subject:

scottvon wrote:

Do any of you know a direct way of getting in touch with someone from Vonage tech support that is within 10 time zones of the USA? They all seem like nice people, but the thick accents make the language barrier steeper than the technology barrier.

After the "switch" to the 2nd phone port the VT2442 became totally unstable dropping internet connectivity every 10 minutes. The entire unit is packaged back in the original box waiting return to Vonage.

It's a shame, I believe their service could be decent, but you get what you pay for. I dialed the Verizon Voicewing tech support line on a whim twice today and got through in under 2 minutes each time. That's a far cry from the 20+ minutes to connect to India with Vonage.

Don't let the VT2442 turn you off to Vonage -- it's been out for less than a month, so it really hasn't been tested yet. Now, I DO fault Vonage for not testing brand new devices better, but once you have a stable adapter, it's a great service, even if their tech support isn't.

Posts: 8 · New Forum Member Joined: Apr 08, 2006 Posts: 8

I got word from a Vonage engineer that they had isolated and resolved the problem...I re-ran my RTP stream analysis and PingPlotter...agree that now all RTP relays in Altanta look good. Updated info is appending to the bottom of this page:

http://www.mynetwatchman.com/KB/NetKb/Vonage/AtlantaRTP.htm

I would not be surprised if this same problem doesn't exist in other Vonage POPs...would encourage others to repeat this same procedure (remember you have to make *inbound* calls to your Vonage line). I believe when you make *outbound* calls your are handled by equipment that is close to the destination phone# you are calling.

It's quite ridculous that end-users have to go to this level of detail to identify problems, but it only shows that it enabled resolution of the problem in *days*...whereas I bet this problem would have NEVER been resolved anyway else.

Posts: 277

Impressive analysis, and your webpage detail does a great job of showing anyone who's at least semi-technical what you did and why. Thanks.

CyberCSI wrote:

I would not be surprised if this same problem doesn't exist in other Vonage POPs...would encourage others to repeat this same procedure (remember you have to make *inbound* calls to your Vonage line). I believe when you make *outbound* calls your are handled by equipment that is close to the destination phone# you are calling.

It's quite ridculous that end-users have to go to this level of detail to identify problems, but it only shows that it enabled resolution of the problem in *days*...whereas I bet this problem would have NEVER been resolved anyway else.

Yes, agreed - its likely that the problems in the upper midwest resulting in that huge Vonage vs. Comcast thread were something similar to this, I'm betting. Unfortunately, as you admitted in the summary at the end of your webpage, even IF I am a technically competent end-user who is willing and able to take the time and effort to do what you did, its likely going to be fruitless.

CyberCSI wrote:

Ok so impressed got that fixed pretty quickly...though had to use my own internal Vonage contacts...somehow I don't think reporting this through the standard channels would have ever resulted in resolution.

So what's the poor user to do? I'm capable of running your analysis, if my service turns to s**t, but I sure don't have those internal Vonage contacts, and I agree that trying to give this info to the inept Vonage customer service infrastructure is likely no more than an exercise in frustration.

I also agree with you that it appears that Vonage's engineering and network operating center personnel are amazingly and shockingly inept, not to have found this themselves. Its really a simple thing for them to monitor, and any network engineer worth his/her paycheck should have such monitoring setup on their critical infrastructure. I've been an IBM/Tivoli systems engineer and technical sales guy for about the past decade, and I could sure show them how to properly monitor and manage their infrastructure - but of course it would cost them some $$.

Hope you stick around here - we may need you to feed more of this info to those Vonage contacts of yours, since it appears that Vonage's customers are going to have to monitor and diagnose the Vonage infrastructure for them. :sigh:

Thanks again...