Sudden unexplainable severe drop of QOS

Posts: 4 · New Forum Member Joined: Jan 31, 2009 Posts: 4

Hello,
I am new to this forum but an early VOIP user and a networking communications engineer by profession. I searched for years for a high quality VOIP provider and finally found Vonage a few months ago as the best of them all.

Since I got Vonage I had excellent service! 2 weeks ago the service suddenly became choppy with bi-directional, split second drops/cutoff clicks at approximately 20 second intervals. The service became unusable because the opposite parties could not hear what I was saying. The only remedy: Powering down the VPortal. That lasts for 40 minutes then the problem returns.

I have Verizon FiOS, again, an early adopter - so my setup (which is an early format...) is an Ethernet cable from the Verizon ONT (Optical Network Terminal) to a Router (a D-Link DGL 4100 Gigabit Router) then out to a 16 port Gigabit switch. The Vonage VPortal is cabled directly to the D-Link Router - before the switch. This was always the setup and absolutely nothing changed before the Vonage QOS drop.
I made sure there was nothing wrong with anything in my premises and even changed the router to a new Netgear WGR614V9. And the problem recurred continuously.

I called Vonage tech support and the issue was elevated to the advanced team in New Jersey and they sent me a new VPortal - and the problem still continues, so I placed the Vonage device actually at the headend of the FiOS ethernet feed, before any router, and after it acquires an IP from FiOS - the problem still occurs.

Since I have addressed every aspect on the client side, including testing (using network diagnostic equipment) all elements on my premises - it seems to me that this is a Vonage network issue.

Any comments are welcomed.

Speed test statistics
---------------------
Download speed: 9382512 bps (This is slow...I have 15Mb service...)
Upload speed: 1703808 bps
Quality of service: 98 %
Download test type: socket
Upload test type: socket
Maximum download pause: 22 ms
Average download pause: 1 ms
Minimum round trip time to server: 47 ms
Average round trip time to server: 56 ms

VoIP test statistics
--------------------
Jitter: you --> server: 0.0 ms
Jitter: server --> you: 3.8 ms
Packet loss: you --> server: 0.0 %
Packet loss: server --> you: 0.0 %
Packet discards: 0.0 %
Packets out of order: 0.0 %
Number of supported VoIP lines: 28
Estimated MOS score: 4.1

Posts: 174 · Posted: Sat Jan 31, 2009 12:04 pm Post subject:

When you put the v portal in the front of the line did you log in to the web interface to ensure qos was enabled? If it is not then changing the setup around is in vain. You really don't need this setup if you can setup qos for voip on your primary router.

Posts: 343 · Vonage Forum Evangelist Joined: Dec 04, 2005 Posts: 343

You are hearing a "click" noise? What type of telephone(s) are you using and how do you have them connected? Have you tried a different telephone and/or different method of connecting them?

Posts: 813 · Vonage Forum Evangelist Joined: Jun 08, 2007 Posts: 813

I know you should not need to do this, but my experience with clicking noises was remidied by lowering the bandwidth settings (bandwidth saver) in the Vonage features page on the Vonage website.

Posts: 4 · New Forum Member Joined: Jan 31, 2009 Posts: 4

I've tried about 4 different phones and replaced the RJ11 cords for each. Same problem with all.
As to the bandwidth saver - I actually thought there may be an issue with that right at the beginning. From the minute I got Vonage, folks on the other end always complained I sounded "too loud". When the Hiccups began (really a better way to describe the click/drops...), I asked some networking techs I know and they suggested dropping the bandwidth. It didn't help...
QoS - good suggestion...Thanks...Tried...No improvement.

Posts: 646 · Posted: Sun Feb 01, 2009 1:36 pm Post subject:

I can assure you that this is NOT an issue with Vonage's network. If it were, each and every single Vonage user(2.6 million people including myself on multiple lines) would be having this problem.

Since you have already replaced the V-Portal and your router and currently have the V-Portal directly connected to the FIOS connection obtaining your IP, I would say the next step is to monitor your LAN activity.

You said your a network professional. Locate a hub or a switch that can go promiscuous and connect it to the LAN port on the V-Portal(assuming that your set up is still FIOS -> V-Portal WAN). Then connect your router to this hub and also a PC. On the PC connected to the hub run ethereal or wireshark and start sniffing you network while at idle and during phone calls. Look for any packets that are saturating your own LAN.

If after an hour or 2 you don' find anything odd, move the hub in between the FIOS connection and the WAN of the V-Portal. Connect you PC to the hub and start your packet sniffing session again. This time your going to look for something that may look like a DOS attack to your public IP.

I wouldn't have a normal home user go through this but since your a Network Engineer by profession, you may be able to pull this off and locate the culprit.

Posts: 879 · Vonage Forum Evangelist Joined: Nov 05, 2005 Posts: 879

VonTechMgr, about two weeks ago there was a firmware update for the V-Portal. Could this new firmware be the cause of his sudden issue? I use the V-Portal myself with the new firmware but I don't have any call quality issues.

Posts: 646 · Posted: Sun Feb 01, 2009 2:57 pm Post subject:

There is always potential when a new firmware build is deployed that some obscure issue could occur. However, chances are it would be affecting everyone using the same build of firmware. Since the V-Portal is the only device Vonage ships and there are ten's of thousands of them in production, and FIOS is a very popular connection type, I think there would be thousands of customers complaining if they were having audio issues that just started 2 weeks ago.

So while we cannot rule out firmware, it is highly unlikely.

Posts: 4 · New Forum Member Joined: Jan 31, 2009 Posts: 4

VonTechMgr,
Thanks so much for your very thorough reply and I will certainly do try some of your suggestions.
I have a few comments though:
1. There was a firmware roll out exactly around the dates my "hiccups" started and although your comment that many users should have experienced difficulties is, at core, true, there are also huge differences in the ways the Vonage service is implemented at each location. Some have differing cable modems, some have different ISP's, some have different bandwidths so on so on... so it is conceivable that after an upgrade just an unlucky few experience a glitch at a certain code locus. Our team's motto is that a software push should be looked at even if a single user may be affected...
2. As I said, I will try what you advise ( Promi hubs are hard to come by these days...) but can you clarify why should I test the LAN if I completely isolate the VPortal from the LAN and it's elements by just having the FiOS ethernet signal connected directly to the VPortal. All I have is FiOS, the VPortal and a phone. No LAN. And the problem persists. So it's either issues with the FiOS feed (and tests show there are none...), or the VPortal hardware (and Vonage replaced that...) or the phone (and I tried a whole bunch of them...). Problem persists.

Posts: 41 · Full Forum Member Joined: May 22, 2007 Posts: 41

Honestly there is probably just some congestion on a network segment between your local Verizon node and Vonage's network. ISP's change their network routes all the time which affects ALL traffic, including Vonage. Unfortunately there is little you can do in this situation but it happens.

I formerly worked in Vonage Advanced Tech support and let me tell you that this does happen from time to time. We previously had numerous problems with Comcast and several other regional cable providers who would not route the traffic well bewteen us and them.

The problem with VOIP is that you can take all the speedtests you want and still have audio dropouts and it won't be yours or Vonage's fault.

As for the suggestion of getting a hub you would theoretically be able to measure any packet loss or jitter of your Inbound RTP traffic but it really would not help with the outbound. You can play around with the different codecs but it really wont help that much in the case of bad packet loss or network routes. If you are going to change codecs though do not even bother with Medium (G726). This codec is fairly useless as it does not give you any significant bandwidth savings over HIGH (G711) and the quality is not that great either. Use NORMAL (G729) if you want as this will use very little bandwith and still has decent quality.