Comcast, outbound dropouts, questions - Resolved

Posts: 114 · Last edited by outrun on Mon Apr 24, 2006 11:48 am; edited 1 time in total

Warning: Long ramble...

I've posted stuff here before about random outbound traffic loss through Comcast. Here's a summary, some updated Comcast support experiences, and questions.

Our Vonage/Comcast calls never entirely drop. The quality, when it works in both directions, is great. Randomly, our Comcast connection will lose outbound traffic, meaning callers can't hear us, but was can hear them. This "symptom" can last between 4 and 16 seconds (as noted through PingPlotter history).

Before the Comcast tech came out, I disconnected Vonage, my wireless, etc. I hooked the cable modem directly up to my PC to eliminate any possibility of Comcast blaming 3rd party equipment.

I had a Comcast tech come out yesterday to look at this. I showed him the logs of PingPlotter going to download.com and yahoo.com. I showed him the identical times that both dropped out. I explained to him that these drop outs, when the Vonage router was hooked up previously, were always outbound traffic issues - we could hear the caller, the caller couldn't hear us, and the conversation would normally resume once the "red line" ended.

He tested the signal, and it was fine. About two minutes after he arrived, the connection died, but he wasn't running any of his own "tests". His own test is to ping -t comcast.net. He said that PingPlotting download.com and yahoo.com isn't good enough since it could be an outside-of-Comcast network issue. Phooey. Looking at the hops, most of them are Comcasts (attbi and attens). Only the last hops are yahoo and download.com. When looking at the history of the dropped outbound traffic, there was 100% packet loss across the board (starting with the 1st hop). He wasn't convinced for some reason.

He unscrewed the coax from the wall, looked at it, blew at it, and plugged it back in. He unscrewed the coas from the cable modem, and noticed the wire was coming out way too far. He cut it off and put another fitting on it, hoping that was the cause of the issue. About 30 minutes after he left, it happened again, and has happened about 30 times since then. So much for the magic pill.

I called Comcast support again. The lady I talked to this time was trying to pawn this off to Comcast Phone support (I mentioned Vonage VoIP during the call). I explained the issue about 10 different ways, another tech is coming on Tuesday.

This is where my question comes in.

Looking at PingPlotter (plotting ANY ip address), I see this IP address as the first hop:
73.171.8.1

She said this was something between my PC and cable modem. When my outbound loss happens, all hops get 100% loss, including this IP. The only thing between my PC and cable modem is a nice blue ethernet cable.

My question: Is 73.171.8.1 an ip of a cable modem, a Comcast piece of equipment, or something else? When the tech came yesterday, he went to 192.168.100.1 to look at the cable modem status screen. I have a feeling that the 73 address is some sort of neighborhood switch that is failing. There isn't a DSNName listed for it, nor the next hop (68.87.157.21). The next hop is 68.87.144.237 (Comast in Milford, MA).

I'm consistently getting failures of about 40-50/day, all for between 4 and 16 seconds.

I'm now without Vonage service (or wireless connection) until Tuesday while I continue to monitor the network without any other devices connected.

Any thoughts on the 73.171 address?

Thanks,

-Craig

Posts: 114 · Posted: Wed Apr 05, 2006 9:20 am Post subject: Update...

Comcast came out again (yesterday) with a different tech. Before he came out, I ran a whois on that 73 address. Comcast owns it.

This guy stayed for about an hour, and changed each and every coax plug from the street all the way to the cable modem, hoping that one of the connections was the cause of the outages.

The signal got a little better (which was already high). After PingPlotting comcast.net for 10 minutes, there was a 6 second outage. Luckily, this happened while he was there and looking at it. He scratched his head, and said he would talk to his supervisor about this tomorrow. He also mentioned that Comcast will send a "regular" tech for calls like this twice. On the third time, they'll send a more specialized (he quoted 10+ years experience) tech who will be scheduled to say for "hours" and monitor the line and figure it out. The "regular" techs have a full schedule and can't stay that long.

So, I'm going to call them back later today or tomorrow (after he talks with this supervisor) and schedule a longer-term appointment.

After monitoring the line with PingPlotter for the next 6 hours, there were 7 more outages.

The saga continues...

-Craig

Posts: 26 · Vonage Forum Junior Joined: Feb 28, 2006 Posts: 26

Are you located in the Midwest, Craig?

Posts: 114 · Posted: Thu Apr 06, 2006 9:37 am Post subject:

Nope (I've read most of that thread). I'm about 30 minutes west of Boston, MA. This issue really doesn't have anything to do with Vonage, but the symptom affects a Vonage call. I've disconnected my RTP300 for a few days at a time, monitored the network, and still see the dropouts.

I've posted this in case anyone else has the same experience - frequently can hear the caller but the caller can't hear you.

-Craig

NJITgrad wrote:

Are you located in the Midwest, Craig?

Posts: 35 · Posted: Thu Apr 06, 2006 9:34 pm Post subject:

95% of the time, outbound audio dropping out is related to packet loss from the ISP. It's frustrating to try and get this fixed because the ISP will say there is no problem, even after you make a fuss. Most people will blame Vonage when it really is an ISP problem. Unfortunately because of the nature of it, 90% of normal web surfers/email checkers will not notice the problem and never complain about it. As soon as you start doing something thats real time with UDP (online gaming, voip, video phone, etc) is when you will notice it right away. Since voip uses RTP which is UDP and does no error correction like TCP packets will do, the audio will drop off and be lost forever to the person on the other end of the call every time packets drop to the ISP.

Posts: 114 · Posted: Fri Apr 07, 2006 9:07 am Post subject:

Exactly. It never crossed my mind to call Vonage, as this is entirely a Comcast issue.

I called yesterday and scheduled the "next tier" tech to come out and look around the neighborhood for an issue. The support guy said that he'll stay until it's resolved.

We'll see.

-Craig

vidmaster wrote:

95% of the time, outbound audio dropping out is related to packet loss from the ISP. It's frustrating to try and get this fixed because the ISP will say there is no problem, even after you make a fuss. Most people will blame Vonage when it really is an ISP problem. Unfortunately because of the nature of it, 90% of normal web surfers/email checkers will not notice the problem and never complain about it. As soon as you start doing something thats real time with UDP (online gaming, voip, video phone, etc) is when you will notice it right away. Since voip uses RTP which is UDP and does no error correction like TCP packets will do, the audio will drop off and be lost forever to the person on the other end of the call every time packets drop to the ISP.

Posts: 8 · New Forum Member Joined: Apr 08, 2006 Posts: 8

outrun wrote:

Looking at PingPlotter (plotting ANY ip address), I see this IP address as the first hop:
73.171.8.1

Open a DOS window:
> ping 73.171.8.1
> ipconfig/all
> ping <default gateway from ipconfig>
> arp -a

arp -a will dump out your arp cache exposing the Ethernet hardware address of any IP address you have talked to. You should find that the 73...addresss AND your default gateway have the same exact Ethernet address (aka: Mac address) as the 73 address is similar a secondary IP for that device.

So for example in my case it's:

00-0b-bf-52-30-8c

Next look take the first 3 bytes of the Ethernet address (this is the VendorID portion) and do an OUI query here:
http://standards.ieee.org/regauth/oui/index.shtml

You should get:

00-0B-BF (hex) Cisco Systems
000BBF (base 16) Cisco Systems
80 West Tasman Dr.
SJ-M/1
San Jose CA 95134
UNITED STATES

In conclusion the 73. address is a secondary IP for your default gateway which is a Cisco CMTS (Cable Modem Termination System)...this is the transition point between the cable plant and a standard IP routed network.

If you are losing connectivity to the CMTS the fault domain is huge..it could be your Ethernet card, the cable modem, anywhere in the cable plant, or the CMTS itself.

I would suggest you figure out how to pull diagnositcs on your cable modem so that you can see if it's providing any info that might shed light on the problem.

For example, my Linksys CM is constantly logging stuff like:

2006-04-09 07:22:09 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-09 04:49:12 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-09 04:07:54 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-08 23:54:08 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-08 22:22:40 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-08 22:14:43 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-08 21:47:22 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-08 20:18:11 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-08 17:22:07 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-08 17:20:27 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out

Also, if you put a second Ethernet adapter in your PC, assign it an IP that is in the same network as your cable modem (e.g. probably 192.168 something), put a hub/switch between your CM And your PC, then plug the second Ethernet card into the hub/switch...then you'll be able to ping the cable modem itself.

For examples on my PC, my second NIC is 192.168.1.100, my CM is 192.168.1.1..and here's my ping results:

C:\>ping 192.168.100.1

Pinging 192.168.100.1 with 32 bytes of data:

Reply from 192.168.100.1: bytes=32 time<1ms TTL=255
Reply from 192.168.100.1: bytes=32 time<1ms TTL=255
Reply from 192.168.100.1: bytes=32 time<1ms TTL=255
Reply from 192.168.100.1: bytes=32 time<1ms TTL=255

Posts: 114

Thanks for all the juicy information to check on, but I will reduce this down to a few facts:

1. It's not my ethernet card on the PC. When connected to Vonage, the outbound audio drops. Since the connection hits the Vonage router before my PC, this has nothing to do with my PC. When I run PingPlotter, I ran it both in the Vonage-enable network topology as well as my PC in standalone mode (no Vonage router used at all). PingPlotter shows the same disruptions in both configurations.

2. I seriously doubt it's the cable modem. This is the 2nd cable modem that has exhibited this same behvaior. The first was a BestData CX110. I replaced it with a Motorola SB5120 about 2 months ago. I've been experiencing this behvaior for many, many months. The problem was what prompted me to switch cable modems (and get an WRT54G, and load a 3rd party firmware, and get a Hawking HBB1, etc).

3. I already hit the diagnostics page of my cable modem (an SB5120). It's at 192.168.1.100. There is nothing useful in there that neither I nor the last tech could see. Even during an outage, the status screen shows the connection as being fine since it only measures the downstream signal, not upstream.

I also tried what you said (regarding arp -a). I only see my default gateway listed. I'm using Win XP Pro SP2.

Thanks again for all the info.

The "next level" tech is coming tomorrow afternoon.

-Craig

CyberCSI wrote:

outrun wrote:

Looking at PingPlotter (plotting ANY ip address), I see this IP address as the first hop:
73.171.8.1

Open a DOS window:
> ping 73.171.8.1
> ipconfig/all
> ping <default gateway from ipconfig>
> arp -a

arp -a will dump out your arp cache exposing the Ethernet hardware address of any IP address you have talked to. You should find that the 73...addresss AND your default gateway have the same exact Ethernet address (aka: Mac address) as the 73 address is similar a secondary IP for that device.

So for example in my case it's:

00-0b-bf-52-30-8c

Next look take the first 3 bytes of the Ethernet address (this is the VendorID portion) and do an OUI query here:
http://standards.ieee.org/regauth/oui/index.shtml

You should get:

00-0B-BF (hex) Cisco Systems
000BBF (base 16) Cisco Systems
80 West Tasman Dr.
SJ-M/1
San Jose CA 95134
UNITED STATES

In conclusion the 73. address is a secondary IP for your default gateway which is a Cisco CMTS (Cable Modem Termination System)...this is the transition point between the cable plant and a standard IP routed network.

If you are losing connectivity to the CMTS the fault domain is huge..it could be your Ethernet card, the cable modem, anywhere in the cable plant, or the CMTS itself.

I would suggest you figure out how to pull diagnositcs on your cable modem so that you can see if it's providing any info that might shed light on the problem.

For example, my Linksys CM is constantly logging stuff like:

2006-04-09 07:22:09 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-09 04:49:12 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-09 04:07:54 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-08 23:54:08 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-08 22:22:40 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-08 22:14:43 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-08 21:47:22 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-08 20:18:11 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-08 17:22:07 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out
2006-04-08 17:20:27 critical Started Unicast Maintenance Ranging - No Response received - T3 time-out

Also, if you put a second Ethernet adapter in your PC, assign it an IP that is in the same network as your cable modem (e.g. probably 192.168 something), put a hub/switch between your CM And your PC, then plug the second Ethernet card into the hub/switch...then you'll be able to ping the cable modem itself.

For examples on my PC, my second NIC is 192.168.1.100, my CM is 192.168.1.1..and here's my ping results:

C:\>ping 192.168.100.1

Pinging 192.168.100.1 with 32 bytes of data:

Reply from 192.168.100.1: bytes=32 time<1ms TTL=255
Reply from 192.168.100.1: bytes=32 time<1ms TTL=255
Reply from 192.168.100.1: bytes=32 time<1ms TTL=255
Reply from 192.168.100.1: bytes=32 time<1ms TTL=255

Posts: 8 · New Forum Member Joined: Apr 08, 2006 Posts: 8

outrun wrote:

Thanks for all the juicy information to check on, but I will reduce this down to a few facts:

1. It's not my ethernet card on the PC. When connected to Vonage, the outbound audio drops. Since the connection hits the Vonage router before my PC, this has nothing to do with my PC. When I run PingPlotter, I ran it both in the Vonage-enable network topology as well as my PC in standalone mode (no Vonage router used at all). PingPlotter shows the same disruptions in both configurations.

Ok, I agree that eliminates your Nic...so fire up ping plotter to the management IP on your cablemodem...if you lose (during audio drop outs) that connectivity then the modem is obviously resetting (even on the LAN side)...if that's clean then its something upstream, starting with the WAN side of the CM.

Posts: 114

CyberCSI wrote:

outrun wrote:

Thanks for all the juicy information to check on, but I will reduce this down to a few facts:

1. It's not my ethernet card on the PC. When connected to Vonage, the outbound audio drops. Since the connection hits the Vonage router before my PC, this has nothing to do with my PC. When I run PingPlotter, I ran it both in the Vonage-enable network topology as well as my PC in standalone mode (no Vonage router used at all). PingPlotter shows the same disruptions in both configurations.

Ok, I agree that eliminates your Nic...so fire up ping plotter to the management IP on your cablemodem...if you lose (during audio drop outs) that connectivity then the modem is obviously resetting (even on the LAN side)...if that's clean then its something upstream, starting with the WAN side of the CM.

Using PingPlotter against the SB5120 is something I haven't thought of - thanks for the suggestion. When I was doing tests last week, I kept hitting reload on the 192.168.100.1 webpage during an outage, and it kept refreshing, so my thought is that it wasn't resetting. PingPlotter may confirm that.

Small correction to something I said previously - the SB5120 is 192.168.100.1.

Over last night, the line dropped about 25 times. Unfortunately, I didn't read this until just now, so I've just started monitoring 192.168.100.1 as well.

I'll report back what I find.

Thanks,

-Craig