Why your fancy router/firewall likely won't work with Comcast/Xfinity's IPv6

TL;DR - Comcast/Xfinity internet won't work with any routers or firewalls (like Cisco ASA-5500-X) using DHCPv6 in certain regions because the handshaking packets contain a hop limit of 0.  This violates RFC 2460 which is the IPv6 specification where firewalls and routers are supposed to discard any IPv6 packets that contain a hop limit of 0.  Comcast refuses to acknowledge or fix the issue and therefore anyone with a router/firewall that supports the spec will not be able to do DHCPv6 with Comcast as an ISP in regions that they send that invalid hop limit.

The Issue, the Tech, the Dumps

I have been a faithful Cisco user for many years as my primary router/firewall.  Their firewalls are certainly pricey, but you get what you pay for.  Ever since the Cisco ASA 5505 was deprecated they lacked any real IPv6 support. I decided to upgrade to one of their newer generation Cisco ASA 5500-X models.  The 5506-X hadn't been released and I trolled eBay to find a raging deal on an ASA 5512-X.  Its a fantastic firewall and have been happy.  The only problem was that even though it supported IPv6, it did not support DHCPv6 Prefix Delegation (PD) client.  This prevented me from being able to use IPv6 since that was what my ISP Comcast/Xfinity uses.

In August 2016, Cisco released a newer firmware 9.6.2 that they touted support for DHCPv6-PD.  So I decided to download and try.  I had difficulty finding anyone who was able to get it working, but found an excellent post by a guy who was able to get it working on Telstra (Australian ISP).  I tried out his configuration and got nothing.  So I put it away for a couple of months and then decided to try again in January 2017.  I set up the config, and I got nothing.  I tried newer versions of the firmware in hopes that it was a Cisco bug, but it was still to no avail.  So I decided to dig deeper.

How DHCPv6 works is that periodically, the DHCPv6 Server sends out a Router Advertisement (RA) a broadcast packet.  This tells your client about server routers and who you will communicate with.  Your DHCPv6, receives this, takes the RA and sends a Router Solicit (RS) from UDP ports 546 to 547 on the server where you ask for a prefix size (how many IPv6 addresses you will want to use) and DNS information.  The DHCPv6 server replies with an Advertisement XID which contains information on the prefix blocks you can use and the DNS information.  The DHCPv6 Client takes that information and sends back a Request XID stating that it will use a certain Prefix and various options.  The DHCPv6 then sends a Reply XID which contains the prefix that has been allocated to you.  The entire process looks like this:

Digging Deeper

Going through the Cisco docs, I can start looking at the debugging of the packets.  So I looked at debugging the Router Advertisements and found this:

router# debug ipv6 nd

ICMPv6-ND: Received RA from fe80::201:5cff:fe62:a246 on outside

ICMPv6-ND: Sending RA to ff02::1 on nlp_int_tap
ICMPv6-ND:     MTU = 1500
ICMPv6-ND:     prefix = fd00:0:0:1::/64 onlink autoconfig
ICMPv6-ND:          2592000/604800 (valid/preferred)

Looks good... Comcast is doing what they are supposed to.  I get the router advertisement. My RS packets appear to be set up to send out every minute and a half, so waiting, I noticed the RS packet being sent to Comcast's router.  So far so good, except I never receive back an Advertisement.

# sh ipv6 dhcp client pd statistics 

Protocol Exchange Statistics:
  Total number of Solicit messages sent:              474
  Total number of Advertise messages received:        0
  Total number of Request messages sent:              0
  Total number of Renew messages sent:                0
  Total number of Rebind messages sent:               0
  Total number of Reply messages received:            0
  Total number of Release messages sent:              0
  Total number of Reconfigure messages received:      0
  Total number of Information-request messages sent:  0

After going through the Cisco docs, it looks like I can set up an ACL to capture a dump of the traffic on the outside interface.  So off I go:

router(config)# access-list test_ipv6 extended permit ip any6 any6

router(config)# cap capout interface outside access-list test_ipv6
router(config)# show cap capout
...
7 Feb 20 2017 21:11:34 fe80::201:5cff:fe62:a246 547 fe80::f60f:1bff:fe76:fa57 546
UDP request discarded from fe80::201:5cff:fe62:a246/547 to outside:fe80::f60f:1bff:fe76:fa57/546

...

Whaaaa? A packet is getting discarded to the outside interface before I can do anything with it. The firewall hardware appears to be tossing it.  Ok, this is not good.  Time to dig deeper.

Luckily I can set up an asp-drop to see why the packets are bring dropped.

271: 22:27:48.672648 fe80::201:5cff:fe62:a246.547 > fe80::f60f:1bff:fe76:fa57.546: udp 121 [hlim 0] Drop-reason: (hop-limit-exceeded) hop-limit exceeded
272: 22:27:48.674983 fe80::201:5cff:fe62:a246.547 > fe80::f60f:1bff:fe76:fa57.546: udp 121 [hlim 0] Drop-reason: (hop-limit-exceeded) hop-limit exceeded

Hmmm.  "Drop-reason: (hop-limit-exceeded) hop-limit exceeded".  Its dropping it because the IPv6 hop limit has been exceeded.  Do I trust it?  Time for a packet capture.  Luckily, on the Cisco ASAs, you can have it do a packet capture and create a PCAP file that can be read with wireshark.  So I have it create a PCAP file, and open it in wireshark.  Low and behold, in the IPv6 header, I see:

Hop limit: 0

Thats not good.  Cisco is following some rule that I wasn't aware of.  Strangely enough, if I plug my Mac directly into the modem, it surely gets an IPv6 address.  Is the Cisco doing something with the hop limit?  Well lets find out.  I proceeded to set up wireshark on my Mac to grab a dump from it to see if the same thing is happening.  I got this in the Advertisement XID sent from Comcast to my Mac:

...
Internet Protocol Version 6, Src: fe80::201:5cff:fe62:a246, Dst: fe80::8c4:4700:22ad:bfd8
0110 .... = Version: 6
.... 0000 0000 .... .... .... .... .... = Traffic class: 0x00 (DSCP: CS0, ECN: Not-ECT)
.... .... .... 0000 0000 0000 0000 0000 = Flow label: 0x00000
Payload length: 133
Next header: UDP (17)
Hop limit: 0 
Source: fe80::201:5cff:fe62:a246
[Source SA MAC: Cadant_62:a2:46 (00:01:5c:62:a2:46)]
Destination: fe80::8c4:4700:22ad:bfd8
[Source GeoIP: Unknown]
[Destination GeoIP: Unknown]
User Datagram Protocol, Src Port: 547, Dst Port: 546
DHCPv6
...

Ok... same thing!  It is indeed a hop limit of 0 coming from Comcast! But why does my Mac get an address, but my Cisco refuses to play?  Further research brought me to RFC 2460, which happens to be the foundational specification for IPv6.  From the RFC:

Section 3 (p. 4):Hop Limit 8-bit unsigned integer. Decremented by 1 by each node that forwards the packet. The packet is discarded if Hop Limit is decremented to zero.

Section 8.2 (p. 27):8.2 Maximum Packet Lifetime Unlike IPv4, IPv6 nodes are not required to enforce maximum packet lifetime. That is the reason the IPv4 "Time to Live" field was renamed "Hop Limit" in IPv6. In practice, very few, if any, IPv4 implementations conform to the requirement that they limit packet lifetime, so this is not a change in practice. Any upper-layer protocol that relies on the internet layer (whether IPv4 or IPv6) to limit packet lifetime ought to be upgraded to provide its own mechanisms for detecting and discarding obsolete packets.

So, the spec basically says that it should discard packets with a hop limit of 0, and recommends that routers should be detecting and discarding packets.  Looks to me like the Cisco ASA is doing what is recommended by the spec, but the Mac is not.  The Mac is ignoring it.  This seems to be par for the course with cheap routers as well.  They don't care about the hop limit and ignore its value.  But the nicer firewalls are certainly abiding by the spec.  Other non-Comcast packet dumps on the net show that the hop limit on DHCPv6 do not have a hop limit of 0.  I have seen them anywhere from 1 to 255.  So this is certainly a Comcast-only problem.

Time To Notify Comcast

I brought this up to Comcast's attention by attempting to open a ticket.  Opening an advanced ticket is rather difficult with their support.  Your initial call will be met by someone in India or southeast Asia whereby their resolution to everything is to "Reset your modem".  It takes quite a bit to get them to create a ticket number for you and forward it to tier 2 support.  Tier 2 support brings you a little closer to home as it usually goes to a call center in Latin America.  Tier 2 is a bit more tolerant to listening to the issue and they immediately told me they would send me to tier 3. Now we are hopefully getting somewhere as the tier 3 folks were here in the US.  However, this problem still was a bit over their heads.  They agreed with me that this would need to go to engineering.  Then we wait.

I finally got a call back from a nice lady named Kristen Niemeyer who is in tier 3 support.  She said she needed to ask a manager on what to do and he was in a conference.  She told me to hold so she could speak to him, and she came back with a dismaying answer.  She said that Comcast would not send my ticket to engineering because the RFC 2460 spec was still in draft and that I was a residential client.  She said that if I was a business client, then she would escalate it.  I explained to her that the spec was a DRAFT STANDARD (meaning it was stable) and that all of IPv6 was based on it.  That did nothing.  I pointed her to a thread in the Xfinity forums that explains the problem, and she actually locked that thread without reason while on the phone with me - that was rude.  She was a decent person, but she was likely following her manager's orders who clearly had no time to deal with engineering issues for residential clients.  

I decided to go another route and reach out to some higher ups myself.  A guy by the name of John Brzozowski, who is also known as ComcastJohn in the Xfinity help forums, touts himself as the IPv6 lead and Fellow for Comcast.  He has quite the pedigree as it appears that he has had his hands in a few RFC standards surrounding IPv6.  I figured he would understand the problem and act quickly, as surely he would care.   He constantly posts in the Xfinity and Reddit formus to reach out to him directly if you have any problems issues with Comcast's IPv6.  I did just that.  3 emails. 1 Xfinity forum private message.  1 Reddit private message.  Number of responses from him: 0.

I reached out to some other people on the Reddit IPv6 forums and the Cisco support forums to see if others experienced similar issues.  Sure enough we found that Comcast's support of the hop limit appears to be regional.  One person from Reddit who has his 5506-X working lives in the Twin Cities Metro area of Comcast on residential service.  His packet dumps yielded a hop limit of 255.  Hence his Cisco ASA accepts the packet.  A person on the Cisco forums lives in the South Florida area using residential and he receives a packet with a hop limit of 0 as well with a 5506-X, and he also is unable to get IPv6 working for the same reasons as me.  I live in Colorado in the Denver area, and well, you know the story on my experience with their IPv6.  So the region appears to have an impact.

Today: Now What?

This leads us to today.  Comcast appears to gloat on the net that they have one of the most advanced IPv6 networks in place are hopefully looking to proliferate that.  But they aren't fixing some of the nasty bugs.  The Cisco ASA products are a staple for people who work from home or have small businesses to connect to their mother ship.  Many companies dole these firewalls out and only allow those to work with their corporate offices.  Until Comcast takes these users seriously and applies the RFC 2460 spec in its entirety, their proliferation with IPv6 will halt.  In time, more and more routers will support the specification, and less and less will work with Comcast.  At some point they will need to acknowledge and fix the problem if they want serious users to leverage their IPv6 stack.

I wrote this blog post to share with people who have attempted to get their Cisco ASA equipment to work with Comcast's IPv6.  I spent a lot of time figuring this out, so it is my hope that if it saves someone else time, I have done my job.  In the mean time, if you are a Comcast customer and own a nice router/firewall, stick to good old IPv4 and hope that one day Comcast will properly implement their DHCPv6 stack.

*UPDATE 5/10/2018* 

ArrisTuska (aka Chris Tuska) who is an outstanding Arris Engineer confirmed the problem is with the Arris CMTS software.  He was instrumental in agreeing this was a bug and getting the patches out to Comcast and getting them deployed to their CMTS units.  You can read the entire thread here.