Why your fancy router/firewall likely won't work with Comcast/Xfinity's IPv6

TL;DR - Comcast/Xfinity internet won't work with any routers or firewalls (like Cisco ASA-5500-X) using DHCPv6 in certain regions because the handshaking packets contain a hop limit of 0.  This violates RFC 2460 which is the IPv6 specification where firewalls and routers are supposed to discard any IPv6 packets that contain a hop limit of 0.  Comcast refuses to acknowledge or fix the issue and therefore anyone with a router/firewall that supports the spec will not be able to do DHCPv6 with Comcast as an ISP in regions that they send that invalid hop limit.

The Issue, the Tech, the Dumps

I have been a faithful Cisco user for many years as my primary router/firewall.  Their firewalls are certainly pricey, but you get what you pay for.  Ever since the Cisco ASA 5505 was deprecated they lacked any real IPv6 support. I decided to upgrade to one of their newer generation Cisco ASA 5500-X models.  The 5506-X hadn't been released and I trolled eBay to find a raging deal on an ASA 5512-X.  Its a fantastic firewall and have been happy.  The only problem was that even though it supported IPv6, it did not support DHCPv6 Prefix Delegation (PD) client.  This prevented me from being able to use IPv6 since that was what my ISP Comcast/Xfinity uses.

In August 2016, Cisco released a newer firmware 9.6.2 that they touted support for DHCPv6-PD.  So I decided to download and try.  I had difficulty finding anyone who was able to get it working, but found an excellent post by a guy who was able to get it working on Telstra (Australian ISP).  I tried out his configuration and got nothing.  So I put it away for a couple of months and then decided to try again in January 2017.  I set up the config, and I got nothing.  I tried newer versions of the firmware in hopes that it was a Cisco bug, but it was still to no avail.  So I decided to dig deeper.

How DHCPv6 works is that periodically, the DHCPv6 Server sends out a Router Advertisement (RA) a broadcast packet.  This tells your client about server routers and who you will communicate with.  Your DHCPv6, receives this, takes the RA and sends a Router Solicit (RS) from UDP ports 546 to 547 on the server where you ask for a prefix size (how many IPv6 addresses you will want to use) and DNS information.  The DHCPv6 server replies with an Advertisement XID which contains information on the prefix blocks you can use and the DNS information.  The DHCPv6 Client takes that information and sends back a Request XID stating that it will use a certain Prefix and various options.  The DHCPv6 then sends a Reply XID which contains the prefix that has been allocated to you.  The entire process looks like this:

Digging Deeper

Going through the Cisco docs, I can start looking at the debugging of the packets.  So I looked at debugging the Router Advertisements and found this:

router# debug ipv6 nd

ICMPv6-ND: Received RA from fe80::201:5cff:fe62:a246 on outside

ICMPv6-ND: Sending RA to ff02::1 on nlp_int_tap
ICMPv6-ND:     MTU = 1500
ICMPv6-ND:     prefix = fd00:0:0:1::/64 onlink autoconfig
ICMPv6-ND:          2592000/604800 (valid/preferred)

Looks good... Comcast is doing what they are supposed to.  I get the router advertisement. My RS packets appear to be set up to send out every minute and a half, so waiting, I noticed the RS packet being sent to Comcast's router.  So far so good, except I never receive back an Advertisement.

# sh ipv6 dhcp client pd statistics 

Protocol Exchange Statistics:
  Total number of Solicit messages sent:              474
  Total number of Advertise messages received:        0
  Total number of Request messages sent:              0
  Total number of Renew messages sent:                0
  Total number of Rebind messages sent:               0
  Total number of Reply messages received:            0
  Total number of Release messages sent:              0
  Total number of Reconfigure messages received:      0
  Total number of Information-request messages sent:  0

After going through the Cisco docs, it looks like I can set up an ACL to capture a dump of the traffic on the outside interface.  So off I go:

router(config)# access-list test_ipv6 extended permit ip any6 any6

router(config)# cap capout interface outside access-list test_ipv6
router(config)# show cap capout
...
7 Feb 20 2017 21:11:34 fe80::201:5cff:fe62:a246 547 fe80::f60f:1bff:fe76:fa57 546
UDP request discarded from fe80::201:5cff:fe62:a246/547 to outside:fe80::f60f:1bff:fe76:fa57/546

...

Whaaaa? A packet is getting discarded to the outside interface before I can do anything with it. The firewall hardware appears to be tossing it.  Ok, this is not good.  Time to dig deeper.

Luckily I can set up an asp-drop to see why the packets are bring dropped.

271: 22:27:48.672648 fe80::201:5cff:fe62:a246.547 > fe80::f60f:1bff:fe76:fa57.546: udp 121 [hlim 0] Drop-reason: (hop-limit-exceeded) hop-limit exceeded
272: 22:27:48.674983 fe80::201:5cff:fe62:a246.547 > fe80::f60f:1bff:fe76:fa57.546: udp 121 [hlim 0] Drop-reason: (hop-limit-exceeded) hop-limit exceeded

Hmmm.  "Drop-reason: (hop-limit-exceeded) hop-limit exceeded".  Its dropping it because the IPv6 hop limit has been exceeded.  Do I trust it?  Time for a packet capture.  Luckily, on the Cisco ASAs, you can have it do a packet capture and create a PCAP file that can be read with wireshark.  So I have it create a PCAP file, and open it in wireshark.  Low and behold, in the IPv6 header, I see:

Hop limit: 0

Thats not good.  Cisco is following some rule that I wasn't aware of.  Strangely enough, if I plug my Mac directly into the modem, it surely gets an IPv6 address.  Is the Cisco doing something with the hop limit?  Well lets find out.  I proceeded to set up wireshark on my Mac to grab a dump from it to see if the same thing is happening.  I got this in the Advertisement XID sent from Comcast to my Mac:

...
Internet Protocol Version 6, Src: fe80::201:5cff:fe62:a246, Dst: fe80::8c4:4700:22ad:bfd8
0110 .... = Version: 6
.... 0000 0000 .... .... .... .... .... = Traffic class: 0x00 (DSCP: CS0, ECN: Not-ECT)
.... .... .... 0000 0000 0000 0000 0000 = Flow label: 0x00000
Payload length: 133
Next header: UDP (17)
Hop limit: 0 
Source: fe80::201:5cff:fe62:a246
[Source SA MAC: Cadant_62:a2:46 (00:01:5c:62:a2:46)]
Destination: fe80::8c4:4700:22ad:bfd8
[Source GeoIP: Unknown]
[Destination GeoIP: Unknown]
User Datagram Protocol, Src Port: 547, Dst Port: 546
DHCPv6
...

Ok... same thing!  It is indeed a hop limit of 0 coming from Comcast! But why does my Mac get an address, but my Cisco refuses to play?  Further research brought me to RFC 2460, which happens to be the foundational specification for IPv6.  From the RFC:

Section 3 (p. 4):Hop Limit 8-bit unsigned integer. Decremented by 1 by each node that forwards the packet. The packet is discarded if Hop Limit is decremented to zero.

Section 8.2 (p. 27):8.2 Maximum Packet Lifetime Unlike IPv4, IPv6 nodes are not required to enforce maximum packet lifetime. That is the reason the IPv4 "Time to Live" field was renamed "Hop Limit" in IPv6. In practice, very few, if any, IPv4 implementations conform to the requirement that they limit packet lifetime, so this is not a change in practice. Any upper-layer protocol that relies on the internet layer (whether IPv4 or IPv6) to limit packet lifetime ought to be upgraded to provide its own mechanisms for detecting and discarding obsolete packets.

So, the spec basically says that it should discard packets with a hop limit of 0, and recommends that routers should be detecting and discarding packets.  Looks to me like the Cisco ASA is doing what is recommended by the spec, but the Mac is not.  The Mac is ignoring it.  This seems to be par for the course with cheap routers as well.  They don't care about the hop limit and ignore its value.  But the nicer firewalls are certainly abiding by the spec.  Other non-Comcast packet dumps on the net show that the hop limit on DHCPv6 do not have a hop limit of 0.  I have seen them anywhere from 1 to 255.  So this is certainly a Comcast-only problem.

Time To Notify Comcast

I brought this up to Comcast's attention by attempting to open a ticket.  Opening an advanced ticket is rather difficult with their support.  Your initial call will be met by someone in India or southeast Asia whereby their resolution to everything is to "Reset your modem".  It takes quite a bit to get them to create a ticket number for you and forward it to tier 2 support.  Tier 2 support brings you a little closer to home as it usually goes to a call center in Latin America.  Tier 2 is a bit more tolerant to listening to the issue and they immediately told me they would send me to tier 3. Now we are hopefully getting somewhere as the tier 3 folks were here in the US.  However, this problem still was a bit over their heads.  They agreed with me that this would need to go to engineering.  Then we wait.

I finally got a call back from a nice lady named Kristen Niemeyer who is in tier 3 support.  She said she needed to ask a manager on what to do and he was in a conference.  She told me to hold so she could speak to him, and she came back with a dismaying answer.  She said that Comcast would not send my ticket to engineering because the RFC 2460 spec was still in draft and that I was a residential client.  She said that if I was a business client, then she would escalate it.  I explained to her that the spec was a DRAFT STANDARD (meaning it was stable) and that all of IPv6 was based on it.  That did nothing.  I pointed her to a thread in the Xfinity forums that explains the problem, and she actually locked that thread without reason while on the phone with me - that was rude.  She was a decent person, but she was likely following her manager's orders who clearly had no time to deal with engineering issues for residential clients.  

I decided to go another route and reach out to some higher ups myself.  A guy by the name of John Brzozowski, who is also known as ComcastJohn in the Xfinity help forums, touts himself as the IPv6 lead and Fellow for Comcast.  He has quite the pedigree as it appears that he has had his hands in a few RFC standards surrounding IPv6.  I figured he would understand the problem and act quickly, as surely he would care.   He constantly posts in the Xfinity and Reddit formus to reach out to him directly if you have any problems issues with Comcast's IPv6.  I did just that.  3 emails. 1 Xfinity forum private message.  1 Reddit private message.  Number of responses from him: 0.

I reached out to some other people on the Reddit IPv6 forums and the Cisco support forums to see if others experienced similar issues.  Sure enough we found that Comcast's support of the hop limit appears to be regional.  One person from Reddit who has his 5506-X working lives in the Twin Cities Metro area of Comcast on residential service.  His packet dumps yielded a hop limit of 255.  Hence his Cisco ASA accepts the packet.  A person on the Cisco forums lives in the South Florida area using residential and he receives a packet with a hop limit of 0 as well with a 5506-X, and he also is unable to get IPv6 working for the same reasons as me.  I live in Colorado in the Denver area, and well, you know the story on my experience with their IPv6.  So the region appears to have an impact.

Today: Now What?

This leads us to today.  Comcast appears to gloat on the net that they have one of the most advanced IPv6 networks in place are hopefully looking to proliferate that.  But they aren't fixing some of the nasty bugs.  The Cisco ASA products are a staple for people who work from home or have small businesses to connect to their mother ship.  Many companies dole these firewalls out and only allow those to work with their corporate offices.  Until Comcast takes these users seriously and applies the RFC 2460 spec in its entirety, their proliferation with IPv6 will halt.  In time, more and more routers will support the specification, and less and less will work with Comcast.  At some point they will need to acknowledge and fix the problem if they want serious users to leverage their IPv6 stack.

I wrote this blog post to share with people who have attempted to get their Cisco ASA equipment to work with Comcast's IPv6.  I spent a lot of time figuring this out, so it is my hope that if it saves someone else time, I have done my job.  In the mean time, if you are a Comcast customer and own a nice router/firewall, stick to good old IPv4 and hope that one day Comcast will properly implement their DHCPv6 stack.

*UPDATE 5/10/2018* 

ArrisTuska (aka Chris Tuska) who is an outstanding Arris Engineer confirmed the problem is with the Arris CMTS software.  He was instrumental in agreeing this was a bug and getting the patches out to Comcast and getting them deployed to their CMTS units.  You can read the entire thread here.

United Airlines: Customer Service at its Worst

Boy how times have changed regarding air travel.  When I was a boy, air travel was a great experience.  You dressed up nice for a flight and you were treated like royalty.  You had a decent meal and even got a nice beverage and snack.  It was an experience to look forward to.  Fast forward to today, and the experience borderlines riding in overcrowded public transportation (like a bus) with nickel-and-diming, ever-smaller seats, fees galore, and some of the rudest employees an industry could ever could ever tolerate.  The rudeness has gotten so bad, that the airlines appear to enjoy a monopolistic customer service experience with their employees that would fire a McDonalds fast-food employee on the spot if it was delivered in that particular industry.  The airline industry as a whole needs a reboot and to re-learn what customer service is all about.  Here is my story...

As some background, I have been a loyal United customer for many years.  I have been 1K and usually make Platinum level each year (these are top elite places in United's mileage plus program).  I do this to help avoid the lines, prevent getting crappy "B" seats when I fly,and having a place to store my carry-on since I get ot board first.  Every year I take my family away during Christmas for a holiday to an international destination.  I usually do this as a mileage run, but the economy experience has gotten so rotten that I generally pay to put the wife and me in business class.  It gives me 1.5 PQM (elite miles) and is an easy way to be sure I am able to attain my status.

This year I decided to take my family of 5 to Hong Kong for Christmas/New Years.  In early October I found a decent fare and booked my family from Denver (DEN) to Hong Kong (HKG) through San Francisco (SFO).  As a platinum member, I get the cool option of being able to book the economy folks (my kids) in Economy Plus, so its a bit more of a comfortable ride for them.  I got a couple of nice seats in business for me and the wife, and the kids sitting together in economy plus.  This was going to be a trip to remember!  Turns out I was right... but not the way one would hope.

The troubles started when we departed on 12/23 from DEN-SFO.  We had planned about a 2 hour layover in SFO, but SFO was having some rain problems, so our flight was delayed... for 1.5 hours.  That was going to make getting to our SFO-HKG flight a challenge.  I spoke to the flight attendant on the Denver flight and she said "You will be fine, you will have 20-30 minutes to spare".  So off we went with confidence that all was fine.  But we landed and had 15 minutes to get from one terminal to the other to catch our Hong Kong flight.

I fly internationally quite a bit and I cannot tell you the number of times I sat on a plane and the pilot announces "Hi folks, we will be delayed another 10 minutes as we are awaiting a group of passengers who are connecting are are currently in transit to our flight".  Knowing this was the case, I was very confident that United would be sure that we got on the plane.  I mean, come on, they track your bags so they must track you, right?  They have to know where you are, right?  That appeared to be an invalid assumption and I guess we are not afforded the same courtesies that they seem to grant other passengers.  When we landed in SFO, we RAN, and yes I mean RAN, to our other gate.  I had my boys run there faster since we were carrying the carry-on and figured they could tell the gate agent that we are in transit and it be there in a few minutes.  When my boys arrived at the gate, they went to the ticket agent and said "My family is running, they will be here in a minute or two".  The agent looked at them and said "Well, they better make it fast because we are about to close the doors".  As my boys had told them, we arrived there literally 1.5 minutes later.  We watched a passenger get on and as we walked up to the door, the ticket agent closed the door behind him and said "I'm sorry, we are closing the door.  You need to go to customer service".  My wife looked at him and said "You just let that passenger get on!  Why couldn't you let us on?" He said, "I'm sorry, you aren't going to like to hear this but we gave your seats away to people waiting for an upgrade".  Wow.  They gave away our seats knowing fully well we were in transit by literally minutes.  Not only am I top-tier elite status but we additional paid for our seats.  The sad thing is that about 8 customers after us had the same issue.  You would think the airline would hold the plane for literally minutes to accommodate connecting passengers and stop he hemorrhaging. 

But it gets worse.  To add insult to injury, the flight from SFO-HKG had a mechanical delay.  By 52 minutes.  They could have easily let us on and put those "upgraders" back in their crappy seats.  Its been done before.  Thats when United's famous rudeness kicked in.  The ticketing manager dragon (that's airline speak for the person who protects the airline and treats you like poo) told us this is the way it is and she was he one who made the decision.  She raised her voice to us and was very unpleasant with a "too bad" attitude.  United generally could care less about their employees' bad behavior, so complaining about will not yield any results.  Sometimes I wonder if they reward their employees for crappy attitudes, offering a "crap-head bonus" to those who treat customers so awful.  Merry-f'ing-Christmas, United!

We went back to the United Club to talk to a customer support person to hopefully 240 us (airline speak to get us on another flight on a competing carrier) or get us on the next flight out.  The ticketing customer services began looking and said "I'm sorry but I have nothing for 3 days".  I pulled up Kayak.com on my phone and said "Wait, I see several flights going out of Seattle, Vancouver, and Los Angeles today, why can't you get me on those flights?"  I was explained that they do not have agreements with Cathay Pacific or Virgin air.  Gosh, I remember a time when you could fly on any of the carriers when you got 240'd.  Now they pick and choose to a select few?

I looked on Kayak again and I saw that the next United flight the next day had several 1st class seats open and they certainly could accommodate us.  The ticket agent said "I cannot give those to you."  I asked why not.  The response was that our tickets are not eligible for a 1st class seat.  Wait.  United inconvenienced us by giving away our tickets, making it so that we won't be able to fly for days, and they cannot give up their precious 1st class seats to us?  Times have surely changed.

Yes.... it gets even worse.  The United agent basically said we can wait for 3 days but they will not pay for 3 days of hotel.  Wha?  They inconvenience us and want us to pay for our own hotel?  Not to mention, by waiting 3 days, we burn half our vacation, so whats the point?  She told us she then had no choice but to give us a full refund and send us home.  We didn't agree or disagree.  We would call the Platinum desk and hopefully get someone with a bit more pull to hopefully make something happen.  As I was talking to the Platinum desk, that agent told us that they couldn't get into the record because someone was working on it.  I went over to the top dragon in the United Club and asked what was up.  This woman was terribly rude.  She showed me our tickets.  We were re-routed back to Denver on 3 different flights because there was no room.  She said she had to pull favors to get us back to Denver.  Imagine, my 10 year old son flying by himself back to Denver.  Yeah, that wasn't going to happen.

I spoke to her and said "Come on, this was my Christmas present to my family and they have to be more cooperative."  The woman was rude and told us there were no other options.  But my my luck, a nice Indian woman agent was sitting next to her and said "Please, let me try".  She immediately pulled up flights to get us to Hong Kong.  She showed them to the manager and the manager rolled her eyes. Wow... the manager was lazy... typical United behavior.  So now that the manager was busted, she had to start looking.  She found a flight to Beijing the next day with a connecting flight on Air China to HKG.  But it was in economy.   I told her that we paid for business class and for us to fly in economy is not acceptable, especially without some form of a refund.  She glared at us and said "Look, this is what I can get you.  You should not question this.  Hold on... I will continue to look."  She came back and said "ok, I have you in business class".  She gave us these blank tickets and told us to come back in the morning.  She said we do not get hotel because it was weather related (yeah....right... weather related - airline excuse for we don't want to cover your hotel).  The nice Indian ticket agent told us "Don't worry, I got you taken care of".  She gave us vouchers to a shitty Holiday Inn (I guess better than nothing) making it look like they were doing us a favor and told us to get our tickets at the counter in the morning.  She said we could get our bags and bring them to the hotel.

Yes, it continues to get worse.  We went down to the baggage claim and asked for our bags.  SFO was trying desperately to track them down, when they found that the wonderful ticket agent manager who processed our "refund" sent the bags back to Denver.  She forgot to get them when getting our new tickets to HKG.  Oops.  No bags.

Exhausted for the multiple hours of trying to have ticket agents do their "magic", we finally got to the hotel and promptly fell asleep.  Our plan was to get up early and get to the airport early to hopefully head off any potential problems in the event our good luck would continue.   We went to the ticket counter and the agent handed us our tickets with a hearty "Enjoy your trip!".  We walked away from the counter and looked at our tickets when something caught my eye.  The belligerent ticket agent from the day before put us in economy.  WTF?  So we ran to the international ticket desk as time was ticking by and we were going to once again miss our window to somehow get to Hong Kong.  We had to get in line and wait... and wait.  We got to an agent and for the first time we got someone who actually cared.  She heard our story and looked at the tickets and finally said "This is completely unacceptable.  I am very sorry.  Let me see what we can do for you".  She had us get into a line whereby we would talk to their top person.  We waited again.  We finally got to speak to a German lady who was the top ticketing manager in the international SFO desk.  In the beginning she wasn't too friendly.  She had a "tough crap, its Christmas and we are overworked" attitude.  But then magically, her face completely changed.  I am not sure if the spirit of Saint Nick tapped her on the shoulder and told her to get a grip, but she finally had a look of pity.  She said "Let me see what I can do for you".  She worked... for about 45 minutes on the computer.  She came back and said "You are set.  I have you on the same flights and you are in business class all the way through.  She said "I am very sorry for you.  During the holidays we hire temproarily and we get people who have no idea what they are doing."  I told her that her desk was the first people who have been anywhere near helpful.  She told us "Get going... you have 20 minutes to get to your plane".  So off to Beijing we went.  The flight was uneventful and we were connecting to Air China from PEK to HKG.

Does it get worse? Yep... you guessed it.  It certainly does.  We get to PEK and we were told to obtain our bags as Beijing will not transfer bags to connecting flights.  So we get to PEK and guess what?  No bags.  We put in a trace and the couldn't find them.  Our only possibility was to get to HKG and deal with it there.  We get on our Air China flight and we started to giggle.  When we got on, all of business class was essentially empty as was economy plus.  The only thing that had passengers was the very back, or economy section.  This underlined that the original ticketing agent was trying to make our trip as miserable as possible.  Thank goodness for that last ticket manager who made it happen.

How much worse can it get? Well, it can.  We get to HKG and of course no bags.  We put in a trace and we were told the bags finally made it to PEK and will be on their way to HKG on the next flight out.  But of course that didn't happen.  The next day we had the hotel try to get a trace on our luggage.  They were finally successful and we were told it would be delivered in the evening.  A full 2.5 days without our luggage and medications.  Thank goodness prescriptions weren't needed in Hong Kong Pharmacies and we were able to obtain what we needed.  We had to purchase clothes, toiletries and other items to tie us over until our baggage arrived.  The rest of the week was pleasant as we were finally out of the clutches of United Airlines and in the hands of the JW Marriott.  Yes Marriott is certainly a company that gets customer service and they made our trip delightful.

It can't get worse now, can it?  Yes... it can.  We unfortunately had to return home via United Airlines.  We hoped that the long haul would be uneventful as we were sure to get to the airport early so we wouldn't have any snafus.  But we didn't think about the fact that we could get on the plane and what would happen there.  We get on and I am sure that my kids have their iPads all charged for the long trip to attach to United's new-fangled wifi whereby they will stream movies directly to your device.  We get on tand United announced that their wifi is down and not functioning.  Oh shit.  I have young kids who are going to stare at the back of their seats for 12 long hours.  My youngest, who has A.D.H.D. is going to be a wreck.  He is going to have a tantrum and the flight attendants are going to get us and complain - yes this has happened before.  This will be a disaster beyond proportions.  Luckily, we were in luck.  The kids slept most of the flight.  But how the heck United can have a completely malfunctioning entertainment system in place for a long haul is unacceptable.  I felt very sorry for those who were not given a  way to watch  movies.  You would think United would have a quick way to swap out their wifi systems in the event of failure, or possibly even a backup.  Being in the computer industry, I know it doesn't take much to think about a fault tolerant system for these sorts of things.

But it does get worse.  We are up in business class, and after we take off and a reasonable meal, we decide to sleep.  I put on my Bose noise-cancelling headphones in hopes to get some peaceful sleep.  But I can't.  Certainly it was not due to my lack of being sleepy.  It was the flight attendants.  They decided to camp out near our seat and chat.  For 12 hours.  Loud.  I could hear their cackling and clucking right through the headphones.  They weren't even trying to be quiet.  They had long conversations about politics, Trump, ISIS, flying, what-to-wear.  For....12...long...hours.  I have never seen a company allow their employees to water-cooler for so long.  Does United not have a policy regarding being quiet when the lights are out and passengers are clearly trying to sleep?  You would think the chief purser would tell the others to STFU, but its difficult when the chief purser is also involved in the activity.  Who watches the purser?

And worse it still gets.  As I mentioned before, as a Platinum elite, one of the perks is that I get is I get to put my party in Economy Plus.  I had them all booked in a single nice row. Well, some ticket agent along the way probably saw 3 non-elites flying in a nice row and thought "Hrm, I need to put my 1K or Global Service customer in a seat, so I will bounce on of those kids".  Yep, that's what they did.   You would think that they would check the PNR code and see that those kids are attached to a Platinum itinerary.  But United doesn't care  Why offer the perk if they give it and then taketh away for yet anohter elite member.  Not cool.  Just don't offer it if you are going to just take it back.

Where are we are at today?  I will likely complain to United and they will probably offer me some token crappy number of miles or a small discount on my next flight.  Of course it will not acceptable.  They SHOULD apologize profusely and refund me me a decent amount for downgrading us from a direct long haul to one with a connection and for interrupting our vacation when they actually had the power to ensure we were on that original flight, in addition to their token crap they usually offer.  Maybe I will be surprised and they will go above and beyond to make us whole again.  I can dream, can't I?  But knowing United, I will get their usual form letter.  They don't care.  They don't need me, no matter how much business I give them.  I own a company whereby the majority of my guys fly United for business.  If they counted the amount that we spend per year on them, I would be Global Service.  But that's not the way they think.  If we all voted with our wallets, then United would change their ways.  But they are such a monopoly, it will be hard to do that.

Its time to rethink my loyalty to United.  It is one of the worst airlines, and its not only me who thinks so.  They are rated with one of the highest customer complaints in the industry with a 1 star rating on Consumer Affairs and dead last in JD Powers Ratings.  You would think that an airline would do its best to offer the very best in customer service and certainly take care of their top elite customers.  The industry has changed, and certainly not for the best.  Fees, poor customer service, treating the customer like cattle, and terribly rude employees plagues one of the worst industries that we have.

Thankfully, Delta has offered to match my status.  I will likely be taking them up on that offer.  My experience with Delta has been pretty good.  Maybe... just maybe... there is light at the end of the tunnel and I can find an airline that gets loyalty and customer service.

Oracle and the fall of Java EE

Recently there has been a lot of ramblings in the interwebz about Oracle removing its funding for Java EE. A long piece by Ars Technica was recently published that had some interesting facts. I wanted to offer an op-ed on this topic since I have been knee-deep in Java EE since its inception. Yes opinions are like assholes… everybody has one… ;-) Of course I am not any different. ;-) For the sake of my “opinion”, I want to say I have been on the Java EE expert groups since post J2EE, either as a representative of Apache or as an individual. This includes JSRs 244, 316, 342, 366. I was one of the core developers of Apache Geronimo (that's where I got my open source start), and at one point I was actually writing a JBoss book. So lets just say I had a huge investment in Java EE. I essentially bet my entire career on it. Being in Java EE and now heavily invested in microservices/SOA, I have seen a huge sea change when it comes to running applications in the enterprise. I remember the days when Java EE/J2EE was the only game in town. JBoss’ success was driven by the ability of an open source project to compete with the monsters-of-app-servers (BEA/IBM/Sun) and relinquish the hold for $10K/core licensing and that methodology was literally unseated. I recall getting a meeting with some top ranks of BEA (pre-Oracle acquisition) at the time to discuss with our company (Virtuas) about how they can go the direction of open source because their licensing model was getting eaten alive. IBM’s investment in Geronimo was driven by that same very reason. I joined IBM on their Websphere CE team (That was based on Apache Geronimo) and I recall the management discussions for why they went that route, because their WAS licensing was getting consumed by JBoss and they needed a play to protect the services component and support (which really made up a large chunk of their revenue streams).

Back in the mid 2000s, I recall the #1 issue devs had with the Java EE stacks was their monolithic size and the incredible long times that it took for these containers to start up. It was no wonder that Spring had become such a success with Rod Johnson’s books that became the basis of the Spring framework. People were using Spring to develop/test their apps. Soon, companies were developing these small apps to capitalize on fast run times and quick development. Companies saw they didn’t need the whole kitchen sink to run their apps. Many companies utilize less than 10% of the offerings inside a Java EE container and they wondered why they needed all this extra “stuff” to run their web app. They just needed some DB access, some transactions, and the web container. They didn’t need corba, or EJBs, or whatever. For this reason. Geronimo went in the direction of making a modular app stack that could pass a TCK for certification, whilst allowing folks to pick and choose what they wanted to run. You could start will a small Geronimo (basically a Tomcat stack - called it Little G) or get the full tomato, which was your typical monolithic Java EE container. The difference between Geronimo and all others was that it was modular. They used this concept called GBeans which was supposed to make addition and removal of components easy. GBeans wasn’t all-that-and-a-bag-of-chips. It was essentially a home-grown OSGi without the matrix classloader. But it was a start to a modular container. In 2006 I made a strong advocation for the Geronimo container to just use OSGi, but the IBM management said, “no”. I had stated it's a modular standard and allows you to deploy/undeploy with a matrix classloader without the warts of a tree based classloader, which was the Achilles heal of all Java EE containers. But thereafter (and I had subsequently left IBM), it seemed that a modular Java EE container was the way to go. In 2012, IBM saw the light and decided it was a necessity. A majority of the Java EE containers seemed to move in that general direction as well. I guess Red Hat (JBoss), Oracle (Glassfish), etc saw that having modularity gave people a choice in how to to run their Java EE stacks. However, as modular as they were, they were still bound by massive components bound together by the required Java EE profiles.

Fast forward to the last few years, the concept of SOA and Microservices began to gain attention. Companies took the concept of a monolithic apps (EAR file, WAR file, whatever), and began to break it up into an app that could be broken into its own components and distributed. A single app could run on multiple servers utilizing persistent messaging and simple in-process upgradeability. Come Java EE 7 and Java EE 8, there was significant discussion about profiles and modularity. Lets just say I was a huge advocate for modularity because I saw the writing on the wall... Profiles were useless. People wanted to piece together what they wanted (JPA, JTA, Resful service, etc), and didn’t want to be tied to anything specific. They wanted to break their apps up into callable APIs and fully separate the GUI from the back end service. Microservices became the way of the Jedi when it came to application development and deployment. It removed the single point of failure and allowed for a cloud-friendly scaleable build with small footprints. But guess what... modularity in its true form never made in as part of the specification.

Today, a large part of my business is migrating companies off the monolithic Java EE containers into lightweight modular containers. Yes, even the tried and true banking and financial industries are moving away from Java EE. Every company has a different set of specifications for what parts they may use in a Java EE stack, but rest assured, its all going by their choice and being as small of a footprint as possible. When companies buy cloud images and the pricing is based on memory size and numbers of cores, it gets real easy to understand why they don’t want to run these stacks that require lots of resources for only needing to use less than 10% of what a Java EE stack may offer. It because it costs more to run crap that you will never use.

Oracle and I have always had a love-hate relationship over the years, primarily because they represent the quintessential corporate identity that many of us love to hate. That, along with my Apache open source roots, Oracle and I have no historical love for each other. That said, I have to somewhat defend Oracle on this decision. Many of us see this sea change. I am sure Oracle sees it as well. Oracle isn’t stupid. They see the writing on the wall. Lightweight, small footprint stacks are the way to go and its where corporate IT is going, like it or love it. I think Java EE lost its way with profiles. Java EE certainly could have been a continued success if it was truly modular and Java EE itself was really a container that *offered* services that could be tested together with a massive TCK to ensure the different components worked with each other. But it should have been small, pick and choose what you needed to use, and made a single stack that fit your needs. Gosh, that sounds an awful lot like an ESB. ;-) But we are now a day late and a dollar short when it comes to who wants what, and the kinds of changes need to be made.

The bottom line here is the market for Java EE stacks is waning and Oracle chooses not to invest in a technology for where they think there is no future. I don’t blame them as I would make the exact same decision if it were my business. That said, the JCP is supposed to be “open” with a group of technology leaders and companies that are supposed to help make the decisions for the Java EE direction. Nothing stops a Red Hat or IBM, or an individual from taking the torch and becoming the steward for Java EE. If the masses want a Java EE of some form to continue, community members should step up. Nothing lasts forever and technology changes. If you want it that bad, then become a steward and stop complaining that its Oracle's responsibility. Its not. They think their money is better spent elsewhere and I don't blame them.

For those of you who want to steward a continued Java EE, get out there and lead the effort. Stop bitching about how Oracle is evil and they are killing Java. They aren't. They are making a business decision. They are not philanthropists and we certainly shouldn't expect them to be. My recommendations for what a successful Java EE needs to be

1. Drop the profiles. They are useless and pointless waste of time.
2. Make Java EE a conglomeration of technologies that are known to work together (i.e. JPA, JTA, JMS), but don't force their full installation. Let the use pick and choose what they want. Just be sure that they work together in pieces or in totality.
3. Make it modular. Start with zero and let people add what they want to use.
4. Get rid of the tree class loader. You need it matrix based. OSGi is fine. Jigsaw is fine. But it needs to have classes that can be loaded and unloaded without an impact on the container.
5. The container itself should be a thin shell with remote capabilities (again OSGi/Karaf looks really good here)
6. Make it cloud friendly. Microservices needs to be core functionality, lightweight, and run in the cloud.

But if people are going to claim the sky is falling while pushing profiles down people's throats of which only few will use, then they are heading the way of the dinosaur. In today's cloud-centric and polyglot implementations, profiles and monolithic containers will certainly not gain momentum or speed. If you disagree, then step up and become a leader in the Java EE space and try to make change. Take on the JSR and be the steward. But I wouldn't try to fragment the supporters or it will just get lost in the shuffle. Perhaps the Java EE Guardians can help make that stewardship change, but I recommend that they really look at what people want and use and think about some of the things that I proposed in my list above.

I hate to say it, but I told you so. Technologies change, paradigms change, and needs change. Stop blaming Oracle for the Java EE demise. If Java EE is needed that much, then there will be enough community to make it continue its path. If not, it will die on the vine. If people claim that Oracle's lack of investment is what will kill Java EE, then, it's already dead.

Google Apps Calendar #sucks

Dear @google, your Apps for domains are awesome but why does it have to suck with real push syncing on the Mac? #icloud ftw!

I have 4 main machines I use in my daily use, a MacPro (for my desk), a MacBookPro for my remote development and work, my iPad for bedtime reading and air travel, and of course my iPhone for my "keep it always with me". I need them to also always be in sync. I also use Google Apps for Domains for my email and calendar. I loved it when Google released real "push" email and calendar by simulating an exchange server for my beloved travel devices. I assumed they also had "push" in the calendar for my computers. I assumed wrong. I noticed that as I added an event on Google or my computers running iCal, the IOS devices got the events, but the Macs did not. The only way my computers got the event was if I ran iCal continuously on each machine. If I didn't, the machine would miss the event... so there I had a manual step in my "syncing" my computers. Luckily my trusty IOS apps got the event.

I think I must have set something screwy, since Google offers exchange style push for Microsoft Outlook on Windows and mobile devices. So off I went and Googled "google push ical exchange" and this appears to be a real complaint for Apple users. Apparently Google made exchange/push for IOS/Android and Microsoft Outlook users (non-Mac of course), but silently left out Apple users to have a true push experience. Not sure I get why. Tin foil hat posters seem to think its a vendetta for the Google vs Apple wars. *shrug*

Then along came Apple's iCloud. Oh how I love thee iCloud. With push and syncing between devices... It just works™. So I exported my calendars from Google and into iCloud they went. Total time to set up on all devices and computers: about 10 minutes from export to fully running synced calendars. So now, Google Apps now handles my email and iCloud wonderfully handles my calendar. I get my cake and can eat it too!

So here is what I don't get. Google must have seen the iCloud thing coming. You would think they would have thought "Hrm... we had better get our act together and give push to our Apple users or they may leave our calendars behind." But they didn't. Was it really that difficult to enable Exchange for use on Lion, especially since Lion is 100% exchange compatible? I really like Google's services and have been a faithful Apps user for Domains for several years. I would have kept my calendars at Google had they "pushed" me my calendars and never even looked at iCloud. But Apple built a better mouse trap and it will need to be something compelling to get my calendars back on Google. If Apple builds an "ICloud for domains", it will probably be adios to Google Apps for good since I can still only use IMAP or POP to my Mac email clients.

Are you listening Google?

Frist Post

First... about me... I am the founder of Savoir Technologies, a SOA/Java consulting company specializing in the Apache Services stack (ServiceMix, Karaf, CXF, ActiveMQ, and Camel). I work with a group of very talented guys and we all have contributor or commit status on some of the various aforementioned projects (and more). We contribute heavily to Open Source and believe in the Open Source methodology for integration. My hobby is Fire Fighting (yes ... really) and am a volunteer firefighter for Evergreen Fire Rescue, which is my community's volunteer fire department. I have been a fire fighter for 10 years. I am happily married to my wife Nazarena for nearly 17 years and have 3 children, Madisyn (12), Weston (10), and Coleton (5). I'm a techy geek type of guy and am a misfit's fanboi. I love hacking the iphone and was on the iPhone Dev Team as "fred_". I was the guy who wrote most of the code for the revirginizer, which fixed Apple's bricking of anyone who unlocked their iphone and was the catalyst for getting iPhone Elite reintegrated with the iPhone Dev Team... long story... ;-) But I no longer spend large swatches of time hacking the iphone because I simply just don't have the time. I found that most folks who are married and have a family lose their luster for hacking and concentrate on the more important things in life.

So... enough about my background...

It's been a long... long... time since I blogged. My old blog, 'Random Neuron Firings' was kept up until 2006. I personally hosted that blog on my own servers using the awesome Apache Roller, but it soon became too much of a headache and blogging just got plain old boring. But I'm back. Why? I'm not a socialite as I just can't get into the Twitter/Facebook thing.

Twitter just seems too much about telling people how big the turd is you just left or using dumb services like Four Square that announces to all thieves that you are not home so just burglarize me. Do I tweet? Yeah... I try once in a while so that I have some epiphany that turns on in my head and get that "ah ha" moment. But it just doesn't happen. Maybe one day?

Facebook is all about "me me me me me". I could care less about the picture of your 8 month old's dirty diaper or how you threw up all over yourself after last night's party. The only time I log into Facebook is to look up someone whose information I forgot, looking for how to connect with them (phone or email - yeah people put way too much personal crap on their Facebook page). I also use Facebook to do "background" checks on potential employees or contractors to be sure I don't hire on someone I may be sorry about later. The good thing about Facebook is it underscores that old saying my mother used to tell me, "Choose the company you keep wisely, for you will tend to be just like them" and "Birds of a feather, flock together". Facebook is great at getting a quick look into someone's personal life and it can be very telling.

Blogging is a different story. Its a great place to journal things you really want to keep. Its keeping it up to date that is the struggle. I must say I shut down my blog in 2006, but I still dig for some great tech postings ind code snippets that I wish I hadn't forgot. So for me, putting my blog here give it a bit more life and using the cloud to handle the ugly stuff.

So this blog... what's it about? Its about manly tech stuff... code snippets that I want to keep... and journal some ideas and thoughts that I want to look back on. I want it to be about my adventures in tech, fire fighting, soa, and family. I want it to be my legacy to look back on and be my little personal imprint on the web.

Thats it for my frist post... more to come...