Wednesday, March 15, 2017

Why your fancy router/firewall likely won't work with Comcast/Xfinity's IPv6

TL;DR - Comcast/Xfinity internet won't work with any routers or firewalls (like Cisco ASA-5500-X) using DHCPv6 in certain regions because the handshaking packets contain a hop limit of 0.  This violates RFC 2460 which is the IPv6 specification where firewalls and routers are supposed to discard any IPv6 packets that contain a hop limit of 0.  Comcast refuses to acknowledge or fix the issue and therefore anyone with a router/firewall that supports the spec will not be able to do DHCPv6 with Comcast as an ISP in regions that they send that invalid hop limit.

The Issue, the Tech, the Dumps

I have been a faithful Cisco user for many years as my primary router/firewall.  Their firewalls are certainly pricey, but you get what you pay for.  Ever since the Cisco ASA 5505 was deprecated they lacked any real IPv6 support. I decided to upgrade to one of their newer generation Cisco ASA 5500-X models.  The 5506-X hadn't been released and I trolled eBay to find a raging deal on an ASA 5512-X.  Its a fantastic firewall and have been happy.  The only problem was that even though it supported IPv6, it did not support DHCPv6 Prefix Delegation (PD) client.  This prevented me from being able to use IPv6 since that was what my ISP Comcast/Xfinity uses.

In August 2016, Cisco released a newer firmware 9.6.2 that they touted support for DHCPv6-PD.  So I decided to download and try.  I had difficulty finding anyone who was able to get it working, but found an excellent post by a guy who was able to get it working on Telstra (Australian ISP).  I tried out his configuration and got nothing.  So I put it away for a couple of months and then decided to try again in January 2017.  I set up the config, and I got nothing.  I tried newer versions of the firmware in hopes that it was a Cisco bug, but it was still to no avail.  So I decided to dig deeper.

How DHCPv6 works is that periodically, the DHCPv6 Server sends out a Router Advertisement (RA) a broadcast packet.  This tells your client about server routers and who you will communicate with.  Your DHCPv6, receives this, takes the RA and sends a Router Solicit (RS) from UDP ports 546 to 547 on the server where you ask for a prefix size (how many IPv6 addresses you will want to use) and DNS information.  The DHCPv6 server replies with an Advertisement XID which contains information on the prefix blocks you can use and the DNS information.  The DHCPv6 Client takes that information and sends back a Request XID stating that it will use a certain Prefix and various options.  The DHCPv6 then sends a Reply XID which contains the prefix that has been allocated to you.  The entire process looks like this:

Digging Deeper

Going through the Cisco docs, I can start looking at the debugging of the packets.  So I looked at debugging the Router Advertisements and found this:

router# debug ipv6 nd

ICMPv6-ND: Received RA from fe80::201:5cff:fe62:a246 on outside

ICMPv6-ND: Sending RA to ff02::1 on nlp_int_tap
ICMPv6-ND:     MTU = 1500
ICMPv6-ND:     prefix = fd00:0:0:1::/64 onlink autoconfig
ICMPv6-ND:          2592000/604800 (valid/preferred)

Looks good... Comcast is doing what they are supposed to.  I get the router advertisement. My RS packets appear to be set up to send out every minute and a half, so waiting, I noticed the RS packet being sent to Comcast's router.  So far so good, except I never receive back an Advertisement.

# sh ipv6 dhcp client pd statistics 

Protocol Exchange Statistics:
  Total number of Solicit messages sent:              474
  Total number of Advertise messages received:        0
  Total number of Request messages sent:              0
  Total number of Renew messages sent:                0
  Total number of Rebind messages sent:               0
  Total number of Reply messages received:            0
  Total number of Release messages sent:              0
  Total number of Reconfigure messages received:      0
  Total number of Information-request messages sent:  0

After going through the Cisco docs, it looks like I can set up an ACL to capture a dump of the traffic on the outside interface.  So off I go:

router(config)# access-list test_ipv6 extended permit ip any6 any6
router(config)# cap capout interface outside access-list test_ipv6
router(config)# show cap capout
7 Feb 20 2017 21:11:34 fe80::201:5cff:fe62:a246 547 fe80::f60f:1bff:fe76:fa57 546
UDP request discarded from fe80::201:5cff:fe62:a246/547 to outside:fe80::f60f:1bff:fe76:fa57/546


Whaaaa? A packet is getting discarded to the outside interface before I can do anything with it. The firewall hardware appears to be tossing it.  Ok, this is not good.  Time to dig deeper.
Luckily I can set up an asp-drop to see why the packets are bring dropped.
271: 22:27:48.672648 fe80::201:5cff:fe62:a246.547 > fe80::f60f:1bff:fe76:fa57.546: udp 121 [hlim 0] Drop-reason: (hop-limit-exceeded) hop-limit exceeded
272: 22:27:48.674983 fe80::201:5cff:fe62:a246.547 > fe80::f60f:1bff:fe76:fa57.546: udp 121 [hlim 0] Drop-reason: (hop-limit-exceeded) hop-limit exceeded

Hmmm.  "Drop-reason: (hop-limit-exceeded) hop-limit exceeded".  Its dropping it because the IPv6 hop limit has been exceeded.  Do I trust it?  Time for a packet capture.  Luckily, on the Cisco ASAs, you can have it do a packet capture and create a PCAP file that can be read with wireshark.  So I have it create a PCAP file, and open it in wireshark.  Low and behold, in the IPv6 header, I see:
Hop limit: 0
Thats not good.  Cisco is following some rule that I wasn't aware of.  Strangely enough, if I plug my Mac directly into the modem, it surely gets an IPv6 address.  Is the Cisco doing something with the hop limit?  Well lets find out.  I proceeded to set up wireshark on my Mac to grab a dump from it to see if the same thing is happening.  I got this in the Advertisement XID sent from Comcast to my Mac:
Internet Protocol Version 6, Src: fe80::201:5cff:fe62:a246, Dst: fe80::8c4:4700:22ad:bfd8
0110 .... = Version: 6
.... 0000 0000 .... .... .... .... .... = Traffic class: 0x00 (DSCP: CS0, ECN: Not-ECT)
.... .... .... 0000 0000 0000 0000 0000 = Flow label: 0x00000
Payload length: 133
Next header: UDP (17)
Hop limit: 0 

Source: fe80::201:5cff:fe62:a246
[Source SA MAC: Cadant_62:a2:46 (00:01:5c:62:a2:46)]
Destination: fe80::8c4:4700:22ad:bfd8
[Source GeoIP: Unknown]
[Destination GeoIP: Unknown]
User Datagram Protocol, Src Port: 547, Dst Port: 546

Ok... same thing!  It is indeed a hop limit of 0 coming from Comcast! But why does my Mac get an address, but my Cisco refuses to play?  Further research brought me to RFC 2460, which happens to be the foundational specification for IPv6.  From the RFC:
Section 3 (p. 4):Hop Limit 8-bit unsigned integer. Decremented by 1 by each node that forwards the packet. The packet is discarded if Hop Limit is decremented to zero.

Section 8.2 (p. 27):8.2 Maximum Packet Lifetime Unlike IPv4, IPv6 nodes are not required to enforce maximum packet lifetime. That is the reason the IPv4 "Time to Live" field was renamed "Hop Limit" in IPv6. In practice, very few, if any, IPv4 implementations conform to the requirement that they limit packet lifetime, so this is not a change in practice. Any upper-layer protocol that relies on the internet layer (whether IPv4 or IPv6) to limit packet lifetime ought to be upgraded to provide its own mechanisms for detecting and discarding obsolete packets.
So, the spec basically says that it should discard packets with a hop limit of 0, and recommends that routers should be detecting and discarding packets.  Looks to me like the Cisco ASA is doing what is recommended by the spec, but the Mac is not.  The Mac is ignoring it.  This seems to be par for the course with cheap routers as well.  They don't care about the hop limit and ignore its value.  But the nicer firewalls are certainly abiding by the spec.  Other non-Comcast packet dumps on the net show that the hop limit on DHCPv6 do not have a hop limit of 0.  I have seen them anywhere from 1 to 255.  So this is certainly a Comcast-only problem.

Time To Notify Comcast

I brought this up to Comcast's attention by attempting to open a ticket.  Opening an advanced ticket is rather difficult with their support.  Your initial call will be met by someone in India or southeast Asia whereby their resolution to everything is to "Reset your modem".  It takes quite a bit to get them to create a ticket number for you and forward it to tier 2 support.  Tier 2 support brings you a little closer to home as it usually goes to a call center in Latin America.  Tier 2 is a bit more tolerant to listening to the issue and they immediately told me they would send me to tier 3. Now we are hopefully getting somewhere as the tier 3 folks were here in the US.  However, this problem still was a bit over their heads.  They agreed with me that this would need to go to engineering.  Then we wait.
I finally got a call back from a nice lady named Kristen Niemeyer who is in tier 3 support.  She said she needed to ask a manager on what to do and he was in a conference.  She told me to hold so she could speak to him, and she came back with a dismaying answer.  She said that Comcast would not send my ticket to engineering because the RFC 2460 spec was still in draft and that I was a residential client.  She said that if I was a business client, then she would escalate it.  I explained to her that the spec was a DRAFT STANDARD (meaning it was stable) and that all of IPv6 was based on it.  That did nothing.  I pointed her to a thread in the Xfinity forums that explains the problem, and she actually locked that thread without reason while on the phone with me - that was rude.  She was a decent person, but she was likely following her manager's orders who clearly had no time to deal with engineering issues for residential clients.  
I decided to go another route and reach out to some higher ups myself.  A guy by the name of John Brzozowski, who is also known as ComcastJohn in the Xfinity help forums, touts himself as the IPv6 lead and Fellow for Comcast.  He has quite the pedigree as it appears that he has had his hands in a few RFC standards surrounding IPv6.  I figured he would understand the problem and act quickly, as surely he would care.   He constantly posts in the Xfinity and Reddit formus to reach out to him directly if you have any problems issues with Comcast's IPv6.  I did just that.  3 emails. 1 Xfinity forum private message.  1 Reddit private message.  Number of responses from him: 0.
I reached out to some other people on the Reddit IPv6 forums and the Cisco support forums to see if others experienced similar issues.  Sure enough we found that Comcast's support of the hop limit appears to be regional.  One person from Reddit who has his 5506-X working lives in the Twin Cities Metro area of Comcast on residential service.  His packet dumps yielded a hop limit of 255.  Hence his Cisco ASA accepts the packet.  A person on the Cisco forums lives in the South Florida area using residential and he receives a packet with a hop limit of 0 as well with a 5506-X, and he also is unable to get IPv6 working for the same reasons as me.  I live in Colorado in the Denver area, and well, you know the story on my experience with their IPv6.  So the region appears to have an impact.

Today: Now What?

This leads us to today.  Comcast appears to gloat on the net that they have one of the most advanced IPv6 networks in place are hopefully looking to proliferate that.  But they aren't fixing some of the nasty bugs.  The Cisco ASA products are a staple for people who work from home or have small businesses to connect to their mother ship.  Many companies dole these firewalls out and only allow those to work with their corporate offices.  Until Comcast takes these users seriously and applies the RFC 2460 spec in its entirety, their proliferation with IPv6 will halt.  In time, more and more routers will support the specification, and less and less will work with Comcast.  At some point they will need to acknowledge and fix the problem if they want serious users to leverage their IPv6 stack.
I wrote this blog post to share with people who have attempted to get their Cisco ASA equipment to work with Comcast's IPv6.  I spent a lot of time figuring this out, so it is my hope that if it saves someone else time, I have done my job.  In the mean time, if you are a Comcast customer and own a nice router/firewall, stick to good old IPv4 and hope that one day Comcast will properly implement their DHCPv6 stack.

*UPDATE 5/10/2018* 

ArrisTuska (aka Chris Tuska) who is an outstanding Arris Engineer confirmed the problem is with the Arris CMTS software.  He was instrumental in agreeing this was a bug and getting the patches out to Comcast and getting them deployed to their CMTS units.  You can read the entire thread here.

Monday, January 2, 2017

United Airlines: Customer Service at its Worst

Boy how times have changed regarding air travel.  When I was a boy, air travel was a great experience.  You dressed up nice for a flight and you were treated like royalty.  You had a decent meal and even got a nice beverage and snack.  It was an experience to look forward to.  Fast forward to today, and the experience borderlines riding in overcrowded public transportation (like a bus) with nickel-and-diming, ever-smaller seats, fees galore, and some of the rudest employees an industry could ever could ever tolerate.  The rudeness has gotten so bad, that the airlines appear to enjoy a monopolistic customer service experience with their employees that would fire a McDonalds fast-food employee on the spot if it was delivered in that particular industry.  The airline industry as a whole needs a reboot and to re-learn what customer service is all about.  Here is my story...

As some background, I have been a loyal United customer for many years.  I have been 1K and usually make Platinum level each year (these are top elite places in United's mileage plus program).  I do this to help avoid the lines, prevent getting crappy "B" seats when I fly,and having a place to store my carry-on since I get ot board first.  Every year I take my family away during Christmas for a holiday to an international destination.  I usually do this as a mileage run, but the economy experience has gotten so rotten that I generally pay to put the wife and me in business class.  It gives me 1.5 PQM (elite miles) and is an easy way to be sure I am able to attain my status.

This year I decided to take my family of 5 to Hong Kong for Christmas/New Years.  In early October I found a decent fare and booked my family from Denver (DEN) to Hong Kong (HKG) through San Francisco (SFO).  As a platinum member, I get the cool option of being able to book the economy folks (my kids) in Economy Plus, so its a bit more of a comfortable ride for them.  I got a couple of nice seats in business for me and the wife, and the kids sitting together in economy plus.  This was going to be a trip to remember!  Turns out I was right... but not the way one would hope.

The troubles started when we departed on 12/23 from DEN-SFO.  We had planned about a 2 hour layover in SFO, but SFO was having some rain problems, so our flight was delayed... for 1.5 hours.  That was going to make getting to our SFO-HKG flight a challenge.  I spoke to the flight attendant on the Denver flight and she said "You will be fine, you will have 20-30 minutes to spare".  So off we went with confidence that all was fine.  But we landed and had 15 minutes to get from one terminal to the other to catch our Hong Kong flight.

I fly internationally quite a bit and I cannot tell you the number of times I sat on a plane and the pilot announces "Hi folks, we will be delayed another 10 minutes as we are awaiting a group of passengers who are connecting are are currently in transit to our flight".  Knowing this was the case, I was very confident that United would be sure that we got on the plane.  I mean, come on, they track your bags so they must track you, right?  They have to know where you are, right?  That appeared to be an invalid assumption and I guess we are not afforded the same courtesies that they seem to grant other passengers.  When we landed in SFO, we RAN, and yes I mean RAN, to our other gate.  I had my boys run there faster since we were carrying the carry-on and figured they could tell the gate agent that we are in transit and it be there in a few minutes.  When my boys arrived at the gate, they went to the ticket agent and said "My family is running, they will be here in a minute or two".  The agent looked at them and said "Well, they better make it fast because we are about to close the doors".  As my boys had told them, we arrived there literally 1.5 minutes later.  We watched a passenger get on and as we walked up to the door, the ticket agent closed the door behind him and said "I'm sorry, we are closing the door.  You need to go to customer service".  My wife looked at him and said "You just let that passenger get on!  Why couldn't you let us on?" He said, "I'm sorry, you aren't going to like to hear this but we gave your seats away to people waiting for an upgrade".  Wow.  They gave away our seats knowing fully well we were in transit by literally minutes.  Not only am I top-tier elite status but we additional paid for our seats.  The sad thing is that about 8 customers after us had the same issue.  You would think the airline would hold the plane for literally minutes to accommodate connecting passengers and stop he hemorrhaging. 

But it gets worse.  To add insult to injury, the flight from SFO-HKG had a mechanical delay.  By 52 minutes.  They could have easily let us on and put those "upgraders" back in their crappy seats.  Its been done before.  Thats when United's famous rudeness kicked in.  The ticketing manager dragon (that's airline speak for the person who protects the airline and treats you like poo) told us this is the way it is and she was he one who made the decision.  She raised her voice to us and was very unpleasant with a "too bad" attitude.  United generally could care less about their employees' bad behavior, so complaining about will not yield any results.  Sometimes I wonder if they reward their employees for crappy attitudes, offering a "crap-head bonus" to those who treat customers so awful.  Merry-f'ing-Christmas, United!

We went back to the United Club to talk to a customer support person to hopefully 240 us (airline speak to get us on another flight on a competing carrier) or get us on the next flight out.  The ticketing customer services began looking and said "I'm sorry but I have nothing for 3 days".  I pulled up on my phone and said "Wait, I see several flights going out of Seattle, Vancouver, and Los Angeles today, why can't you get me on those flights?"  I was explained that they do not have agreements with Cathay Pacific or Virgin air.  Gosh, I remember a time when you could fly on any of the carriers when you got 240'd.  Now they pick and choose to a select few?

I looked on Kayak again and I saw that the next United flight the next day had several 1st class seats open and they certainly could accommodate us.  The ticket agent said "I cannot give those to you."  I asked why not.  The response was that our tickets are not eligible for a 1st class seat.  Wait.  United inconvenienced us by giving away our tickets, making it so that we won't be able to fly for days, and they cannot give up their precious 1st class seats to us?  Times have surely changed.

Yes.... it gets even worse.  The United agent basically said we can wait for 3 days but they will not pay for 3 days of hotel.  Wha?  They inconvenience us and want us to pay for our own hotel?  Not to mention, by waiting 3 days, we burn half our vacation, so whats the point?  She told us she then had no choice but to give us a full refund and send us home.  We didn't agree or disagree.  We would call the Platinum desk and hopefully get someone with a bit more pull to hopefully make something happen.  As I was talking to the Platinum desk, that agent told us that they couldn't get into the record because someone was working on it.  I went over to the top dragon in the United Club and asked what was up.  This woman was terribly rude.  She showed me our tickets.  We were re-routed back to Denver on 3 different flights because there was no room.  She said she had to pull favors to get us back to Denver.  Imagine, my 10 year old son flying by himself back to Denver.  Yeah, that wasn't going to happen.

I spoke to her and said "Come on, this was my Christmas present to my family and they have to be more cooperative."  The woman was rude and told us there were no other options.  But my my luck, a nice Indian woman agent was sitting next to her and said "Please, let me try".  She immediately pulled up flights to get us to Hong Kong.  She showed them to the manager and the manager rolled her eyes. Wow... the manager was lazy... typical United behavior.  So now that the manager was busted, she had to start looking.  She found a flight to Beijing the next day with a connecting flight on Air China to HKG.  But it was in economy.   I told her that we paid for business class and for us to fly in economy is not acceptable, especially without some form of a refund.  She glared at us and said "Look, this is what I can get you.  You should not question this.  Hold on... I will continue to look."  She came back and said "ok, I have you in business class".  She gave us these blank tickets and told us to come back in the morning.  She said we do not get hotel because it was weather related (yeah....right... weather related - airline excuse for we don't want to cover your hotel).  The nice Indian ticket agent told us "Don't worry, I got you taken care of".  She gave us vouchers to a shitty Holiday Inn (I guess better than nothing) making it look like they were doing us a favor and told us to get our tickets at the counter in the morning.  She said we could get our bags and bring them to the hotel.

Yes, it continues to get worse.  We went down to the baggage claim and asked for our bags.  SFO was trying desperately to track them down, when they found that the wonderful ticket agent manager who processed our "refund" sent the bags back to Denver.  She forgot to get them when getting our new tickets to HKG.  Oops.  No bags.

Exhausted for the multiple hours of trying to have ticket agents do their "magic", we finally got to the hotel and promptly fell asleep.  Our plan was to get up early and get to the airport early to hopefully head off any potential problems in the event our good luck would continue.   We went to the ticket counter and the agent handed us our tickets with a hearty "Enjoy your trip!".  We walked away from the counter and looked at our tickets when something caught my eye.  The belligerent ticket agent from the day before put us in economy.  WTF?  So we ran to the international ticket desk as time was ticking by and we were going to once again miss our window to somehow get to Hong Kong.  We had to get in line and wait... and wait.  We got to an agent and for the first time we got someone who actually cared.  She heard our story and looked at the tickets and finally said "This is completely unacceptable.  I am very sorry.  Let me see what we can do for you".  She had us get into a line whereby we would talk to their top person.  We waited again.  We finally got to speak to a German lady who was the top ticketing manager in the international SFO desk.  In the beginning she wasn't too friendly.  She had a "tough crap, its Christmas and we are overworked" attitude.  But then magically, her face completely changed.  I am not sure if the spirit of Saint Nick tapped her on the shoulder and told her to get a grip, but she finally had a look of pity.  She said "Let me see what I can do for you".  She worked... for about 45 minutes on the computer.  She came back and said "You are set.  I have you on the same flights and you are in business class all the way through.  She said "I am very sorry for you.  During the holidays we hire temproarily and we get people who have no idea what they are doing."  I told her that her desk was the first people who have been anywhere near helpful.  She told us "Get going... you have 20 minutes to get to your plane".  So off to Beijing we went.  The flight was uneventful and we were connecting to Air China from PEK to HKG.

Does it get worse? Yep... you guessed it.  It certainly does.  We get to PEK and we were told to obtain our bags as Beijing will not transfer bags to connecting flights.  So we get to PEK and guess what?  No bags.  We put in a trace and the couldn't find them.  Our only possibility was to get to HKG and deal with it there.  We get on our Air China flight and we started to giggle.  When we got on, all of business class was essentially empty as was economy plus.  The only thing that had passengers was the very back, or economy section.  This underlined that the original ticketing agent was trying to make our trip as miserable as possible.  Thank goodness for that last ticket manager who made it happen.

How much worse can it get? Well, it can.  We get to HKG and of course no bags.  We put in a trace and we were told the bags finally made it to PEK and will be on their way to HKG on the next flight out.  But of course that didn't happen.  The next day we had the hotel try to get a trace on our luggage.  They were finally successful and we were told it would be delivered in the evening.  A full 2.5 days without our luggage and medications.  Thank goodness prescriptions weren't needed in Hong Kong Pharmacies and we were able to obtain what we needed.  We had to purchase clothes, toiletries and other items to tie us over until our baggage arrived.  The rest of the week was pleasant as we were finally out of the clutches of United Airlines and in the hands of the JW Marriott.  Yes Marriott is certainly a company that gets customer service and they made our trip delightful.

It can't get worse now, can it?  Yes... it can.  We unfortunately had to return home via United Airlines.  We hoped that the long haul would be uneventful as we were sure to get to the airport early so we wouldn't have any snafus.  But we didn't think about the fact that we could get on the plane and what would happen there.  We get on and I am sure that my kids have their iPads all charged for the long trip to attach to United's new-fangled wifi whereby they will stream movies directly to your device.  We get on tand United announced that their wifi is down and not functioning.  Oh shit.  I have young kids who are going to stare at the back of their seats for 12 long hours.  My youngest, who has A.D.H.D. is going to be a wreck.  He is going to have a tantrum and the flight attendants are going to get us and complain - yes this has happened before.  This will be a disaster beyond proportions.  Luckily, we were in luck.  The kids slept most of the flight.  But how the heck United can have a completely malfunctioning entertainment system in place for a long haul is unacceptable.  I felt very sorry for those who were not given a  way to watch  movies.  You would think United would have a quick way to swap out their wifi systems in the event of failure, or possibly even a backup.  Being in the computer industry, I know it doesn't take much to think about a fault tolerant system for these sorts of things.

But it does get worse.  We are up in business class, and after we take off and a reasonable meal, we decide to sleep.  I put on my Bose noise-cancelling headphones in hopes to get some peaceful sleep.  But I can't.  Certainly it was not due to my lack of being sleepy.  It was the flight attendants.  They decided to camp out near our seat and chat.  For 12 hours.  Loud.  I could hear their cackling and clucking right through the headphones.  They weren't even trying to be quiet.  They had long conversations about politics, Trump, ISIS, flying, what-to-wear.  For....12...long...hours.  I have never seen a company allow their employees to water-cooler for so long.  Does United not have a policy regarding being quiet when the lights are out and passengers are clearly trying to sleep?  You would think the chief purser would tell the others to STFU, but its difficult when the chief purser is also involved in the activity.  Who watches the purser?

And worse it still gets.  As I mentioned before, as a Platinum elite, one of the perks is that I get is I get to put my party in Economy Plus.  I had them all booked in a single nice row. Well, some ticket agent along the way probably saw 3 non-elites flying in a nice row and thought "Hrm, I need to put my 1K or Global Service customer in a seat, so I will bounce on of those kids".  Yep, that's what they did.   You would think that they would check the PNR code and see that those kids are attached to a Platinum itinerary.  But United doesn't care  Why offer the perk if they give it and then taketh away for yet anohter elite member.  Not cool.  Just don't offer it if you are going to just take it back.

Where are we are at today?  I will likely complain to United and they will probably offer me some token crappy number of miles or a small discount on my next flight.  Of course it will not acceptable.  They SHOULD apologize profusely and refund me me a decent amount for downgrading us from a direct long haul to one with a connection and for interrupting our vacation when they actually had the power to ensure we were on that original flight, in addition to their token crap they usually offer.  Maybe I will be surprised and they will go above and beyond to make us whole again.  I can dream, can't I?  But knowing United, I will get their usual form letter.  They don't care.  They don't need me, no matter how much business I give them.  I own a company whereby the majority of my guys fly United for business.  If they counted the amount that we spend per year on them, I would be Global Service.  But that's not the way they think.  If we all voted with our wallets, then United would change their ways.  But they are such a monopoly, it will be hard to do that.

Its time to rethink my loyalty to United.  It is one of the worst airlines, and its not only me who thinks so.  They are rated with one of the highest customer complaints in the industry with a 1 star rating on Consumer Affairs and dead last in JD Powers Ratings.  You would think that an airline would do its best to offer the very best in customer service and certainly take care of their top elite customers.  The industry has changed, and certainly not for the best.  Fees, poor customer service, treating the customer like cattle, and terribly rude employees plagues one of the worst industries that we have.

Thankfully, Delta has offered to match my status.  I will likely be taking them up on that offer.  My experience with Delta has been pretty good.  Maybe... just maybe... there is light at the end of the tunnel and I can find an airline that gets loyalty and customer service.