Welcome to The Long Viewâwhere we peruse the news of the week and strip it to the essentials. Letâs work out what really matters.
20 Years of Neglect Led to âMeltdownâ
Last monthâs dÃĐbÃĒcle of canceled flights was caused by decades of technical debt. Thatâs the analysis of Columbia University professor Zeynep Tufekci.
Analysis: SWA needs a cloud burst
Although there were several contributing factors, a lack of scalability in a critical crew scheduling system led to days of near-total paralysis: In many cases, the staff were in the right place to fly and crew the planes, but the SkySolver system had no way of knowing that. Making things worse, manual fallbacks collapsed under the weight of the workload.
Whatâs the story? Prof. Zeynep Tufekci explains technical debt to a mainstream audienceââThe Shameful Open Secret Behind Southwestâs Failureâ:
âTechnical debt is real debtâ
Itâs been an open secret within Southwest for some timeâŊâĶâŊthat the company desperately needed to modernize its scheduling systems. âĶ This problem â relying on older or deficient software that needs updating â is known as incurring technical debt [and it] appears to be a key factor in why Southwest Airlines couldnât return to business as usual the way other airlines did after last [monthâs] major winter storm.
âĶ
When hiccups or weather events happen, the employees have to go through a burdensome, arduous processâŊâĶâŊbecause Southwest hadnât sufficiently modernized its crew-scheduling systems. For example, ifâŊâĶâŊtheir flight was canceledâŊâĶâŊemployees have had to manually call in to let the company know where they are [sometimes] being left on hold on the phone forâŊâĶâŊhours just to let the company know their whereabouts. âĶ Online forums are full of employee accounts of such misery.
âĶ
This can easily cascade to a systemwide halt. âĶ Such breakdowns resulting from technical debt are often triggered by external events, like weather. âĶ So why didnât Southwest simply update its software and systems? âĶ Updating software is costly and difficult. [But] we canât just keep leaving the operation of more and more of our infrastructure and our lives to antiquated software. âĶ Technical debt is real debt. It will eventually be paid by someone.
And Andrew Paul knows what to blameââoutdated softwareâ:
âMultiple logistical collapsesâ
While many weary travelers are reportedly still waiting to reunite with their lost luggage, others continue to voice bewilderment at how such a logistical nightmare occurred within one of the nationâs most popular aviation providers. The answer:âŊâĶâŊemployee scheduling software that debuted around the same time as the Xbox 360 and PlayStation 3. âĶ Southwest pilots have reportedly begged company executives to update the âantiquatedâ systems since at least 2015.
âĶ
Southwestâs long-running reliance on a crew scheduler program called SkySolver is largely to blame. âĶ The nearly two-decade-old program couldnât [scale] to tackle the multiple waves of cancellations and delays. This left actual Southwest employees to manually attempt matching flight crews with available planes.
âĶ
Despite Southwestâs previous CEO Gary Kelly praising the airlineâs âwonderful technologyâ in 2021, it has since experienced multiple logistical collapses due to its long-delayed adoption ofâŊâĶâŊcloud-based data systems that can handle crises as large as winter storms. âĶ Until companies like Southwest actually start investing in bringing their software into the modern age, travelers can continue to expect future headaches.
Hello, IT. Tobias Mann asks, âHave they tried turning it off and back on again?â:
Over the past year several airlines have announced sweeping upgrades to their IT infrastructure, in part to avoid pitfalls like the one Southwest is now digging its way out of. âĶ Delta announced it would modernize and migrate its workloads to Amazon Web Services.
âĶ
American Airlines went even farther announcing a similar partnership with Microsoft to migrate its data warehousing and legacy apps to a single operations hub on Azure [aiming to] speed up bag tracking, enable preemptive rerouting based on weather conditions, and simulate larger changes using digital twins. The collaboration also involved the development of a mobile application that allows airline staff to access applications and communicate from wherever they are.
But are we really talking about debt? jgwil2 tries to unpick and translate:
Technical debt could be better explained here. I think the issue is that very old software systems tend to have a lot of technical debt, which could lead to resistance to adding new features. In this instance, the tech debt seems to have led to a fragile system that could not be updated to handle more than baseline levels of traffic, forcing the company to resort to manual solutions when a crisis occurred.
Do we need more regulation? skogs thinks not:
I don’t think we need to press some stupid government oversight into this. âĶ More useful than a thousand angry phone calls and a government writeup or fine is the fact that they literally made zero dollars during this busy time of the year. They lost money:
Aircraft that aren’t moving don’t make money.
Refunds.
Hotel reimbursements.
Lost luggage deliveries.
âĶ
This is industry correcting itself. You screw up big enough and it hurts your wallet. Ignore it and it hurts more later. Then Southwest simply goes out of business and firesales all their assets to a reasonable company that canâŊâĶâŊmanage properly.
However, Glenn Ribotsky fundamentally disagrees:
No self-respecting economy can rely on just unregulated laissez-faire capitalism. âĶ There need to be legal checks and balances on all aspects of economic behavior. Even if they might occasionally dampen those entrepreneurial animal spirits.
So itâs a management failing? SWA pilot Larry Lonero is not a fan:
âTwo decades of neglectâ
Iâve been a pilot for Southwest Airlines for over 35 years. âĶ Unfortunately, the frontline employees have been watching this meltdown coming like a slow motion train wreck for sometime. And weâve been begging our leadership to make much needed changes.
âĶ
We were watching in frustration and disbelief as our once amazing airline was becoming a house of cards. A half dozen small scale meltdowns occurred during the mid to late 2010âs. âĶ We could see that the wheels were about ready to fall off the bus. But no one in leadership would heed our pleas.
âĶ
[But] in early 2022, Bob Jordan was named CEO. He was a more operationally oriented leader. He replaced our Chief Operating Officer with a very smart man and they announced their priority would be to upgrade our airlineâs technology and provide the frontline employees the operational tools we needed to care for our customers and employees. âĶ But two decades of neglect takes several years to overcome. âĶ This meltdown was not his failure but the failure of those before him.
Does that ring true? Yes, says u/Untgradd:
My father retired from SWA as a captain after ~30 years. âĶ When I asked him, this was his take as well.
Iâve seen their bidding / scheduling software. If that was what the system looked like with lipstick on I am sure the backend isâŊâĶâŊa sight to see.
From the other side of the cockpit door, this Anonymous Coward agrees:
âGreedy, nasty, worthless bean counterâ
Herb Kelleher built the best airline in the world bar none in the decades he ran SouthWest. âĶ Then Herb retired and soon after this bean counter named Gary Kelly took over. âĶ Once Herb died Kelly was soon treating the staff as badly as almost all the other big carriers. âĶ Then he bailed last year and handed the big stinking mess on the point of collapse over toâŊâĶâŊthe designated fall guyâŊâĶâŊBob Jordan.
âĶ
The people I feel sorriest for are the employees. âĶ To see the company they dedicated their lives to, and for which they worked so hard for, utterly destroyed by some greedy, nasty, worthless bean counter.
Meanwhile, Local ID10T offers this neat summary:
Technical debt builds up because it costs money to dedicate developer time to fix the issues. So they get ignored. âĶ Every business has a problem with technical debt. Most just get lucky and don’t get publicly smacked in the face with theirs.
The Moral of the Story:
I like criticism. It makes you strong.
âLeBron James
You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or [email protected].
Image: Gabriela Natiello (via Unsplash; leveled and cropped)
