◀️ Previous Article | ▶️ Next Article | 📚 The Series Index

Executive Summary

This article argues that the greatest danger of modern machine systems is not their errors but their invisibility. The crashes of Lion Air 610 and Ethiopian 302 illustrate this: both crews were fighting MCAS, a software function they did not know existed. MCAS had been made powerful, opaque and reliant on a single faulty sensor, while being kept out of manuals and training to avoid simulator costs and regulatory scrutiny. FAA oversight, partially delegated to Boeing itself, further blurred accountability.

From this case, the article generalizes a broader pattern: hidden automation is cheaper than visible automation. Disclosure, training and documentation impose immediate costs, while the benefits of transparency—disasters avoided, human agency preserved—are diffuse. This economic logic drives invisible systems across industries, from social media ranking to driver-assistance crashes and algorithmic decision tools.

The article contrasts Boeing with BYD, whose deep vertical integration gives engineers and leadership end-to-end visibility of hardware and software interactions, speeding development and reducing blind spots. This “visibility premium” solves the Boeing-style fragmentation problem, though not the deeper opacity of neural-network systems.

The conclusion calls for a “visibility clause”: humans must be able to perceive, in real time, what machines are doing and why. If a rule-based hidden system like MCAS can kill 346 people, the article asks what future probabilistic systems might cost unless visibility becomes a design and regulatory requirement.

OPENING: SIX MINUTES AGAINST AN INVISIBLE ENEMY

On the morning of March 10, 2019, Captain Yared Getachew and First Officer Ahmed Nur Mohammod lifted off from Addis Ababa in a Boeing 737 MAX 8, bound for Nairobi with 157 souls aboard. Within two minutes of becoming airborne, something began pushing the nose of their aircraft down.

They pulled back on the control column. The nose came up. Then, seconds later, something pushed it down again.

They pulled back. It pushed down. They pulled back. It pushed down.

This struggle continued for six minutes. The pilots fought with every technique they knew. They followed emergency procedures. They cut power to the electric trim system. When that made the manual trim wheel physically impossible to turn against the aerodynamic forces at their airspeed, they restored electric power, hoping to use it to save themselves. The moment they did, the invisible force seized control again and drove the nose down one final time.

Ethiopian Airlines Flight 302 struck the ground at approximately 575 miles per hour. All 157 people on board were killed. It was the second crash of a 737 MAX in five months. Five months earlier, Lion Air Flight 610 had plunged into the Java Sea under nearly identical circumstances, killing all 189 aboard.

The combined death toll was 346 people.

Here is what makes these crashes unlike almost any other disaster in aviation history: the pilots did not know what they were fighting. They were battling a software system called MCAS, the Maneuvering Characteristics Augmentation System, and they had never been told it existed.

MCAS was not in their flight manuals. It was not covered in their training. Boeing had deliberately decided not to tell pilots about it. The company's reasoning followed a brutal commercial logic: if pilots knew about MCAS, they would need to be trained on it. Training costs money. Training raises questions. Questions invite scrutiny. Scrutiny might delay certification. Delay costs more money.

So Boeing chose silence. The machine would work in the background. Pilots did not need to know.