When I first thought about the 2013 crash of Asiana Airlines Flight 214, overreliance on automated flight systems was the explanation that came to mind. It was easy to read the accident as a story about people forgetting how to do work that automation had taken over. The investigation was more specific than that.
The US National Transportation Safety Board identified approach management, an unintended loss of automatic airspeed control, airspeed monitoring, and the timing of the go-around as causes. Inadequate documentation and training about the complexity of the automation were among the contributing factors.1 The accident required examining the automated systems alongside human understanding, training, and coordination.
That distinction matters when building AI. Whether a system works correctly, whether a person understands its current state, and whether they can respond to a problem are separate questions. Effective automation and a person prepared to respond should both be design objectives.
Performing and overseeing require different abilities
Most of the time, the system handles the work. A person monitors the outcome and responds to exceptional situations. That role requires understanding the system's limits, deciding when direct action is appropriate, and maintaining the skills needed to take that action.
Aviation has addressed this problem for years. FAA guidance issued in 2017 recommends manual flight training and proficiency assessment, while allowing crews to choose an appropriate level of automation for the workload and conditions. It also cautions against unsupported manual-flight quotas and overly general rules.2
There is a concrete principle here for AI. The ability to operate automation and the ability to handle exceptions both need maintenance. Instead of repeating every step of real work, organizations can design separate practice for recognizing and responding to important failures. The first decision is which abilities the task and its risks require people to retain.
Governance starts before the work does
AI governance often brings policy documents and committees to mind. Inside a product, it can start with smaller questions. Who owns this task? What may the AI decide? Under which conditions should it seek a human decision? Who can revise or reverse the result?
The answers should be established before execution. Consider a message to a customer. Drafting it internally and sending it externally have different consequences. A draft can go through several automated revisions; sending requires criteria for the recipient, content, and authority. Governance establishes the scope of automation, the conditions for intervention, and responsibility in advance.
The EU's AI Act, adopted in 2024, includes human oversight among its requirements for high-risk systems.3 The useful principle is that requirements vary with risk. An everyday recommendation and an employment decision need different review arrangements. Each should be designed around its consequences for people.
Reviewers need authority to decide
The presence of a review step is not enough. If a reviewer finds an error but needs separate permission to halt the work or change the result, their role is limited. It also matters whether they have the information and time needed to review it properly.
Responsibility should come with defined authority. Which decisions can the person change directly? When should they involve another owner? How will the organization handle a rejection? The person responsible needs the practical ability to change the decision. That requirement applies both to the product interface and to the organization operating it.
Operational records should serve the same purpose. Along with the approver's name, record the information they saw, their reasoning, and the changes they made. When a problem occurs, those records can support a review of the system and the process. Assigning responsibility and identifying what needs improvement are different questions.
Human intervention has conditions too
One limitation deserves attention. Combining people and AI does not always improve performance. A 2024 meta-analysis of 106 experimental studies found that, on average, combined performance was below that of the better standalone participant, human or AI. Outcomes varied with the task and their relative abilities.4
Intervention therefore needs to be designed by task. Repetitive work with verifiable results may proceed automatically once it meets defined criteria. Decisions involving competing goals or consequences that are difficult to reverse may need informed human judgment. Guaranteeing the authority to intervene is different from requiring intervention every time.
These arrangements also need evaluation in use. Does intervention improve accuracy? Does it reveal errors earlier? How much review time does it require? The criteria that trigger human involvement should themselves be open to revision. NIST's AI Risk Management Framework likewise connects governance with identifying, measuring, and managing risk.5
Prepared people and effective automation
What I want to take from aviation is a structure, rather than a particular flight procedure. Manage automated performance and human response capability separately. Define roles and authority. Improve operating practices using evidence from real work and training. Specific methods must fit the tasks and risks of each industry.
While AI performs the work, a person need not manipulate every step. They do need to understand the delegated scope and be prepared for important changes. When necessary, they should be able to revise the result and explain why. AI governance carries the conditions for human accountability into everyday operations while automation continues to work.
Notes
[1] NTSB, Asiana Airlines Flight 214, DCA13MA120, accident report AAR-14/01, 2014. Referenced with the distinction between probable causes and contributing factors retained. ↩
[2] FAA, SAFO 17007: Manual Flight Operations Proficiency, May 4, 2017. Guidance on training, proficiency assessment, and context-sensitive decisions about manual flight. ↩
[3] European Commission, AI Act enters into force, August 1, 2024. Referenced for the risk-based approach and the inclusion of human oversight among requirements for high-risk systems. ↩
[4] Michelle Vaccaro, Abdullah Almaatouq, Thomas Malone, When combinations of humans and AI are useful, Nature Human Behaviour 8, 2293–2303, 2024. Meta-analysis of 106 experimental studies and 370 effect sizes. ↩
[5] NIST, AI Risk Management Framework, AI RMF 1.0, 2023. Organizes risk management into GOVERN, MAP, MEASURE, and MANAGE. ↩