In safety-critical engineering, strong technical performance is only part of what makes a system useful and reliable. Engineers also need to understand how decisions are made, where accountability sits and what can be learned when something goes wrong.
All these questions are at the heart of a recent episode of The Innovation Crowd, a podcast hosted by innovation specialist Helen Dawson, featuring Jonathan Burnip, Head of Advisory Services at Marshall Aerospace.
In the second of three articles unpacking Helen and Jonathan’s discussion, we explore how AI can support engineering decision-making without undermining human judgement or weakening the safety culture around it.
Jonathan Burnip is Head of Advisory Services at Marshall Aerospace and a Fellow of the Royal Aeronautical Society.
The Advisory Services team draws on Marshall Aerospace’s experience gained through complex engineering and integration programmes to help innovators address new technical and organisational challenges.
Learning when things go wrong
Aerospace safety depends heavily on understanding and responding appropriately to incidents, hazards and near misses.
One of the principles supporting this approach is a just culture, in which people are encouraged to report mistakes and concerns without worrying about facing unfair punishment.
“It’s important to gather all the safety learning possible from an event or a near miss, and use that to improve in the future, in the hopes that it never happens again.”
As Jonathan points out, incidents in complex systems rarely result from one isolated action: equipment, processes, operating conditions, information and human decisions can combine in ways that allow several layers of protection to fail.
Investigating those interactions helps organisations understand what needs to change, although AI can complicate this process if the behaviour of a system cannot easily be explained or reproduced.
Jonathan uses automated aircraft inspection as an example. Machine vision could potentially identify defects and help engineers focus their attention on areas that deserve closer examination. If an AI-supported system misses something, investigators would still need to understand how the failure occurred and what could be changed to prevent it happening again.
“AI should never go into a place in a safety-critical industry where it undermines my ability to learn from what has gone wrong or what nearly went wrong”
With some AI systems, it may be difficult to explain precisely why a particular output was produced or to recreate the same result. In an industry where understanding the causes of an incident is central to improving safety, that limitation has to inform where and how these systems are used.
Human judgement and accountability
Keeping people accountable for safety-critical decisions is not the same thing as assuming that human judgement will always be better.
People make mistakes, while AI may identify patterns, anomalies or defects that an individual could overlook. The ability of AI to process information and highlight areas requiring attention could become extremely valuable, as long as the chain of decision-making around the technology remains robust and understandable.
“I believe that the role of AI in these safety-critical industries is as an aid to decision-making: it should help people make decisions and gain insights into what's happening, but it should not make the decisions.”
The suitability of the system for its task, the way it has been tested, the information presented to the user and the reasons for accepting or challenging its recommendation can all be examined when an outcome needs to be investigated.
In this model, AI supports the decision by providing information and insight, while a person retains responsibility for the safety-critical judgement. That arrangement only works if the person still has enough information, expertise and authority to exercise genuine judgement rather than simply confirm the output produced by the system, so accountability needs to be considered throughout the design of the process.
More than a “human in the loop”
The phrase “human in the loop” is often used as shorthand for responsible automation, but it says little about the nature and quality of human involvement.
A person might technically remain responsible for approving an AI-supported recommendation, while having very little practical reason to challenge it. If an automated system produces reliable outputs repeatedly, accepting its conclusions can gradually become routine. Over time, this can affect both behaviour and capability.
“The danger is that if you design a system that is an aid to decision-making, it becomes the thing making the decision because we implicitly trust it.”
Jonathan illustrates this through the everyday example of how navigation technology in cars can “detrain” drivers. While digital systems undoubtedly make unfamiliar journeys much easier, regular reliance can turn into dependence, making people less practised at navigating and less able to do so independently.
The consequences are very different in safety-critical engineering, but the underlying effect is recognisable. A person who rarely has to exercise a particular skill may gradually become less effective at it.
For this reason, Jonathan argues that human factors need to be considered alongside the technical design of an AI-enabled system. Engineers need to have enough context to assess recommendations properly, and their role needs to preserve the knowledge required to recognise when an output does not make sense.
Helping people make better decisions
AI assistance does not always need to take the form of providing a direct answer.
Jonathan returns to aircraft inspection to illustrate this point. A system might identify a particular area and state that a defect is present. Alternatively, it could flag the area for closer examination and prompt the engineer to review their own work.
The second approach leaves more of the judgement with the engineer, while still using AI to focus attention.
“Over time, the system is helping me to be better at my job, that critical job, rather than becoming something upon which I rely.”
Jonathan also refers to research he has encountered involving AI-supported coaching for wire locking, a process used to secure critical fasteners so that they cannot work loose.
In the example he discusses, a system that simply showed someone how to complete the task did not necessarily improve their ability. Prompting the individual to reconsider whether they had completed it correctly produced a different effect by encouraging them to apply their own judgement.
The appropriate approach may well vary across applications, but the effect on human performance over time deserves attention alongside the immediate quality of the output.
Looking beyond technical conformance
Assuring an AI-enabled application will involve demonstrating that the technology meets the necessary technical requirements. Its interaction with the wider engineering environment also needs to be understood.
A system may perform well in isolation while producing unintended effects if users become overly reliant on its recommendations or lose expertise needed to challenge them. Conversely, a well-designed tool may improve both the decision and the capability of the person making it.
This is familiar territory for an industry accustomed to considering people, equipment and procedures as parts of the same wider system. AI adds new interactions to analyse, but human factors remain part of the safety case.
The ability to investigate incidents remains equally important. Clear responsibilities, appropriate records and an understandable basis for decisions help preserve the learning culture that has underpinned improvements in aerospace safety.
As AI takes on a greater role in activities such as modelling, coding and analysis, many of the tasks it accelerates are also tasks through which engineers have traditionally built experience and judgement.
The final article in this series looks at how that development path could change, and what aerospace organisations may need to do differently when training and supporting the next generation of engineers.
In the meantime, you can listen to Jonathan’s full conversation with Helen Dawson on The Innovation Crowd.
Images used in this article were created using generative AI.