Previously blind to possible delays during the turnaround process, more and more airports are turning to AI and computer vision to see ‘inside the box’.
Everywhere you looked at this year’s Passenger Terminal Expo in Frankfurt in April, it seemed there was a new solution designed to help airports improve aircraft turnaround performance.
“Until very recently, the turnaround has been a ‘black box’,” says Royal Schiphol Group’s Jeffrey Schäfer, process owner, aircraft turnaround. Traditionally, Schäfer says, airports allocated a gate to an airline for a flight, but what happened beyond that point was largely out of their control.
“Most of the processes are not carried out by the airport; instead, they are conducted by the handling agent or the airline. As a result, we didn’t have any insight into what was happening. For decades, we only knew we had a problem when we noticed the flight should have departed – there was no early warning or ability to predict off-block times.”
Handling agents kept their own records, applying time stamps to mark the completion of each stage (Figure 1). However, they were not always the most robust in their application. “Getting high-quality data is challenging,” notes Schäfer. “But we need to know because there are other flights – your delay might not be a problem for you, but it is for the next aircraft waiting to use your gate.”
Delays are a particular concern during peak times, when the airport needs the stand back immediately for the next flight. “It’s a revolving process,” says Schäfer. “The next flight has to wait until the stand is available, which creates extra emissions – so it’s not just about waiting times. The impact can quickly snowball.”
The airport could switch the incoming flight to a new gate, but this is never popular – with passengers or the airport: “A gate change incorporates so much movement, for staff on the ground, vehicles, baggage, cargo – everything must be moved, often at the last minute. This is why so many airports are now looking for better information – we want to have a predictable delay. Flights will always be delayed, but we want to know it as soon as possible, so we can act without hampering other processes and protect the continuous flow of flights, avoiding serious disruption.”
Schiphol was already using AI to track overall flight performance (see How Schiphol uses AI for AHM732 codes, below), which is how it knew it needed more data about turnarounds: “We already had an algorithm that can calculate the origin of a flight delay,” explains Schäfer. “We call it ADAM [Amsterdam Delay Allocation Method]. It sorts the reasons behind delays into five main categories, one of which is turnaround. And we saw that around 40% of our total delay was turnaround delay, but we couldn’t explain it any further.”
“The next flight has to wait until the stand is available, which creates extra emissions – so it’s not just about waiting times.”
Jeffrey Schäfer, Royal Schiphol Group
Deep dive
To remedy the situation, Schiphol is now using a dedicated turnaround monitoring system, developed by Aviation Solutions, called Deep Turnaround. It uses historic, real-time and predictive insights to radically improve the aircraft turnaround process for all stakeholders.
Using AI image-based processing, an algorithm detects and reports more than 70 unique turnaround events in 30 turnaround processes. It can detect delays as early as 40 minutes before the targeted off-block time, helping Schäfer and his team make more informed decisions.
“Most importantly of all, we can now predict when a flight will be ready for departure,” he says. “We can do this with greater accuracy than the handling agents can. As a result, our gate planners know an extra 20 minutes in advance when a conflict will arise and can react accordingly.”
Two cameras mounted above every stand track each turnaround event, sending snapshots to the cloud every five seconds so an end-to-end AI model can accurately register exactly what’s going on at the apron, in real time.
“The algorithm can recognize each individual vehicle or process that’s going on – not just when the fuel truck arrives and is in position, but also when the actual hose is connected from the truck to the aircraft,” explains Schäfer. “We are rolling this out for all stands at Schiphol. It’s live at 70 stands so far, with a further 30 due to come online soon.”
Capital investment
Berlin Brandenburg Airport (BER) has also embraced new computer vision technology to get a tighter grip on its turnaround, becoming the first German airport to use an AI-based self-learning software solution to optimize performance, having selected Assaia’s TurnaroundControl at 49 of its terminal apron positions.
“We noticed a lack of visibility on the turnaround,” explains Thomas Hoff Andersson, BER’s COO, about the situation prior to the adoption of TurnaroundControl. “Sharing real-time information between airlines, ground handling and airports was impossible. We found it very difficult to prioritize decision making between the airlines, ground handling, airport and Eurocontrol. As a result, our on-time performance was being impacted negatively by the reduced visibility of all the moving parts taking place on the turnaround. Our aim was to improve on-time performance for our passengers and airlines, as this impacts their profitability.”
The airport has already deployed the technology across half of its stands, providing process managers with instant updates via specific alerts. BER plans to retrofit the remaining stands by the end of July 2024.
“Process managers can now communicate with ground handlers – when they receive an alert, they can liaise with the relevant stakeholders,” continues Andersson. “They are already responding to alerts from the system when events such as catering, baggage loading and off-loading are not occurring as planned; and are getting in touch with the ground handlers to rectify the situation in a timely manner, so that the passengers can still reach their destinations on time.”
“Process managers can now communicate with ground handlers.”
Thomas Hoff Andersson, Brlin Brandenburg Airport
The insights gained allow process managers to have fact-based conversations in the operations center and act in real time to step in to try to remedy any potential problems.
“We have created a resilience team whose task is to ensure turnaround events are occurring as per the predictions and in all situations,” explains Andersson. “For example, if we receive alerts about an event such as baggage unloading not occurring according to schedule, the process manager can request they step in to assist the ground handlers to ensure the passenger is not delayed and the aircraft can take off for its next destination on time. This is also helping us ensure all stakeholders adhere to the agreed service delivery plans. The beauty now is that we can combine the effectiveness of our human talent, digital and physical assets to ensure we are giving the passenger the best-possible experience.”
Fault finder
Schiphol’s Schäfer says these new AI tools are not there to apportion blame, but to assign help: “We did not implement this to prove the handlers are making mistakes, but to find where the exact problem lies,” he says. “It’s not about blaming, it’s about improving operations, because we want more on-time flights. Before we had this data, we only had written reports from our operational team, while the handling agents had the same from their staff. Any discussions about what went wrong were rarely productive because we had our opinions and they had theirs, and quite often we were on opposite pages. Now we can take the data or the video footage to see exactly what happened.”
Staff shortages, rather than individual processes, continue to cause challenges: “It’s not that any one particular task or stage is more likely to go wrong,” notes Schäfer. “What we have been affected by most is staffing levels. There are 1,001 reasons why a process might have been delayed, from staffing issues to the previous task running late and having an impact. Now we have AI, rather than just having an incident report where the handling agent says it was an issue with PRM boarding, for example – which is the responsibility of the airport – we can now see that there’s a second layer that’s caused the delay to PRM loading. It’s nearly always complex and it’s rarely linear.”
The increased visibility means handling agents no longer need to constantly report the flight’s status back to base: “In the past, when there was a problem, the agents had to waste time communicating this to the control center, by calling or even driving over – but now they can just get on and fix it. Removing the workload of communicating and reporting ensures they can focus on the main task of making the flight safe and on time. This is a major improvement.”
Ground handlers can also use Deep Turnaround to better manage planning: “The main benefit is a huge improvement in predictability,” says Schäfer. “Our handlers have doubled their ability to predict and forecast departures and delays. They can then optimize their resources because they know who’s going to be ready first, so they can reallocate them to enable the next on-time departure. It’s all about collaboration. ATC can also see which flight is going to be ready first, which ensures we also make the most of our own scarce resources – runways and gates. Optimizing turnaround improves throughput and on-time performance.”
For Berlin’s Andersson, the application of AI and computer vision to the turnaround process has been worth every cent: “Financially, the investment was easy to justify based on the benefits and expected benefits TurnaroundControl will bring in terms of efficiency, safety and sustainability,” he concludes.
“This is just the beginning – we are only using a limited amount of the potential we can get from the technology.”
Thomas Hoff Andersson, Berlin Brandenburg Airport
“This is just the beginning – we are only using a limited amount of the potential we can get from the technology. We are already thinking about how it can help us in other areas – for example, keeping passengers updated on when their aircraft will be ready to board, based on real-time data rather than the current estimations we all use. I feel there will also be further benefits to come. For example, enabling us to identify safety matters, such as if passengers are on the apron or in the wrong place.”
How Schiphol uses AI for AHM732 codes
On-time performance is crucial in aviation, and pinpointing delay sources is essential for maintaining efficient operations. IATA’s AHM732 delay code scheme marks a significant advance, providing detailed, unbiased insights into delays. However, it also adds complexity, with an overwhelming number of codes for dispatchers to choose from in real time.
At the recent Passenger Terminal Conference, Schiphol’s Jeffrey Schäfer made a presentation in which he argued that the path to superior delay reporting lies in leveraging AI algorithms to aid decision makers in selecting delay reasons.
“Reporting on flight delays based on the new delay code scheme AHM732 is vastly more complicated than using the old scheme,” he says. “The new delay code scheme has more than 10,000 delay reasons to report on. Our AI-based systems, such as Deep Turnaround but moreover ADAM, enable us to automatically report up to 80% of the occurring delays fully automated, leaving more time for staff on the ground to perform main duties – or have a well-deserved rest after their turnaround departs.
“Reducing delays and improving turnaround times begins in understanding the origin of delays in the first place,” he continues. “The bigger picture (more than only the turnaround process) is vital here: the devil is in the details and the dynamics of flight arrival – turnaround – departure and the influence they have on each other matter a lot.”
Schäfer gives the following example: If a flight arrives too late, it affects the ability of handling agents to turn around the flight in an optimal way as scarce resources and staff might have been reallocated to another flight. When looking at just the turnaround, it might appear that the handler was delaying the process but in reality the origin lay in the arrival delay.
“All these details can be – and are – captured by our data,” he says. “This enables reporting to become easier and brings more accurate information at the same time. This way of working and better understanding the origin of delays will put us all on the right track from the early start as we are now focused on improvement strategies instead of finding root causes.”
However, the airport is not the only actor in the ecosystem that has a role in the end-to-end journey that an aircraft, passenger, bag or piece of cargo makes through the airport. Also, most delays and complex processes occur at airports – not in the air. As a result, Schäfer argues that airports have the most knowledge, but only if they collect the right information in the right processes.
“The airport should not only act as a hub for flights but also for data and insights: if airports can standardize insights and make them available for all, everyone can benefit and improve their own operation,” he says. “Then I’d like to coin the performance strategy, which in our opinion the airport is in charge of – the four CARE pillars: capacity, assets, reliability and execution (Figure 2). At Schiphol, all performance management activities can be put under the umbrella of the CARE pillars and are the core of our own activity, aside from assisting others (small and big) in improving their performance through collaboration and information sharing.”
All airlines operating at Schiphol are constantly analyzed for their on-time performance, with the airport using the generic ADAM algorithm results to see the breakdown of causes of flight delays. Schäfer says it’s fascinating to see the differences between them (see graphabove).
“This helps us tremendously in finding the right things to talk about with the specific airlines, as we understand the nature of their challenges,” he concludes. “If I were to approach Airline 7 to address reactionary delays or talk to Airline 10 to address schedule delays, they would call me a fool!”
GDPR compliance
Despite the clear benefits of using computer vision to track turnaround progress, some airports may have concerns around GDPR. Schiphol’s Jeffrey Schäfer says Deep Turnaround cannot identify individuals: “We’re not using high-resolution 8MP cameras,” he says. “We’ve positioned them high above the gate and we’re well below the threshold of the number of pixels needed between the eyes to recognize a person. None of the images can be downloaded by companies or individuals. The dashboard just shows a snapshot – it isn’t a continuous video feed where you can see people walking around. From a GDPR perspective, there’s a big difference between still photos and moving video.”
Dusseldorf’s checklist
Henning Pfisterer, senior VP aviation and accountable manager at Düsseldorf Airport (DUS), is currently looking to install an AI-based turnaround monitoring system at his airport: “We are seeking to collaborate with a supplier who is eager to consistently enhance system capabilities alongside an experienced and demanding client in search of solutions for complex operational challenges,” he says. Pfisterer offers the following advice to airports…
- The key consideration in implementing new technologies to optimize turnaround is to establish the business case concerning the return on investment, achieving a fair costsharing arrangement with the airlines and ground handling services, which are the primary beneficiaries of such technologies. However, it is likely the airport operator must take the lead.
- Most turnaround management systems use video analytics and depend on full CCTV coverage of the apron area. Many legacy CCTV systems do not fulfill the specific needs of advanced turnaround management systems and necessitate significant infrastructure expenses. Having an open architecture is essential for systems to integrate airport-specific needs.
- Effective turnaround management isn’t solely about technology; cultural aspects must also be considered. DUS fosters a culture of operational excellence within its airport stakeholder community, where top management commitment and dedication to operational excellence drive performance improvements. DUS expects full commitment to operational performance from the shop floor to the board level and for all to actively ‘live’ this idea.
This article originally appeared in the June 2024 issue of Passenger Terminal World. To view the magazine in full, click here.
