Collision avoidance for air taxis
By Keith Button|February 2024
If the visions of a slew of entrepreneurs come to pass, the skies around planned vertiports will swarm with electric air taxis coming and going with passengers. How will calamity be averted? Keith Button looked at one possible solution.
This story has been updated to include MIT Lincoln Laboratory as a co-developer of the Airborne Collision Avoidance System for small rotorcraft software, and to note that the project was sponsored by FAA.
On a clear day in October, at 2,000 feet above the Long Island Sound, NASA test pilot Scott Howe and a safety pilot from Sikorsky scanned the skies around their S-76 for the Black Hawk helicopter that they knew would soon be closing in. The Black Hawk also had two pilots aboard who were ready to intervene. But it was controlled by a computer, and it was doing as it was told: Fly on a course that would take it directly into the path of the S-76.
Howe wore monitors that recorded his heart rate, respiration, brain blood flow and pupil dilation as well as glasses that tracked where his eyes were focused. Howe’s helicopter was also computer-controlled, and these were its instructions: Make an evasive maneuver about one minute before the potential collision.
This scripted encounter was one of 60 or so that played out between the two helicopters in June and October off the Connecticut coast.
As companies press to begin flying passengers with electric air taxis as part of the advanced air mobility, or AAM, movement, of special concern to regulators are the areas around the coming vertiports that these aircraft must fly into and out of.
“It’s the most complex situation,” Howe says. That complexity will be compounded by “vehicles that have never been flown, in flight paths that don’t match typical flight paths, at least in their takeoffs and landings. That’s all new, and so we have to figure out how to do that safely.”
Back in October, Howe — sitting in the right pilot’s seat, with a safety pilot from Sikorsky to his left — looked down at a computer tablet in his lap. It showed a compass-like graphic depicting his heading and a yellow arc showing the heading to avoid, the one that would mean colliding with the Black Hawk.
The yellow arc grew larger as the Black Hawk drew closer. When the Black Hawk closed to within one minute of Howe, the yellow arc turned red and a green wedge appeared — the path to safety — and the S-76 turned into it. The collision avoidance software worked.
Researchers are now digging through the data they collected from that and other flights in which the two helicopters flew at each other from every angle through a range of scenarios. At least four teams of researchers this year plan to separately publish their evaluations of two distinct collision avoidance software packages, the human pilot responses during the flight, and the “middleware” program that allowed the collision avoidance software to take control of the S-76 at the appropriate time.
The scripted encounters primarily tested two research software packages: ACAS Xr (the Airborne Collision Avoidance System for small rotorcraft) was designed by the Johns Hopkins University Applied Physics Laboratory and MIT Lincoln Laboratory under sponsorship of the FAA to avoid imminent collisions. And NASA’s flight path management software, the Autonomous Operations Planner, reroutes aircraft around potential collisions that could occur three minutes or more in the future. Researchers were looking for any weaknesses in the software that hadn’t been uncovered in computer simulations.
“Is this type of algorithm feasible? What are the challenges with it? What are the things that the pilots like about it, don’t like about it?” says Nancy Baccheschi, chief engineer of NASA’s Integration of Automated Systems, the AAM research project that managed the flight tests.
A complex enchilada of software controlled the helicopters throughout the test flights. There were no software breakdowns that required pilot intervention, Baccheschi says.
Onboard the two helicopters, an automated flight software package from Sikorsky and DARPA enabled the researchers to order the helicopters to fly predetermined routes into the scripted encounters. This package — MATRIX software developed by Sikorsky working as an operating system for ALIAS software by DARPA — wasn’t part of the evaluation, but it allowed the helicopters to operate either autonomously, with full human pilot control, or with a limited degree of autonomy.
Other software, the NASA-developed middleware, then acted like a music conductor, allowing either the ACAS Xr or the Autonomous Operations Planner software to command the MATRIX/ALIAS automated flight software to, for example, make a specific evasive maneuver. Meanwhile, the MATRIX/ALIAS software also served as a “safety wrapper” that prevented other software or human operators from commanding one of the helicopters to try a maneuver that would fly it into the ocean or was physically impossible, Baccheschi says, meaning outside its flight envelope.
For each of the scripted encounters, the Black Hawk played the role of the “intruder,” while the S-76 was the “ownship” employing the software packages to automatically avoid a collision or help its human pilot do so.
In the ownship cockpit, the NASA test pilot had two tablets: one showing a map of the air traffic in the area, another showing the two helicopters and their scripted flight paths. During the tests for the flight path management software, the test pilot was shown a choice of three buttons for three flight path options to avoid the intruder. In the ACAS tests, the test pilot saw the yellow arc and then its shift to the red arc and green wedge.For crib notes, the pilot carried a stack of index cards bound by a ring. Each card had bullet points describing the basics for one of the scripted encounters. The bullets included whether the flight maneuver would be automated or performed by the human pilot and when it would be executed. Each index card also conveyed the starting position for the encounter, a cartoon map of the encounter with arrows showing flight paths, and the “knock it off” headings: the direction each helicopter was to fly in if the encounter was aborted.
For the ACAS encounters, the researchers tested potential collision flight paths at different speeds from all directions, such as a head-on clash, the intruder overtaking from behind, the ownship overtaking the intruder, a 90-degree intercept angle from the side, the intruder climbing up from below or the intruder descending from above.
Testing such a range of scenarios helps researchers uncover situations that the software can’t handle, Howe says. Some of the scenarios set up the ownship to fly at slightly above hover speed and for the intruder to simulate an unknown aircraft interfering with a landing or takeoff.
“What we were trying to uncover is anything unique to eVTOLs in those spaces,” he says, referring to the electric vertical takeoff and landing air taxis now in testing by various companies.
For the flight path management software scenarios, some tests also created virtual intruders that appeared on the tablet in addition to the Black Hawk for the software to calculate a safe passage around.
The idea behind this software is to avoid another aircraft that might suddenly pop up and that wasn’t previously broadcasting its flight path and then chart a path around that aircraft (or multiple aircraft) in a way that still gets everyone to their destinations on time, Baccheschi says. If each aircraft were to know the intended flight path of the other aircraft, then the flight path management software could help pack more aircraft into a tighter airspace because less margin of error would be required to avoid the other traffic.
At one point during the testing, researchers in a Sikorsky trailer 50 kilometers away in Bridgeport, Connecticut, noticed that some of the choreography was off. They were watching the same live maps that the pilots had on their tablets, and the triangles representing the helicopters would drift off their planned trajectories. The team would have to abort the encounter and move on to the next one.
Between flights, software engineers traced the problem to a message formatting error. Their solution: Reformat the data messages but find the “sweet spot” that wouldn’t bog down the data links between the ground station and helicopters.
“You need to be very smart about what things you say have to be guaranteed delivery,” Baccheschi says. “You can bog down your system if you call everything a must-go-through message.”
In addition to testing the software, another goal was to help pilots of the coming air taxis navigate crowded airspace. So the researchers needed to learn how the NASA test pilots interacted with varying levels of autonomy and what elements contributed to their workload during the scripted scenarios.
But the flights presented a problem for the human factors researchers from NASA and Lockheed Martin, which owns Sikorsky: In a flight simulator, researchers can simply stop a test in the middle of a scenario, or pause it immediately after a simulated encounter, to ask the human pilot or operator what they are thinking. That would be impossible during live flights. Even between scripted encounters during a flight, time in the air was too valuable for the pilots to spend more than a few seconds answering questions. So the researchers turned to biometrics.
Lockheed provided a combination heart-respiration monitor, as well as a device that fit in the NASA test pilots’ helmets to measure the volume of blood in the front of their brains. Faster heart and respiration rates and more blood in the brain signaled increased cognitive and physical workload for the pilots. Eye tracking glasses provided the researchers with video of the pilot’s general field of view and dots showing where the eyes were focused. Tiny cameras in the glasses that tracked the focal points also recorded pupil dilation — more dilation meant greater workload. After the flights concluded, the researchers asked the pilots about what they were thinking and feeling during specific encounters, aided by notes the pilots took during flights.
The researchers are now combing through the time-stamped biometric data and focal points — where and how long the pilot focused on particular screens versus looking out the window, for example —to figure out when and why the pilots’ workloads spiked during specific maneuvers, says Kevin Monk, NASA human factors engineer.
Howe says he quickly got used to wearing the biometric devices, but he initially had some misgivings about what the glasses might reveal about his behavior. “They’re going to know when I’m stressed and maybe know when I am, I don’t know, confused. Or they can even see if I didn’t understand what was happening and maybe looking around, searching for clues.”
The biometric data and pilot interviews will also show which display features are essential in certain situations and which are clutter. As the levels of automation get higher and higher with AAM, especially when pilots move from cockpits to ground stations to remotely monitor multiple aircraft, this flight test data can help create a baseline for determining pilot/operator workload and tasks to automate, Monk says. Future flight automation might also change displays during high-stress situations or when a pilot/operator shows physiological signs of stress, filtering out nonessential information and making the display graphics extremely clear and bright.
The post-flight interviews should also provide some insights for the software engineers into pilot thinking, Howe says: Why a pilot might decide to take a longer turn from the yellow arc, for example, “and maybe that’s obvious to pilots but not obvious to researchers or engineers.” Plus, the research could help determine whether warnings in the software are well-timed and whether a pilot is given enough time to react in specific situations.
“There’s always that trade-off between are we just annoying you with too much information that is distracting from what you really need to be concerned with? Or are we notifying you too late and you don’t have enough time to adequately create a nice, safe separation distance in a timely manner?” Howe says. “All of those questions are really at the core of that research.”
