NASA concludes initial phase of video archive project for air taxis

By Paul Brinkmann|December 16, 2024

Helicopter flights were a step toward training autonomous flight software to navigate real-world scenarios

NASA aerospace engineer Nelson Brown watched a computer display at Kennedy Space Center in Florida as multiple video streams showed views from a pod slung beneath a helicopter flying over the center’s landscape.

Brown is the lead researcher in NASA’s AIRVUE initiative, short for Airborne Instrumentation for Real-world Video of Urban Environments. The pod gathered video, laser range finding and other flight data. This kind of information from flights beyond KSC could one day populate a dataset of actual flight scenarios that developers of electric air taxis could tap to train their aircraft to fly autonomously with onboard software. Analysts have often said that avoiding the need to train and pay numerous pilots could be essential for air taxi services to become widespread and profitable.

For more about advanced air mobility, receive the True Mobility newsletter in your inbox.

The November flight monitored by Brown was one of the last in an initial test phase of AIRVUE that concluded that month. One of KSC’s conventionally piloted Airbus H135s that are normally flown for security operations was fitted with the pod containing off-the-shelf cameras; a light detection and ranging sensor, or lidar; and an inertial navigation system unit consisting of a GPS receiver and an inertial measurement sensor to determine the aircraft’s attitude. With this combination, the videos and any images extracted from them were matched with the aircraft’s location, altitude, airspeed and attitude.

KSC was adequate for the testing, Brown says, even though it doesn’t look much like the urban places where air taxis are expected to fly. There are some features, including a helipad and launch towers off in the distance. Most significant, though, were the birds. Encounters with them were especially valuable, Brown explains, because they are obstacles to be avoided. These encounters were annotated manually in the dataset so that a person or a computer could later learn how the aircraft maneuvered in response.

To elaborate on the need for AIRVUE, Brown offers a comparison to the auto industry: The self-driving car company Waymo, owned by Google’s parent Alphabet, has collected vast amounts of video from the autonomous taxis it operates on city streets in Los Angeles, San Francisco and Phoenix. The video and still images are overlaid with driving data such as location and speed, all of which is annotated by humans to identify specific threats or obstacles. Other companies and researchers can download Waymo’s open-source dataset and train their own software to recognize hazards on the road, Brown points out.

While Waymo is permitted to operate its vehicles on city streets, autonomous air taxis probably won’t be able to learn by doing in the same way in crowded, three-dimensional airspace without traffic signals. Hence, AIRVUE: “I really want to get a large airborne dataset curated by NASA and available for public use,” Brown says.

It would be open source so that researchers and developers of air taxis can access it and conduct additional research or train their aircraft to fly without a pilot on board.

Among the first orders of business when the initiative began in 2022 was to decide which cameras to include in the pod. NASA conducted standard “risk reduction” flights that year at KSC to evaluate commercially available cameras and their responses to vibrations and other factors. Brown and others then went back to their home base at Armstrong Flight Research Center in Southern California to create the pod and test it on drones there before returning to KSC to start the first pod flights.

AIRVUE is just a first step in collecting data for airborne computer vision, and Brown doesn’t expect that NASA will be able to collect enough data to ensure accurate computer vision for flights in widespread urban areas. He says that the agency expects others, including in the private sector, will eventually contribute to the dataset.

Ultimately, the goal is to have software capable of recognizing potential hazards such as powerlines, rogue aircraft that are not where they are supposed to be, weather balloons, large birds, hailstones and other weather phenomena, or even ground hazards such as people walking across landing pads, Brown says.

That means others beyond KSC must gather and share video too. “NASA would offer the pod to helicopter operators that are already flying in cities and partner with them to kind of ‘crowdsource’ video” and other types of data, Brown says.

Also, air taxi companies could be given the design of the pod and the data specifications so they can build their own pods and share fly data to expand the depth of the airborne dataset. When air taxi companies start flying commercially, NASA could also partner with them, he says.

Right now, AIRVUE is funded through 2027, and there’s much to do between now and then.
“We want to really slim down the bulkiness of the pod and get it approved by FAA for several types of aircraft,” Brown says.

Those who monitor progress toward autonomous flight have long called for development of an airborne dataset, says Ella Atkins, a professor of aerospace engineering at Virginia Tech and editor-in-chief of AIAA’s Journal of Aerospace Information Systems.

“It is in everyone’s best interest to share data so the [machine learning] training can be as comprehensive as possible. Kudos to NASA for taking this initiative,” Atkins tells me.

She says the AIRVUE dataset could also help train pilots if fed into simulators that mimic flight conditions. She notes that autopilot functions on today’s airliners or corporate jets aren’t relevant to electric air taxis that will spend most of their flight time at lower altitudes, closer to ground-based obstacles and birds, and perhaps far from tightly controlled airspace around airports.

Most importantly, she says, is that NASA and FAA encourage the recording and reporting of “corner cases,” unusual scenarios where something goes wrong and aircraft are forced to maneuver suddenly to avoid calamity.

“It is super important that NASA work with FAA to ensure no organization that contributes data can be found liable for breaking rules,” Atkins said. “We want this AIRVUE dataset to have the problematic data. If not, it will be good for training but will fail to inform in the event of pilot error, sensor failure, bad weather, rogue aircraft, et cetera.”