‘The sky is falling’: Let’s reduce the false positives in conjunction data messages
By Moriba Jah|January 2024
Believe it or not, some satellite operators elect not to move their spacecraft when faced with an alarm about a potential collision. Why take what seems like a great risk at a time when our determination to prevent catastrophic collisions should be nonnegotiable? One reason is that nothing bad has happened so far from ignoring an alarm. Statistics tell us that luck will run out, but so far these nonmovers have yet to collide with anything.
The root of the problem is that conjunction data messages are ridden with false positives.
While I’m about to offer a suggestion for how to reduce the number of false alarms, I should underscore that even now conjunction data messages are a necessary space safety product. These alerts, generated when the probability of collision reaches or exceeds one in 1000, serve as a potential aegis against space debris-generating events. The caveat is the prevalence of false positives, or “type I errors,” within these warnings.
Before we get to my suggestion, let’s look at classical hypothesis testing, since conjunction analysis requires starting with hypotheses. You have a null hypothesis and an alternative hypothesis. The null is your default belief, and you only have two choices to make: You either reject the null or you fail to reject it. There are three criteria for failing to reject the null: 1) You have no evidence; 2) you have insufficient evidence; 3) you have overwhelming evidence that the null is probably true. For orbital collisions, the null hypothesis is, “All is good, and my satellite is safe.”
Based on evidence and models, one needs to select the conditions by which the null would be rejected. Sometimes people use what is known in probability and statistics as p-values — a statistic that quantifies what you’re concerned about testing — to indicate the likelihood of obtaining evidence that would show that the null hypothesis is correct. The p-value is based on a test statistic, which for us is the probability of collision. A small p-value implies a small chance that the null, “all is good,” can be proven correct. Since our null says that the satellite is safe, our small p-value indicates that we’re unsure about that safety. Declaring a small chance of being able to prove something as correct is another way of saying there is uncertainty. In this case, that means there is a greater chance of incorrectly warning of a collision — a false positive. Put another way, which p-value to choose tends to depend on the rate of false positives we’re willing to tolerate. The caveat is that typically, when you have a process that generates fewer false positives or type I errors, you get an increase of false negatives or type II errors, and of course that’s not acceptable either, since a false negative in this case means you failed to notify spacecraft operators of a real collision.
Let’s first look at the consequences of false positives. Warnings that weren’t in fact needed are not benign. They can be economically and environmentally devastating. Satellites are directed to alter their trajectories or perform evasive maneuvers, incurring unnecessary costs and potentially disrupting critical services. Ironically, maneuvering needlessly creates new risks of collision: The operators could miscalculate their deconfliction maneuvers or unwittingly steer into the path of debris, given our currently incomplete knowledge about the tracks of these objects. This situation demands a reevaluation of our current space traffic management strategies.
We must select a p-value that helps us control the false positive or type I errors, but we also want zero type II errors or false negatives. That is to say, we want zero collisions to go unpredicted. But zeroing out type II errors tends to result in more type I errors and thus means we get lots of collision warnings that are not physically possible or meaningful. This then desensitizes the spacecraft operator community and makes the warnings unactionable and easy to ignore, an undesired outcome given that mixed among them are what could turn out to be legitimate warnings.
A solution would need to focus on processes, methods, and data that reduce type I errors without increasing type II errors. This might be achieved by refining the algorithms used to generate conjunction data messages, incorporating advanced machine learning techniques, and successful use of artificial intelligence. By fine-tuning these algorithms, we may be able to reduce the occurrence of false positives, ensuring that satellite operators receive warnings only when the probability of a collision is genuine. Simultaneously, efforts should be directed toward minimizing type II errors, as the consequences of missing a real collision would be catastrophic.
Collaboration among space agencies, satellite operators and experts in space debris mitigation is essential. A unified effort to improve the accuracy of conjunction data messages will enhance the overall reliability of space traffic management. Research and development initiatives should prioritize the exploration of innovative technologies that can discern between benign close approaches and genuine collision risks.
No country can solve this alone. International cooperation must be fostered to standardize procedures and share data. The global nature of space activities demands a unified approach to space traffic management, where information is seamlessly exchanged and effective practices are universally adopted.
As we stand at the forefront of a new era of space exploration, it is crucial to address the challenges that threaten to hinder our progress. By acknowledging the prevalence of false positives in conjunction data messages, we pave the way for a more effective and sustainable approach to space traffic management. Only striking the right balance between error types can we ensure the safety and longevity of our orbital environment.
Related Topics
Space safety
