Saving GOES-17

April 2019

Cooling pipes were not working inside the camera aboard NOAA’s geosynchronous GOES-17 weather satellite, degrading the performance of its crucial infrared channels. It sounded like game over, but not to NOAA and Harris Corp., the company that built the camera. John Van Naarden of Harris and Dan Lindsey of NOAA explain how engineers learned to operate GOES-17 and its camera without those critical cooling pipes.

Two months after NOAA launched its newest geostationary weather satellite last year, the NOAA operations team encountered a significant issue with the thermal system of its primary payload, the Advanced Baseline Imager that Harris built at its factory in Indiana. The problem discovered during post-launch testing was serious enough that it threatened to end the GOES-17 mission before it even started. For part of the day and in key channels, the ABI could not gather the infrared images that help tell weather models and forecasters when severe weather is brewing. Nearly 10 months and many sleepless nights later, we are proud to note that on Feb. 12, NOAA declared GOES-17 operational to take over the GOES-West slot over the equator. Looking back, this period was one of the more difficult and stressful times of our careers, but also one of the most rewarding.

Our journey to improve GOES-17’s ABI performance was successful. Despite a thermal system operating at only about 5 percent of its capacity, we expect ABI to provide 97 percent of its intended data, far better than initial estimates when we discovered the problem a year ago. Our success would not have been possible without the inherent flexibility in the modern designs of both the ABI and the spacecraft — and a path to optimization that required real-time learning from nearly 36,000 kilometers away.

We had the added pressure of following in the footsteps of GOES-17’s sister satellite, GOES-16. With GOES-16 already occupying the GOES-East slot, GOES-17 would complete a two-decade journey toward modernization of NOAA’s operational fleet. With the ABI designs already launched on Japan’s Himawari-8 and Himawari-9 satellites, GOES-17 would also link advanced capabilities from east to west — covering 85 percent of the Earth. It would be a game changer in the weather community for improving hurricane and typhoon forecasts across the Atlantic and Pacific oceans. Expectations for GOES-17 were high.

At stake

A single Geostationary Operational Environmental Satellite takes years to build, including its primary instrument, a single 315-kilogram ABI camera. Each ABI instrument provides 90 percent of the data products that enable a GOES satellite to help save lives and protect property. GOES-17 is the second of NOAA’s four planned next-generation, advanced geostationary weather satellites. One GOES satellite is always designated as GOES-East off the U.S. East Coast near 75 degrees west longitude, and another as GOES-West located off the U.S. West Coast near 137 degrees west longitude. These sentinels in the sky watch developing weather and other environmental conditions 24 hours a day, 365 days a year, and must deliver data in near real-time.

One of the capabilities that distinguishes ABI is the ability for forecasters to request on-demand observations of evolving severe weather. Going from photons to forecasting products in less than 30 seconds, GOES provides unprecedented near real-time situational awareness about hurricanes, tornadoes and wildfires.

ABI distinguishes itself from legacy geostationary imagers by its ability to collect three times the spectral data with 16 channels (colors) of visible-through-longwave infrared imagery
needed for weather forecasting, volcanic ash monitoring, low cloud/fog monitoring and air-quality monitoring.

The realization

The ABI on GOES-17 spent the first two months post-launch outgassing, ensuring that none of the contaminants that are inevitably carried from Earth aboard a satellite were deposited on critical optical surfaces. On April 12, 2018, the mission operations team determined it was safe to open our optical port cover and begin taking visible images. The first visible images were incredible, and we looked forward to first light for the infrared channels.

Fifteen days later, we sent commands to tell the ABI’s thermal system to cool down the detectors, nearly 39,000 elements, each smaller than a human hair, etched into silicon and mercury cadmium telluride wafers. The ABI’s primary thermal components include a cryocooler (plus a backup) that must keep the ABI’s detectors at the very cold temperatures required for infrared observations. The visible and near-infrared detectors operate at minus 68 Celsius (170 Kelvin), and the infrared detectors operate at minus 178 C (60 K).

A network of constant conductance, heat pipes circulate ammonia that absorbs heat from the cryocooler, electronics and any components warmed by the sun. The ammonia transports the heat to a central depository called a thermal bus that intersects with what are called loop heat pipes. Propylene inside these pipes warms up and transports the heat to the radiator, which sheds the heat to space. The cooled propylene circulates back to the thermal bus to continue the cooling cycle.

The first signs of trouble appeared a few hours after the loop heat pipes were commanded to start. Parts of the ABI that should have become cold remained hot. Heat that should have been transferred to the radiator and shed to space did not move. Near midnight, when the sun shined on the far side of the Earth, it also shined directly into our optical aperture and drove temperatures inside the ABI to unsafe levels. We knew within hours we had to do something right away to save the ABI from overheating and potentially suffering permanent damage. The cryocooler’s electronics generate most of the internal heat produced by the ABI, so we quickly reduced its power, helping the ABI make it through the night. Saving ABI from excessive heat meant no cold detectors, which also meant no infrared imagery — impacting 13 of the imager’s 16 channels around spacecraft local midnight. We could make visible images but lost the full spectrum of capabilities required by forecasters and GOES users. We had prevented damage to the ABI, but the work was just beginning.

Investigation and backup plans

On-orbit troubleshooting requires a rigorous plan and disciplined action to safely determine root cause. From review of telemetry, we were able to deduce that the propylene fluid flow through the loop heat pipes was not distributing heat across the radiator. Several immediate attempts to restart the loops the following day were also unsuccessful, and again near midnight we turned off the cryocooler.

The investigation and troubleshooting continued over the next few weeks, and after a month of unsuccessful attempts, we realized we may never recover loop heat pipe performance. NOAA, NASA and Harris decided to establish two parallel teams. The first team, augmented by independent reviewers from NASA and loop heat pipe experts from across the country, continued the root cause investigation and attempted to restore GOES-17 to full operations. The second team assumed the loop heat pipes were unrecoverable and investigated optimizing the ABI’s performance with minimal radiator capacity. As time moved on without a solution, NOAA established two more teams — one that considered how to use NOAA’s existing satellites to augment GOES-17 and another to investigate supplementing algorithms to compensate for missing data from the ABI. This case study is about the work of the second team, the one focused on optimization.

The optimization effort started in early June at Harris’ facility in Fort Wayne, Indiana, with a team that included Harris, Lockheed Martin, NASA and NOAA. The team had one advantage working in its favor on the path to improvement: the ABI’s flexible design. When Harris designed the ABI, we wanted to ensure its electronics and software design were reconfigurable by command. We attacked the problem from two directions:

1. Limit the heat buildup and in this way increase infrared channel availability throughout the night.

2. Improve the imagery at warmer detector temperatures.

Limiting the heat buildup

The beta angle of the sun plays a critical role in determining how hot our infrared detectors will get on any given day. Beta angle is the angle between the orbital plane and the sun vector. For geostationary satellites, this angle varies by plus or minus 23.5 degrees throughout the year and determines how hot our infrared detectors will peak on any given day. The Lockheed Martin spacecraft built for this geostationary satellite series has the capability to yaw flip (rotate 180 degrees about the nadir vector so the north side of the spacecraft now faces south). While this type of maneuver was not planned for nominal operations, it certainly provided a big boost to the ABI optimization. Our thermal models indicated an inverted orientation for six months of the year would reduce sun penetration and restrict solar heat from reaching deep into ABI, reducing the overall heat buildup, thus allowing us to keep the cryocooler running longer and keep the detectors colder overnight.

The day we performed the yaw flip maneuver exhibited an incredible 10C reduction in peak detector temperature over the previous day. The first major change implemented to GOES-17 operations was a twice-a-year yaw flip near each equinox to keep the sun in its summer geometry — shining down from above. This minimizes how deep the sun can penetrate into the aperture and keeps heat away from critical components. The yaw flip maneuvers guarantee the ABI maintains this advantageous sun geometry year-round. “It’s always summer on ABI,” was the rallying cry.

Since the cryocooler directly controls the infrared focal plane temperature, maximizing its capability was vital to optimization. We employed several strategies related to cryocooler operation that were key to our success. We designed the ABI with fully redundant cryocooler units, with only one in use in normal operations. In our on-orbit testing, we found that operating both units simultaneously was a much more efficient configuration for GOES-17. With each unit extracting only part of the heat from the detectors, the cryocoolers were more efficient and thus generated less heat to be transported.

Another change we made to cryocooler operation was increasing the maximum allowable temperature at the reject surface of the cryocooler. This surface is the interface to the thermal system and where it transfers its internally generated heat to be “rejected” to space by the radiator. On a normally functioning spacecraft, like GOES-East, the highest operating temperature for the cryocooler reject surface would be 27 C. On GOES-17 we decided to allow the ABI to run up to 57 C each day, which is the maximum we had demonstrated on a life test unit. This increase was necessary because the thermal system was not allowing heat to flow out as it normally would, meaning the ABI routinely reached its maximum safe reject temperature, something that would never happen in normal operations. The new, higher temperature became the limiting factor of our daily operations. We were forced to reduce cryocooler power once that temperature was reached. Increasing the temperature limit allowed us to maximize cryocooler operation throughout the day. Not only did we increase this limit, we modified our entire cryocooler operating procedure around it. A new version of the ABI onboard control software was developed and uploaded from the ground to autonomously control operations near this limit so we could gain every possible minute of imagery for forecasters.

The last significant cryocooler strategy was increasing the nominal operating temperature of our infrared detectors. Driving the cryocoolers to maintain the nominal infrared detector temperature of minus 213 C requires significant power. The colder we tried to keep the detectors during the day, the hotter they became at night, because the thermal system could handle only a certain amount of total heat in a day, and the more heat produced by the cryocooler during the day, the less heat we could accept from the sun at night. We needed to identify an optimum balance between daytime performance and additional hours of infrared imagery. We evaluated 24 hours of images at various set points for the detector temperature ranging from minus 159 C up to minus 148 C. In the end, we settled at minus 157 C (81K). Per assessments by NOAA scientists, this compromise still provided high-quality imagery during the day for all 16 channels while not drawing significant amounts of power to overwhelm the thermal system at night.

Taking the heat

Despite all the success we had keeping temperatures cooler each day, it was not enough. The detectors were not designed to work at minus 157 C, and they certainly would not perform well at our expected annual peak temperature of minus 129 C. To improve performance at these elevated temperatures, we relied on the flexible configurability designed into the ABI detector operation. The two parameters we found to have the most impact were detector bias, which controls the flow of a detector’s photocurrent, and its sensitivity, called gain.

Detector bias voltage is critical to acceptable detector operation. We use it to drive the photocurrent from individual detectors in the right direction so that it can be properly read by the electronics. Through our on-orbit testing, we found that the elevated detector temperatures caused photocurrent to flow into adjacent detector elements rather than heading toward the readout electronics. The result was data coming from one part of Earth was measured by detectors staring at other parts of Earth, which caused unacceptable blurring of images at warmer detector temperatures. After reoptimizing the bias voltage value, we observed significant improvement in many of the channels and regained several hours of daily imagery in the 9.61, 12.30 and 13.30 micrometer channels (the most impacted). The 13.30 micrometer channel measures carbon dioxide in the atmosphere, which is critical to several of the cloud property retrievals produced by the ABI. Gaining hours of operation for that channel is a big win.

Detector gain is the other detector parameter we used to great success. Dark current, which is the signal produced by detectors even when no light shines on them, increases significantly with operating temperature. As detector temperatures began to rise each night, the dark current could become larger than the photo current produced by the signal from Earth. The increased dark current saturated our analog-to-digital converters, reducing the number of hours of useful imagery. We dialed back the detector gain settings overnight. Reducing the gain lowers the current value read by the digitizer and allows us to stay in its useable current range for a longer time. We currently operate the GOES-17 ABI with two distinct gain sets. The first set maximizes performance at our new nominal temperature of minus 192 C during the day. We command a second set of reduced gains shortly before night to minimize the impact of increased dark current once temperatures increase. These gain changes occur automatically each day and maximize the hours of useful operation.

The future is bright

Our experience over the past year has been nothing we ever could have imagined. Through the efforts of numerous people, GOES-17 will be the game-changing weather satellite it was promised to be. We found a way, without use of a radiator, to optimize the ABI performance and reclaim hours of lost operation each day.

NASA and NOAA continue to review options for the future of the third and fourth spacecraft in the series, GOES-T and GOES-U, but the future of GOES-17 looks bright. With the operational approach described in this case study, we expect ABI to provide 97 percent of its intended data over the course of a year, which is an astounding number considering the circumstances. That some of the infrared channels remain unavailable for a few hours each night during times of the year is regrettable, but we are in a far better position overall with GOES-17’s optimized configuration and certainly better off from legacy capability.

Our success would not have been possible without the inherent flexibility in the modern designs of both the ABI and the spacecraft. We used that flexibility to identify a multifaceted operations approach that put GOES-17 on the path to success and ultimately to its spot as the operational GOES-West satellite. All from 36,000 kilometers away.

John Van Naarden is chief engineer for Harris Corp.’s Advanced Baseline Imager program in Fort Wayne, Indiana.

Dan Lindsey is NOAA’s senior scientific adviser for the GOES-R satellite series. He works for NOAA’s National Environmental Satellite, Data, and Information Service in Fort Collins, Colorado.

GOES-S with the Advanced Baseline Imager (wrapped in silver thermal blankets) was prepared for launch at Astrotech in Florida. Once in orbit last March, the satellite was renamed GOES-17. Credit: Harris Corp.

The 16 channels on GOES-17’s Advanced Baseline Imager show the northeastern Pacific Ocean and western U.S. The infrared bands are those with blue and/or yellow colors. Credit: NOAA/CIMSS

The Advanced Baseline Imager. Credit: NOAA

Engineers install an Advanced Baseline Imager on the GOES-R satellite. Credit: NOAA