Feature The Voyager probes have entered a new phase of operations. As recent events have shown, keeping the venerable spacecraft running is challenging as the end of their mission nears.
As with much of the Voyager team nowadays, Kareem Badaruddin, a 30-year veteran of NASA’s Jet Propulsion Laboratory (JPL), divides his time between the twin Voyager spacecraft and other flight projects. He describes himself as a supervisor of chief engineers but leaped at the chance to fill the role on the Voyager project.
Suzanne Dodd, JPL Director for the Interplanetary Network Directorate, is the Project Manager for the Voyager Interstellar Mission.
“She knew that the project was sort of entering a new phase where there was likely to be a lot of technical problems,” Badaruddin told The Register. “And so chief engineers, that’s what they do. They solve problems for different flight projects.”
Dodd needed that support for Voyager. Badaruddin would typically have found someone from his group, “but I was just so excited about Voyager, I said, you know, look no further, right? I’m the person for the job. I’m your engineer. You know, please pick me.”
Badaruddin has spent the last two years on the Voyager project. After decades of relatively routine operation, following plans laid out earlier in the mission when the team was much larger, the Voyager spacecraft have begun presenting more technical challenges to overcome as the vehicles age and power dwindles.
The latest problem occurred when engineers warmed up part of the spacecraft, hoping that some degraded circuits might be “healed” by an annealing process. “There’s these junction field effect transistors in a particular circuit that have become degraded through radiation,” Badaruddin explained. “We don’t have much protection from radiation in an interstellar medium because we’re outside the heliosphere where a lot of that stuff gets blocked.
“So we’ve got this degradation in these electronic parts, and it’s been proven that they can heal themselves if you get them warm enough, long enough.
“And so we knew we had some power margin, and we were hopeful that we had enough power margin to operate this heater … and as it turned out, we didn’t.
“It was a risk we took to try and ameliorate a problem that we have with our electronics. So now the problem is still there, and we realize that we can’t solve it this way. And so we’re going to have to come up with another creative solution.”
The problem was that more power was demanded than the system could supply. A voltage regulator might have smoothed things out, but the Voyagers no longer have that luxury. Instead, engineers took a calculated risk and ran afoul of the then-innovative software onboard the spacecraft.
The under-voltage routine of the fault protection software shuts down loads on the power supply, but since the Voyager team has turned off anything that is not essential, there isn’t much left.
“So the under-voltage response doesn’t do much except turn off the X-band transmitter and turn on the S-band transmitter,” explained Badaruddin. “And that’s because the S-band transmitter uses less power, and so it’s like the last safety net to save you.”
And save the mission it did. While the S-band is great for operations near Earth, such as the Moon, it is almost useless at the distance of the Voyager spacecraft. However, by detecting the faint carrier signal of the S-band transmission, the team was able to pinpoint the problem to the act of turning on the heater, even without telemetry from the spacecraft.
The challenge for engineers isn’t just the time it takes to get a command to the Voyagers and receive a response (apparently not as frustrating as you might think – Badaruddin said: “This is the rhythm we work in; we’ve gotten accustomed to it … it used to be a very small time delay and it’s gradually gotten longer and longer over the years”) but also checking and rechecking every command that gets sent to the spacecraft.
With physical hardware long gone, the team has an array of simulators. “We have a very clear understanding of the hardware,” said Badaruddin. “We know exactly what the circuitry is, what the computers are, and where the software runs.”
And the software? It’s complicated.
There have been so many tweaks and changes over the years that working out the exact revision of every part of Voyager’s code is tricky. “It’s usually easier to just get a memory readout from the spacecraft to find out what’s on there,” said Badaruddin.
We’re sure there are more than a few engineers on Earth who are not entirely sure what their systems are running. The challenge for the Voyager team is that the spacecraft are nearing the half-century mark, as is the documentation.
“We have documents that were type-written in the 70s that describe the software, but there are revisions … and so building the simulators, we feel really good about the hardware … but we feel a little less good about understanding exactly what each instruction does.”
The latest bit of recoding occurred with the failure of one of Voyager 1’s integrated circuits, which manifested itself as meaningless data last year.
Badaruddin explained the process: “The basic problem was figuring out what was wrong with no information. We could see a carrier signal; we knew we were transmitting in the X-band … we knew we could command the spacecraft because we could tweak that signal slightly with commands. So we knew the spacecraft was listening to us, and we knew the spacecraft was pointing at Earth because otherwise, we wouldn’t get a signal at all.”
The engineers went further down the fault tree, and eventually managed to get a minimum program to the spacecraft to give a memory readout. That readout could be compared to one retrieved when the spacecraft was healthy. 256 words were corrupted, indicating a specific integrated circuit. Code was then written to relocate instructions around that failed area.
“The problem there is the code was very compact. There was no free space that we could take advantage of. So we had to sacrifice something.”
That something was one of Voyager 1’s higher data rate modes, used during planetary flybys.
The current challenge involves dealing with the probes’ thrusters. Silicon from bladders inside the fuel tanks has begun to leach into hydrazine propellant. Since silicon doesn’t ignite like hydrazine, a tiny amount gets deposited in the thrusters and slowly builds up in the thruster capillaries. Badaruddin uses the analogy of clogging arteries. Eventually, the blockage will prevent the spacecraft from firing its thrusters to point at Earth.
However, the pitch and yaw thrusters, each of which have three branches, are clogging at different rates. The current software works on the basis that branch 1, 2, or 3 will be used. But could it be operated in mixed mode, where branch 2 is used for the pitch thruster, but branch 3 is used for the yaw?
“So that’s a creative solution. It would be very complicated … this would be another software modification in interstellar space.”
Getting it right the first time is not just nice to have; it’s almost essential. By the time the results of a command come back from the Voyager spacecraft, it might be impossible to deal with the fallout of a failure.
The Voyager spacecraft are unlikely to survive another decade. The power will eventually dwindle to the point where operations will be impossible. High data rates (relatively speaking – Voyager’s high data rate is 1.4 kilobits per second) will only be supported by the current Deep Space Network (DSN) until 2027 or 2028. After that, some more creativity will be needed to operate Voyager 1’s digital tape recorder.
Badaruddin speculates that shutting off another heater (the Bay One heater) used for the computers would free up power for the recorder, according to the thermal model, but it’ll be a delicate balancing act. And, of course, the recent annealing attempt demonstrated the limitations of modeling and simulations on Earth.
Does Badaruddin have a favorite out of the two spacecraft?
“Well, Voyager 2 is the one that’s been flying the longest, and Voyager 1 is the one that’s furthest from Earth. So they both have a claim to fame.”
To use another analogy: “They’re essentially twins … they’re basically the same person, but they live different lives, and they have different medical problems and different experiences.”
Badaruddin hopes to stick with the mission until the final transmission from the spacecraft.
“I love Voyager. I love this work. I love what I’m doing. It’s so cool. It just feels like I’ve got the best job at JPL.” ®
