Sandia telemetry group’s software improves how data is gathered, analyzed

A sign near the offices of Sandia National Laboratories’ Telemetry and Software Systems department says it all: Any data. Anywhere. Anytime.

That’s possible nowadays largely because of software the group developed beginning in 2005. Sandia copyrighted the Telemetry Analysis and Visualization Suite, first as a version for Sandia that expanded to other national labs and lab-connected contracts, and just last year, as a commercial version.

The telemetry group created the tool, TAVS for short, “because we were getting too much data to keep up with,” says Gary Ashcraft (2662), who has worked in telemetry at Sandia for more than 20 years. “Its original purpose was to keep my ground station data processing team sane.”

"We were getting too much data to keep up with."

Sandia does test flights for mock nuclear weapons systems launched by missiles or carried on submarines and airplanes. The tests produce terabytes of telemetry — radio signals transmitted to a ground station that show how weapon systems are performing.

The analysis and visualization system saves data during a test. Immediately afterward, the telemetry group loads raw archive information into the system, allowing engineers to analyze it on a desktop computer.

Handles massive amounts of data

The suite has three parts: a core engine that enables real-time display of data and 3-D visualization; post-test analysis; and a virtual ground station that makes processing data easier and faster. The group created the software to support nuclear weapons stockpile surveillance, later built in the real-time visualization, and lastly developed the virtual ground station.

TAVS can handle massive amounts of data. Multiple operators can look at different displays, quickly plot data, and filter it to pinpoint specific information. “You’re ready to analyze the test as soon as it’s over,” Gary says. “It’s definitely an engineering tool.”

Better data handling has been a goal for decades. Gary opens a 1955 report with a section of recommendations — third on the list: improve data reduction.

Sandia first displayed real-time data during a test about 2007, when its software was too new to run as the primary data acquisition system. Gary says the tool performed flawlessly and the team presented a real-time, 3-D visualization of the payload, a concept he and colleagues Mark Platzbecker (2626) and Dave Sandison (5200) developed in 1999.

But instant visualization languished because there was no vision for how to use it. Gary recalled that after its inaugural use, “People said, ‘That’s really cool, and I really don’t need that.’”

Then came a 2008 test, when Sandia Labs’ data acquisition system was the primary system. Suddenly, the 3-D visualization showed an anomaly, and “since our software was young, I was sure it was our software,” Gary says. “Then I looked up and saw the cameras that were on board and they were showing the same anomaly.”

The colonel’s eyes kind of lit up and he said, "Did it really do that?"

Two hours after mission’s end, everyone assembled to view the data and figure out what went wrong. Gary says missileers showed the colonel in charge the “cool squiggly lines of data” that form the backbone of information for engineers and analysts. When it was time for Sandia Labs’ payload presentation, “we showed some squiggly lines and said, ‘We also did a 3-D visualization of it during real time and here it is.’ The colonel’s eyes kind of lit up and he said, ‘Did it really do that?’ and we’re like, ‘Yeah, it really did.’ He thanked us. He said, ‘I have a much better understanding of what happened now.’”

Software now makes many jobs easier

The software began spreading through Sandia, expanding “from a grassroots ‘we need to make our jobs easier’ to making a lot of people’s jobs easier,” Gary says. “Anyone at Sandia can download the software and with a little bit of training can start using it.” The TAVS site is http://tavs.sandia.gov/.

When Sandia began planning a B61-12 flight test at its Tonopah Test Range in Nevada for summer 2015, the telemetry group, supported by integrated stockpile evaluation senior manager Jay Vinson (2250), provided ground station equipment hardware and its software to Tonopah and helped range personnel lay out stations and connections, Gary says. Tonopah now has a modern ground station and the ability to send data to Sandia/New Mexico in real time.

The final part of the suite, the virtual ground station, does digital signal processing. Sandia started using it for flight test data in 2006.

“Its original purpose was to keep my ground station data processing team sane.”

Data is recorded on hard drives, which go to the telemetry group. A flight test might consist of 10 to 15 drives totaling 3 to 10 terabytes of information. Traditionally, analysts plug the hard drive into their equipment, playing it back through a receiver and then to machines that extract the data. Playing it back takes the same length of time as acquiring it, so if a recording is 30 minutes long, it takes 30 minutes for playback. That’s a lot of time to process, for example, 10 half-hour recordings one by one.

The virtual ground station reduces equipment and steps by taking data directly off the hard drive and processing it on a computer. “There’s no reason to play it back through equipment,” Gary says. “With a multicore processor I can run 10 at a time and I don’t have to watch it. I don’t have to do anything. I can go home and let it run overnight and come back the next day and my data’s ready.

“But the real advantage is that the signal analyst can adjust the signal processing algorithms on a signal-by-signal basis and increase the quality and quantity of data extracted. Sometimes the most important data from a test occurs in a noisy environment, and the virtual ground station provides the analyst complete control of signal manipulation to extract all of the information possible. 

“The work is still being accomplished. It’s just a different way, better quality, and less expensive.”

Current TAVS team members are Michael Rimbert, Michael Bridges, and Ray Prior (all 2662), Leisa King and Ron Dulaney (both 2669), Andres Jaramillo (2666), and Adam Peters (5333).