New cognitive science research aids nuclear treaty verification inspectors

New cognitive science research from Sandia shows that while maps can help you identify landmarks while being escorted, using one also limits situational awareness and knowledge of surroundings not highlighted on the map.

This finding is one of several coming from a three-year project that paired cognitive scientists and international nuclear safeguards experts to conduct human performance tests and develop recommendations for International Atomic Energy Agency safeguards inspectors. The goal is to help inspectors be more accurate, efficient and situationally aware while on the job, which ultimately helps ensure nuclear nonproliferation.

The tests focused on visual inspection, wayfinding and knowledge transfer tasks that are important for inspectors. While cognitive scientists have studied these areas in detail before, the existing research did not address the unique needs of international safeguards inspectors.

“Inspectors are on the road at least half of the time, sometimes going from country to country to save costs,” said Zoe Gastelum, nuclear safeguards researcher and project lead. “They are jet-lagged, they are working in industrial, hazardous environments and they are often limited by the types of technology they can bring in with them.”

IAEA inspectors visit nuclear energy facilities to verify that they are not misused, and nuclear material is not diverted from peaceful uses. While at a facility, they must complete specific tasks and record information while also being aware of their surroundings and analyzing discrepancies. They typically are communicating in a non-native language with people who are also speaking in a non-native language. Plus, inspection teams often change so they have to work from the notes and observations of others to compare changes in a facility over time, Zoe said.

“All these complicating factors make it a really challenging job and place a heavy cognitive load on the inspectors,” she said. “With this project, we saw that changing the way we provide information to folks who are working in the field can help them be more effective and efficient.”

The research team performed a survey to determine the biggest needs for the nuclear safeguards community, and then reviewed it to identify the gaps in cognitive science literature.

“A lot of cognitive science work has been very lab-based and theory-heavy about mechanisms and processes in the mind and brain,” said Mallory Stites, a cognitive scientist and research team member. “Sometimes there’s a gulf between what you do in the lab and what you need for applied science. Our goal was to do lab-based work grounded in cognitive theory that could be applied in the field.”

The team designed a series of human performance tests based on inspectors’ activities. For each task, the team provided relevant information in several ways to the participants and observed how the way the information was presented changed performance.

Balancing accuracy, speed while searching lists and items

“While there has been a lot of research on visual search in cognitive science, the specific things the inspectors do in the field, which are usually comparing one list to another list or to physical objects, hadn’t been looked at much,” said Laura Matzen, a cognitive scientist and project team member. “We wanted to fill that gap by looking at different ways of organizing an inspector’s list to make the job easier.”

The team studied visual search and list formatting. One test had participants compare one list to another, much like an inspector would compare a facility’s current inventory with the inventory list provided by the national safeguards regulator to note what had gone in and out. They had each participant group try a different technique for organizing and comparing their lists, such as using color coding, numerical ordering or reorganizing the current list to match the order of the previous list as much as possible.

“We found that color coding really did help,” Laura said. “Anything we could do to the inspector’s list to constrain where they looked on the facility’s list helped speed up the process because they could zero in on the relevant column or section.”

Laura said the results also showed that reorganizing the lists to better match sped up the comparison process even more, but participants started missing subtle errors, such as number transpositions, and their accuracy decreased.

Choose your emphasis: finding landmarks or observing details

The team set up experiments at a Sandia facility that was previously used for work with nuclear materials but is no longer active.

“The facility had a lot of the same visual characteristics that inspectors see,” Mallory said. “It was big, kind of loud, pretty warm and an easy space to get turned around in.”

The goal, she said, was to see how or whether maps could help inspectors learn the layout of a facility and remain aware of their general surroundings, particularly while they are being led on a guided route.

“When you’re not in control of your own navigation, like being a passenger in a car, it’s really hard to keep track of where you’re going,” she said.

Some participants were given a map to study before going into the facility for a guided tour, some were allowed to bring a map with them on the tour and some didn’t get any time with a map before or during the tour.

“We found that if inspectors can get a map of the facility in advance, that’s definitely better for enhancing their ability to know their route, recognize landmarks and form a birds-eye view of the facility,” Mallory said. “However, the inspection team should make a choice based on their objectives about whether or not to carry the map with them inside the facility. If their main goal is learning the layout, the answer is yes. If their main goal is situational awareness, we found they are better off not bringing in and consulting a map while inside.”

Taking notes for knowledge transfer

The research team set up a series of experiments to determine how notetaking and pictures can help or hinder inspectors from transferring knowledge to other inspectors, often years later without any in-person conversation or interaction.

The team created four poster boards full of images of nondescript industrial-looking parts and gadgets.

In different studies, the participants had 12 minutes with each board and were asked to take notes by hand, take pictures with a camera, or take handwritten notes and pictures with a camera.

In the first set of experiments, the same person came back either two days or six months later and were given their notes and a new set of similar but different boards and then were asked to determine what had changed from the first set of boards and images.

In the second set of experiments, the notes from the participants who had seen the first set of boards were given to new participants who were only shown the second set of boards. They, too, were asked to determine what had changed, simulating a second inspection team going in with notes from a previous team.

Laura said the items on the board might have changed places, or their texture, material or orientation might have changed, but participants were not told in advance what kind of changes to expect. Participants also were free to take notes with their pencil and paper in any way they wished.

“We did some analysis on the premise of, if you only have pen and paper, how do you make the most of it?” Laura said. “We found that people who had something in writing and something visual in their notes were better able to detect changes. Try to organize your information spatially, list all the important features, write descriptions and draw whenever possible, to help the next person to understand.”

Laura said participants who had cameras were unsurprisingly more accurate on detecting changes in the objects, but also were slower to analyze the objects and would often lose track of where the object they had photographed had been placed on the board. Inspectors using a camera were understandably more confident in their responses, but that confidence included when they were incorrect. For inspectors who are able to bring in a camera, the team recommends enhancing this by jotting down a few notes about the photos you are taking — orientation, part of the building you were in or other helpful situational information.

Getting the word out

The research team has presented papers on visual inspection and notetaking for knowledge transfer at the International Conference on Human-Computer Interaction, on human performance testing at the IAEA Safeguards Symposium and on list-comparison activity research and simulating safeguards information environments for human performance testing at the Institute of Nuclear Materials Management Annual Meeting. They also have published in The European Safeguards Research & Development Association Bulletin, presented at the ESARDA annual symposium and published in Cognitive Research: Principles and Implications.

The team has started a related two-year project to characterize the impact of errors from machine-learning and deep-learning algorithms on human cognitive performance, with the goal of providing evidence-based recommendations for integrating AI with human decision making in analytical systems.

This project was funded by Sandia’s Laboratory Directed Research and Development program.