The Greenland and Antarctic ice sheets will make a dominant contribution to 21st century sea-level rise if current climate trends continue. However, predicting the expected loss of ice sheet mass is difficult due to the complexity of modeling ice sheet behavior.

To better understand this loss, a team of Sandia researchers has been improving the reliability and efficiency of computational models that describe ice sheet behavior and dynamics. The team includes Irina Demeshko, Mike Eldred, John Jakeman, Mauro Perego, Andy Salinger, Irina Tezaur, and Ray Tuminaro.

This research is part of a five-year project called Predicting Ice Sheet and Climate Evolution at Extreme Scales (PISCEES), funded by DOE’s Scientific Discovery through Advanced Computing (SciDAC) program. PISCEES is a multi-lab, multi-university endeavor that includes researchers from Sandia, Los Alamos, Lawrence Berkeley, and Oak Ridge national laboratories; the Massachusetts Institute of Technology; Florida State University; the University of Bristol; the University of Texas Austin; the University of South Carolina; and New York University.

Sandia’s biggest contribution to PISCEES has been an analysis tool: a land-ice solver called Albany/FELIX (Finite Elements for Land Ice eXperiments). The tool is based on equations that simulate ice flow over the Greenland and Antarctic ice sheets and is being coupled to Earth models through the Accelerated Climate for Energy (ACME) project.

“One of the goals of PISCEES is to create a land-ice solver that is scalable, fast, and robust on continental scales,” says computational scientist Irina Tezaur, a lead developer of Albany/FELIX.  Not only did the new solver need to be reliable and efficient, but it was critical that the team develop a solver capable of running on new and emerging computers, and equipped with advanced analysis capabilities for model calibration and uncertainty quantification.

“The data we get from climate scientists are usually measurements from the top surface of the ice,” she says. “To initialize an ice sheet simulation, we need information about what is happening inside and at the bottom of the ice. Determining interior and bedrock ice properties is what we call model calibration, and requires the solution of an inverse problem. A lot of our work has been in developing and implementing optimization algorithms that are able to solve these inverse problems robustly and efficiently.”

She stresses that the success of PISCEES is due in large part to strong collaborations among glaciologists, climate modelers, computational scientists, and mathematicians.

“Glaciologists and climate scientist collaborators on PISCEES provide us with data sets to go into our model, while computational scientists come up with the right algorithms to use and implement them efficiently,” she says.

Improving computational modeling

The PISCEES project began in 2012 because there was no robust land-ice model as a building block for earth system models that calculate sea-level rise predictions to support DOE’s climate missions.

“In just three years of work, we have created a next-generation land-ice model that is verified, scalable, and robust and is portable to new and emerging architecture machines,” she says. “These models are equipped with advanced analysis capabilities.”

While code performance is critical to the success of Albany/FELIX, equally important are verification and validation, two procedures for evaluating a model and its code. Verification is aimed at ascertaining that it is bug-free. In contrast, validation is aimed at checking that the physical process described by a model is consistent with what is seen in the real world.

Sandia has done a thorough verification of the Albany/FELIX solver using the method of manufactured solutions, code-to-code comparisons on canonical land-ice benchmarks, and by performing convergence studies on realistic Greenland and Antarctica landscapes with real data.

Irina Tezaur explains that, in general, validation is much harder to do than verification. The degree to which ice sheet models have been validated by observations is fairly limited, due in part to the limited duration of the satellite observation era and the long adjustment time scales of ice sheets. The PISCEES team has checked that its solver predicts ice sheet quantities such as surface velocities, and surface mass balance, and that these quantities are consistent with past and recent observations.

“We are in the process of doing a validation study for the Greenland Ice Sheet for the period 1991-2012,” she says. “The ice sheet model output will be compared to ice surface elevation and ice sheet mass change observations from ICESat, the Ice, Cloud and land Elevation Satellite.”

Early results show promise for assessing the performance of different model configurations.  A verification and validation test suite, known as the Land Ice Validation and Verification Kit, is being developed by PISCEES collaborators at Oak Ridge National Laboratory, and goes hand-in-hand with efforts performed at Sandia.

In recent months, climate scientists Steve Price and Matt Hoffman at Los Alamos have begun to use Albany/FELIX for production runs on Edison and Cori, two of the newest Cray supercomputers at the National Energy Research Scientific Computing Center. During the next two years, calculations provided by the team will be used to estimate expected sea level rise during the 21st century.

Albany/FELIX: Agile components and novel approaches

The Albany/FELIX solver was written using the so-called “Agile Components” software development strategy, an approach advocated by PISCEES Sandia principal investigator Andy Salinger, in which new application codes are written using mature modular libraries. New solvers created using this approach are “born” scalable, fast, robust, and capable of advanced analysis since they are based on a collection of world-class algorithmic capabilities developed and tested by domain experts.

The components comprising Albany/FELIX are the Trilinos libraries, a collection of open-source packages developed at Sandia. In addition to recommending and executing the “Agile Components” code development strategy, the Sandia PISCEES team has developed approaches for improving the robustness of the nonlinear solver using homotopy continuation. The Albany/FELIX code has demonstrated scalability up to one billion unknowns and tens of thousands of cores thanks to parallel scalable iterative linear solvers and newly developed preconditioning methods by Ray Tuminaro. Adjoint-based deterministic inversion algorithms and software developed and implemented by Mauro Perego have enabled rigorous model calibration.  In collaboration with experts from the QUEST SciDAC institute, Mike Eldred and John Jakeman, a framework for forward and inverse uncertainty quantification (UQ) has been developed. Finally, Albany/FELIX has been made portable to new architecture machines thanks in large part to the efforts of computer scientist Irina Demeshko.