See below: Sustainability features come together

Sandia’s 725E data center addition for high-performance computing at the Labs’ Albuquerque campus has earned the LEED v4 Gold BD+C: Data Center Certification.

Leadership in Energy and Environmental Design, or LEED, was developed by the U.S. Green Builders Council in 1993 to measure and define green buildings, provide a roadmap for developing sustainable buildings and establish a baseline for reducing environmental impact.

LEED certification is a lengthy process with stringent guidelines. In pursuit of the certification, a building is evaluated on a point system, earning points for various green building strategies to achieve one of four LEED rating levels: Certified, Silver, Gold or Platinum. Prior to earning the certification, a building must operate and function for a period of time to make sure all applicable green design and build goals are met. The building also must demonstrate continued operational sustainability to retain the certification.

The 725E data center was completed in October 2018, and LEED certification took 14 months to achieve. This is the first certification earned under Sandia’s LEED v4 Campus effort.

Sandia has four corporate data centers. The 725E data center is home to the Labs’ Astra and Vanguard high-performance computing systems. David J. Martinez, engineering program/project lead, and other team members in data center services and facilities management and engineering worked together to design, build and operate the facility as a LEED-certified building.

The certification “is a great milestone for the Labs,” David said. “This is something that I had a vision for 20-plus years ago, and we have been working on it for some time.”

David’s passion for and experience with green LEED building increased with the work he did on the campus of DOE’s National Renewable Energy Laboratory in Golden, Colorado, where he helped design a LEED Platinum-certified HPC data center. He spent several years collaborating with NREL to help with the design and assist in overseeing the construction of their data center.

“NREL is still the world’s most energy-efficient center,” he said. “However, our certification now puts Sandia in the top 20. Eventually, we would like to place our mark as one of the top five energy-efficient data centers in the world.”

Getting to Gold

HPC systems consume substantial amounts of energy to perform the large-scale computations required by these supercomputers. A biproduct of that energy consumption is a substantial amount of heat, requiring stringent cooling regimens to keep the computers running.

Although a typical building is designed to meet heating and cooling needs for the comfort of its human occupants, HPC data centers must provide massive cooling power for their banks of servers. This usually results in high water and energy usage. Employing energy efficiency and conservation techniques in the building of data centers is crucial for operation and reducing ongoing costs. Numerous green building strategies and new, innovative systems were implemented in the 725E data center to get it to the LEED Gold level.

“From the beginning, our goal was to design/build to get the Gold certification. Approximately 25-30% into the design, we sent out to bid for a contractor/engineer to keep us focused on the certification goal requirements and where we could get points for certification,” David said.

Verdacity, the Albuquerque sustainability firm who won the bid, is no stranger to LEED design, with more than 75 LEED certification projects, including Sandia’s LEED v4 Campus masterplan. They helped the Sandia team find and implement many green building features for the data center.

“We designed based on what we needed and wanted for energy efficiencies,” David said. “Verdacity guided us along in the design to find and earn LEED certification points.”

Learn more about Sandia’s Vanguard program, the Astra supercomputer and the facility:

Sustainability features come together

Warm water cooling

One way Sandia’s LEED v4 Gold certified 725E data center addition for high-performance computing conserves energy is by using warm water to cool HPC systems. The computers generate a sizable amount of heat, and warm water is more efficient in cooling the computers than cool water.

The complex water-cooling system includes numerous tubes and pipes flowing through the HPC computer racks. Water swirls through flexible tubes that are visible through the rows of glass tiles spanning two racks. At the end of the racks sits a unique pumping system called the Cooling Distribution Unit, which controls the water flow and rates.

Cooling with outdoor air, thermosyphon technology

Some of the computers housed within the data center do not use water cooling exclusively. To handle this, the building was designed with the Airside Economizer, an outdoor air-cooling system.

“Since New Mexico is a dry climate, we can leverage 75-80% of our outdoor air without having to cool it,” said David J. Martinez, engineering program/project lead.

The Airside Economizer brings fresh air in and cycles it to the floor. Heat generated by the computers rises, and some of it escapes outside via exhaust, while a portion remains near the top of the 25-foot ceiling, allowing the facility to maintain an indoor temperature of around 78 degrees. On the data center roof is a cell deck, similar to a high-powered swamp cooler, that is used only about 17% of the year. During other times, it cycles the cool air in.

“As long as it’s not above 78 degrees, we don’t need to enhance the cooling at all,” David said. “Above 96 degrees, then we use mechanical cooling. It’s never that hot at night, though, so we go back to outdoor air. We can rely on outdoor air to help cool the building and components. We don’t want to throttle back the computer’s performance. We’re able to find the balance of operating our structure and the computers as well.”

The 725E data center also is equipped with a thermosyphon cooling system, which uses passive heat transfer to make the building more efficient. The system saved more than a half-million gallons of water during its first six months of operation.

More design features provide energy, water savings

Many other green building elements helped the 725E data center earn its LEED Gold certification, including highly efficient windows that offer natural lighting. “You rarely see a data center with natural lighting,” David said. “We have windows facing north. During the day, you don’t need lights. Sensors turn lights on when needed.”

Other factors that helped the center earn points toward the certification include a 250-kilowatt solar unit that contributes some of the energy used in the building, as well as water-conserving elements like native vegetation and low-flow toilets.