Publications

Abstract not provided.

Abstract not provided.

This study describes a cyber security research & development (R&D) gap analysis and research plan to address cyber security for industrial control system (ICS) supporting critical energy systems (CES). The Sandia National Laboratories (SNL) team addressed a long-term perspective for the R&D planning and gap analysis. Investment will posture CES for sustained and resilient energy operations well into the future. Acknowledgements The authors would like to acknowledge the funding and technical support from the Department of Energy Office of Electricity Delivery & Energy Reliability for the development of this report. The authors are very appreciative of the key contributions by other SNL personnel in supporting the analysis, particularly from Jennifer Depoy, Abraham Ellis, Derek Hart, Jordan Henry, John Mulder, and Jennifer Trasti. The authors would also like to thank the following government and non-government organiza- tions for their invaluable input to this study: Government Massachusetts Institute of Technology Lincoln Laboratory Construction Engineering Research Laboratory (CERL) Idaho National Laboratory Marine Corps Air Ground Combat Center, Twentynine Palms, California National Renewable Energy Laboratory National Institute of Standards and Technology Pacific Northwest National Laboratory U.S. Army Corps of Engineers U.S. Army Cyber Command U.S. Navy Installations Command Non-Government Customized Energy Solutions Electric Power Research Institute Enchanted Rock ICETEC Integrated Energy Solutions NEC Energy Solutions OpenADR Alliance PJM POWER Engineers Schweitzer Engineering Laboratory Southwest Research Institute Typhoon HIL, Inc. Executive Summary This study describes a long-term cyber security R&D plan to address ICS cyber security for CES. Long-term goals for ICS were assumed to be those that would require significant action and R&D to achieve, as opposed to being addressable by applying existing technology and best practices. Long-term R&D would roughly fall into the window of 5-10 years out. Investing in the identified R&D will posture CES for sustained resilient energy operations well into the future. The gaps were identified using a conventional gap analysis process. The current state of cyber security R&D was surveyed and summarized. Then, the desired future state of ICS cyber security was characterized, in terms of required capabilities for a secure and resilient ICS. Afterward, gaps were identified by comparing the current state of cyber security to the desired end-state. Finally, the gaps were prioritized and paired (where important) with the appropriate communities (industry, vendors, academia, etc.) suitable to address them. The baseline survey of the existing R&D focused on efforts in government, academia, feder- ally funded research and development centers (FFRDCs), and industry (including vendors). One primary source was existing DOE, Department of Homeland Security (DHS), and Department of Defense (DoD) programs, including Cybersecurity for Energy Delivery Systems (CEDS) and Defense Advanced Research Projects Agency (DARPA). Crucial documents from the National In- stitute of Standards and Technology (NIST) were also surveyed. On the academic side, the group included work from the Institute for Information Security & Privacy (IISP) and Trustworthy Cyber Infrastructure for the Power Grid (TCIPG) research consortiums. Numerous other smaller efforts were cataloged as well. Overall, the results show significant attention on the cyber security issues faced by ICS, but with a definite tendency toward near-term solutions, and less defined long-term goals, particularly in terms of needed R&D. The surveyed concepts and goals were used to develop the desired state for long-term ICS cyber security. These were complemented by concepts and frameworks previously used for ICS cyber security. The overall result was the development of a matrix of needed technical capabilities for secure and resilient ICS in the long term. Eighteen cyber security concepts (referred to as "topics" for gap analysis) were identified and sorted according to their positions in the security lifecycle (secure design, reinforced implementation, operation and deployment, or cross-cutting capabilities) and security category (protect, detect, react, or recover). For each topic, a description was provided, as well as other discussion, including a comparison to existing work. The comparisons formed the basis for the gap analysis. Some security topics, although an essential part of a desired secure ICS state in the future, have significant R&D resources alieady working to realize the goal. Others, however, are only partially addressed. Besides the severity of the R&D gap, an important consideration is that perfect security is unattainable; therefore, strong security engineering must be complemented with additional security monitoring. The final rankings for long-term R&D, including specific opportunities and challenges, along with suggestions about which group or groups should be targeted for funding opportunities, are in Chapter of the report. Some of the key results include: 1. Trusted monitors, which act as out-of-band security sentinels, and security analytics, which fuse weak indicators to detect security anomalies, have very high priority for R&D. As men- tioned previously, no system can be completely trusted (or, given the potential ramifications, even reasonably trusted); therefore, monitoring is essential. 2. Virtualization is a key capability for many aspects of ICS cyber security; potential applica- tions include training environments, pre-deployment change testing, red/blue engagement, evaluating tactics-techniques-procedures (TTPs), and others. Virtualization capability would be greatly enhanced with better support for ICS field devices (like relays, programmable logic controllers, etc.) and automated model generation from design or operational system information. 3. Field devices have unique cyber security issues, and are critical to cyber risk given their application: straddling the cyber/physical domains Addressing these issues in an organized fashion (including their virtualization) is a priority R&D gap. This is also an example where industry (particularly vendors) must complement other R&D organizations.

Programmable logic controllers (PLCs) and other field devices are important components of many weapons platforms, including vehicles, ships, radar systems, etc. Many have significant cyber vulnerabilities that lead to unacceptable risk. Furthermore, common procedures used during Oper- ational Test and Evaluation (OT&E) may unexpectedly lead to unsafe or severe impacts for the field devices or the underlying physical process. This document describes an assessment methodology that addresses vulnerabilities, mitigations, and safe OT&E. Acknowledgements The authors would like to acknowledge the funding and technical support from the Office of the Director, Operational Test and Evaluation (DOT&E) for the development of this paper. Also, there were key contributions by other Sandia National Laboratories (SNL) personnel supporting the analysis, particularly from Mitch Martin, Tricia Schulz, Chris Davis, and Nick Pattengale, and from Pacific Northwest National Laboratory (PNNL), especially Chris Bonebrake, Jim Brown, and Katy Bragg. Executive Summary Industrial control system (ICS) field devices like PLCs play a critical role in the safe and reliable operation of Department of Defense (DOD) platforms and weapon systems operations. Unfor- tunately, these sorts of devices are often rife with cyber security vulnerabilities that can lead to significant risks for mission performance, or even unsafe conditions during routine OT&E. The cyber security issues faced by ICS differ from typical information technology (IT), and this re- quires a different and more specific approach to assess, test, and mitigate ICS vulnerabilities. In a typical IT system, data confidentiality and integrity are the primary concerns. In an ICS, mission operations, safety, public health, and avoiding equipment damage are the primary con- cerns. ICS devices directly control time critical processes and have little margin for delay. Outages or interruptions (even something as simple as a reboot) might not be acceptable, and if unplanned can result in significant risk to mission. Unlike IT system updates or patches, which can be done using automated server-based tools and are widely applicable, ICS updates are specific to the equipment vendor. OT&E on ICS field devices (on deployed platforms, or in high value test rigs) is often a neces- sary requirement, but this causes significant concern within the DOD ICS community. The concern is that implementing routine cyber security measures and testing on active ICS components and systems may damage the ICS or even underlying physical systems. Of particular concern are ICS field devices, which encompasses the specialized hardware that covers the boundary between the cyber and physical domains. Examples of field devices include PLCs, electric power relays, remote terminal units (RTUs), and other embedded devices. According to an Office of the Secretary of Defense (OSD) memorandum regarding "Proce- dures for Operational Test and Evaluation of Cybersecurity in Acquisition Programs," operational test agencies (OTAs) will "include cyber threats... with the same rigor as other threats" [1]. The purpose of cyber security operational test and evaluation is to evaluate the ability of a unit equipped with a system to support assigned missions in the expected environment. The "system" in this case is considered to encompass hardware, software, user operators, etc. This memorandum also spec- ifies the procedures to be used for testing oversight systems. The purpose of this docuemnt is to introduce a Field Device Assessment Methodology (FDAM) that parallels (with some differences due to the focus on ICS hardware and not the entire system) the procedures suggested in the mem- orandum. The FDAM approach is not intended to cover the entire oversight system as referenced in the memorandum; rather, it explains the procedures necessary to evaluate the ICS hardware devices. This focused approach on the hardware subset of the system is warranted because ICS field devices face very different issues than IT systems, and the risks associated with ICS cyber vulnerabilities can be significant. The goals of the FDAM are to research and rank field device vulnerabilities to be tested, sum- marize associated mitigations, and determine cyber test concerns by summarizing potential OT&E test damage/safety issues. The FDAM primarily supports the cooperative assessment stage of OT&E, although the results can also support adversarial assessments. This document provides guidance on tools and procedures that have been developed by SNL that are used to implement the FDAM approach, including an assessment framework, quantitative risk calculation, and ranked access/procedure pairs (APPs). The FDAM process itself is presented in Chapters through -- from initial research and discovery, to standalone lab testing, through to compiling the final report. It should be noted that because cyber security testing is inherently complex and detail-oriented, those performing the tests will generally have a wealth of knowledge and experience that is dif- ficult to fully document or simplify into a step by step process. In every testing situation, the background of the testers may influence how they choose to implement the process, and in which order. Although this document is presented as a logical process, it is not necessary to follow every step in the document as laid out. For example, a tester that is intimately familiar with ICS systems might choose to do the literature review and vulnerability scoring in conjunction with lab testing. Or, if project resources are limited, the best choice might be to do only a literature review and risk scoring without standalone lab testing or even a device teardown. The FDAM is intended to support OTAs, cyber protection teams (CPTs), and other organiza- tions within DOD that support OT&E on weapons platforms and systems, but it can also be applied to ICS used within DOD installations and other bases, particularly for infrastructure support. The DOT&E FDAM is applicable for mission platforms, which are heavily reliant on ICS, including naval shipboard systems (electrical plant management, machinery control, aircraft launch/recovery, radar, fire control, and others), advanced ground vehicle management, and aircraft/avionics. The FDAM also supports a range of DOD assessment requirements [2, 3] and the approach is suitable to varying classification levels, as application details and close-held government information can be included when desirable (and useful).

Distributed Energy Resources (DER) are being added to the nation's electric grid, and as penetration of these resources increases, they have the potential to displace or offset large-scale, capital-intensive, centralized generation. Integration of DER into operation of the traditional electric grid requires automated operational control and communication of DER elements, from system measurement to control hardware and software, in conjunction with a utility's existing automated and human-directed control of other portions of the system. Implementation of DER technologies suggests a number of gaps from both a security and a policy perspective. This page intentionally left blank.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Microgrids are a focus of localized energy production that support resiliency, security, local con- trol, and increased access to renewable resources (among other potential benefits). The Smart Power Infrastructure Demonstration for Energy Reliability and Security (SPIDERS) Joint Capa- bility Technology Demonstration (JCTD) program between the Department of Defense (DOD), Department of Energy (DOE), and Department of Homeland Security (DHS) resulted in the pre- liminary design and deployment of three microgrids at military installations. This paper is focused on the analysis process and supporting software used to determine optimal designs for energy surety microgrids (ESMs) in the SPIDERS project. There are two key pieces of software, an ex- isting software application developed by Sandia National Laboratories (SNL) called Technology Management Optimization (TMO) and a new simulation developed for SPIDERS called the per- formance reliability model (PRM). TMO is a decision support tool that performs multi-objective optimization over a mixed discrete/continuous search space for which the performance measures are unrestricted in form. The PRM is able to statistically quantify the performance and reliability of a microgrid operating in islanded mode (disconnected from any utility power source). Together, these two software applications were used as part of the ESM process to generate the preliminary designs presented by SNL-led DOE team to the DOD. Acknowledgements Sandia National Laboratories and the SPIDERS technical team would like to acknowledge the following for help in the project: * Mike Hightower, who has been the key driving force for Energy Surety Microgrids * Juan Torres and Abbas Akhil, who developed the concept of microgrids for military instal- lations * Merrill Smith, U.S. Department of Energy SPIDERS Program Manager * Ross Roley and Rich Trundy from U.S. Pacific Command * Bill Waugaman and Bill Beary from U.S. Northern Command * Tarek Abdallah, Melanie Johnson, and Harold Sanborn of the U.S. Army Corps of Engineers Construction Engineering Research Laboratory * Colleagues from Sandia National Laboratories (SNL) for their reviews, suggestions, and participation in the work.

This document describes a microgrid cyber security reference architecture leveraging defense- in-depth techniques that are executed by first describing actor communication using data exchange attributes, then segmenting the microgrid control system network into enclaves, and finally grouping enclaves into functional domains. To illustrate the design approach, two notional microgrid control implementations are presented. Both include a discussion on types of communication occurring on that network, data exchange attributes for the actors, and examples of segmentation via enclaves and functional domains. The second example includes results from Red Team analysis and quantitative scoring according to a novel system that derives naturally from the implementation of the cyber security architecture. Acknowledgements Sandia National Laboratories and the SPIDERS technical team would like to acknowledge the following for help in the project: * Mike Hightower, who has been the key driving force for Energy Surety Microgrids * Juan Torres and Abbas Akhil, who developed the concept of microgrids for military installations * Merrill Smith, U.S. Department of Energy SPIDERS Program Manager * Ross Roley and Rich Trundy from U.S. Pacific Command * Bill Waugaman and Bill Beary from U.S. Northern Command * Tarek Abdallah, Melanie Johnson, and Harold Sanborn of the U.S. Army Corps of Engineers Construction Engineering Research Laboratory * Colleagues from Sandia National Laboratories (SNL), Oak Ridge National Laboratory (ORNL), Idaho National Laboratory (INL), Massachusetts Institute of Technology Lincoln Laboratory (MIT-LL), United States Pacific Command (USPACOM), and the Joint Information Operations Warfare Center (JIOWC) for their reviews, suggestions, and participation in the work.

Many critical loads rely on simple backup generation to provide electricity in the event of a power outage. An Energy Surety Microgrid TM can protect against outages caused by single generator failures to improve reliability. An ESM will also provide a host of other benefits, including integration of renewable energy, fuel optimization, and maximizing the value of energy storage. The ESM concept includes a categorization for microgrid value proposi- tions, and quantifies how the investment can be justified during either grid-connected or utility outage conditions. In contrast with many approaches, the ESM approach explic- itly sets requirements based on unlikely extreme conditions, including the need to protect against determined cyber adversaries. During the United States (US) Department of Defense (DOD)/Department of Energy (DOE) Smart Power Infrastructure Demonstration for Energy Reliability and Security (SPIDERS) effort, the ESM methodology was successfully used to develop the preliminary designs, which direct supported the contracting, construction, and testing for three military bases. Acknowledgements Sandia National Laboratories and the SPIDERS technical team would like to acknowledge the following for help in the project: * Mike Hightower, who has been the key driving force for Energy Surety Microgrids * Juan Torres and Abbas Akhil, who developed the concept of microgrids for military installations * Merrill Smith, U.S. Department of Energy SPIDERS Program Manager * Ross Roley and Rich Trundy from U.S. Pacific Command * Bill Waugaman and Bill Beary from U.S. Northern Command * Melanie Johnson and Harold Sanborn of the U.S. Army Corps of Engineers Construc- tion Engineering Research Laboratory * Experts from the National Renewable Energy Laboratory, Idaho National Laboratory, Oak Ridge National Laboratory, and Pacific Northwest National Laboratory

Abstract not provided.

Microgrids consist of a combination of generation resources and load, forming an electrically sustainable entity. Although the feeder configuration, including location of circuit breakers or switches, and selection of protective devices can change from one microgrid to another, some characteristics like size of microgrid and behavior of sources feeding a fault remains similar. Due to the non-uniformity of configuration, no definite choices of protection schemes have emerged. This paper analyzes the performance of three most commonly used principles of protection - overcurrent, distance, and differential - on a microgrid topology based on three actual microgrid designs. Importance and implementation of safe islanding and resynchronization are also discussed. Though this research was done primarily for microgrids at United States military bases, the analysis and conclusions may be applied to microgrids in general.

Abstract not provided.

Abstract not provided.

Abstract not provided.

In 2012, Hurricane Sandy devastated much of the U.S. northeast coastal areas. Among those hardest hit was the small community of Hoboken, New Jersey, located on the banks of the Hudson River across from Manhattan. This report describes a city-wide electrical infrastructure design that uses microgrids and other infrastructure to ensure the city retains functionality should such an event occur in the future. The designs ensure that up to 55 critical buildings will retain power during blackout or flooded conditions and include analysis for microgrid architectures, performance parameters, system control, renewable energy integration, and financial opportunities (while grid connected). The results presented here are not binding and are subject to change based on input from the Hoboken stakeholders, the integrator selected to manage and implement the microgrid, or other subject matter experts during the detailed (final) phase of the design effort.

Abstract not provided.

Abstract not provided.

This white paper focuses on "advanced microgrids," but sections do, out of necessity, reference today's commercially available systems and installations in order to clearly distinguish the differences and advances. Advanced microgrids have been identified as being a necessary part of the modern electrical grid through a two DOE microgrid workshops, the National Institute of Standards and Technology, Smart Grid Interoperability Panel and other related sources. With their grid-interconnectivity advantages, advanced microgrids will improve system energy efficiency and reliability and provide enabling technologies for grid-independence to end-user sites. One popular definition that has been evolved and is used in multiple references is that a microgrid is a group of interconnected loads and distributed-energy resources within clearly defined electrical boundaries that acts as a single controllable entity with respect to the grid. A microgrid can connect and disconnect from the grid to enable it to operate in both grid-connected or island-mode. Further, an advanced microgrid can then be loosely defined as a dynamic microgrid.

Abstract not provided.

Abstract not provided.

Abstract not provided.