Sangoleye, Fisayo S.; Johnson, Jay; Chavez, Adrian R.; Tsiropoulou, Eirini E.; Marton, Nicholas
L.; Hentz, Charles R.; Yannarelli, Albert Y.
Microgrids require reliable communication systems for equipment control, power delivery optimization, and operational visibility. To maintain secure communications, Microgrid Operational Technology (OT) networks must be defensible and cyber-resilient. The communication network must be carefully architected with appropriate cyber-hardening technologies to provide security defenders the data, analytics, and response capabilities to quickly mitigate malicious and accidental cyberattacks. In this work, we outline several best practices and technologies that can support microgrid operations (e.g., intrusion detection and monitoring systems, response tools, etc.). Then we apply these recommendations to the New Jersey TRANSITGRID use case to demonstrate how they would be deployed in practice.
The electric grid has undergone rapid, revolutionary changes in recent years; from the addition of advanced smart technologies to the growing penetration of distributed energy resources (DERs) to increased interconnectivity and communications. However, these added communications, access interfaces, and third-party software to enable autonomous control schemes and interconnectivity also expand the attack surface of the grid. To address the gap of DER cybersecurity and secure the grid-edge to motivate a holistic, defense-in-depth approach, a proactive intrusion detection and mitigation system (PIDMS) device was developed to secure PV smart inverter communications. The PIDMS was developed as a distributed, flexible bump-in-the-wire (BITW) solution for protecting PV smart inverter communications. Both cyber (network traffic) and physical (power system measurements) are processed using network intrusion monitoring tools and custom machinelearning algorithms for deep packet analysis and cyber-physical event correlation. The PIDMS not only detects abnormal events but also deploys mitigations to limit or eliminate system impact; the PIDMS communicates with peer PIDMSs at different locations using the MQTT protocol for increased situational awareness and alerting. The details of the PIDMS methodology and prototype development are detailed in this report as well as the evaluation results within a cyber-physical emulation environment and subsequent industry feedback.
We present our research findings on the novel NDN protocol. In this work, we defined key attack scenarios for possible exploitation and detail software security testing procedures to evaluate the security of the NDN software. This work was done in the context of distributed energy resources (DER). The software security testing included an execution of unit tests and static code analyses to better understand the software rigor and the security that has been implemented. The results from the penetration testing are presented. Recommendations are discussed to provide additional defense for secure end-to-end NDN communications.
There are now over 2.5 million Distributed Energy Resource (DER) installations connected to the U.S. power system. These installations represent a major portion of American electricity critical infrastructure and a cyberattack on these assets in aggregate would significantly affect grid operations. Virtualized Operational Technology (OT) equipment has been shown to provide practitioners with situational awareness and better understanding of adversary tactics, techniques, and procedures (TTPs). Deploying synthetic DER devices as honeypots and canaries would open new avenues of operational defense, threat intelligence gathering, and empower DER owners and operators with new cyber-defense mechanisms against the growing intensity and sophistication of cyberattacks on OT systems. Well-designed DER canary field deployments would deceive adversaries and provide early-warning notifications of adversary presence and malicious activities on OT networks. In this report, we present progress to design a high-fidelity DER honeypot/canary prototype in a late-start Laboratory Directed Research and Development (LDRD) project.
The electric grid is rapidly being modernized with novel technologies, adaptive and automated grid-support functions, and added connectivity with internet-based communications and remote interfaces. These advancements render the grid increasingly 'smart' and cyber-physical, but also broaden the vulnerability landscape and potential for malicious, cascading disturbances. The grid must be properly defended with security mechanisms such as intrusion detection systems (IDSs), but these tools must account for power system behavior as well as network traffic to be effective. In this paper, we present a cyber-physical IDS, the proactive intrusion detection and mitigation system (PIDMS), that analyzes both cyber and physical data streams in parallel, detects intrusion, and deploys proactive response. We demonstrate the PIDMS with an exemplar case study exploring a packet replay attack scenario focused on photovoltaic inverter communications; the scenario is tested with an emulated, cyber-physical grid environment with hardware-in-the-loop inverters.
Recent trends in the growth of distributed energy resources (DER) in the electric grid and newfound malware frameworks that target internet of things (IoT) devices is driving an urgent need for more reliable and effective methods for intrusion detection and prevention. Cybersecurity intrusion detection systems (IDSs) are responsible for detecting threats by monitoring and analyzing network data, which can originate either from networking equipment or end-devices. Creating intrusion detection systems for PV/DER networks is a challenging undertaking because of the diversity of the attack types and intermittency and variability in the data. Distinguishing malicious events from other sources of anomalies or system faults is particularly difficult. New approaches are needed that not only sense anomalies in the power system but also determine causational factors for the detected events. In this report, a range of IDS approaches were summarized along with their pros and cons. Using the review of IDS approaches and subsequent gap analysis for application to DER systems, a preliminary hybrid IDS approach to protect PV/DER communications is formed in the conclusion of this report to inform ongoing and future research regarding the cybersecurity and resilience enhancement of DER systems.
Networked microgrids are clusters of geographically-close, islanded microgrids that can function as a single, aggregate island. This flexibility enables customer-level resilience and reliability improvements during extreme event outages and also reduces utility costs during normal grid operations. To achieve this cohesive operation, microgrid controllers and external connections (including advanced communication protocols, protocol translators, and/or internet connection) are needed. However, these advancements also increase the vulnerability landscape of networked microgrids, and significant consequences could arise during networked operation, increasing cascading impact. To address these issues, this report seeks to understand the unique components, functions, and communications within networked microgrids and what cybersecurity solutions can be implemented and what solutions need to be developed. A literature review of microgrid cybersecurity research is provided and a gap analysis of what is additionally needed for securing networked microgrids is performed. Relevant cyber hygiene and best practices to implement are provided, as well as ideas on how cybersecurity can be integrated into networked microgrid design. Lastly, future directions of networked microgrid cybersecurity R&D are provided to inform next steps.
The penetration of Internet-of-Things (IoT) devices in the electric grid is growing at a rapid pace; from smart meters at residential homes to distributed energy resource (DER) system technologies such as smart inverters, various devices are being integrated into the grid with added connectivity and communications. Furthermore, with these increased capabilities, automated grid-support functions, demand response, and advanced communication-assisted control schemes are being implemented to improve the operation of the grid. These advancements render our power systems increasingly cyber-physical. It is no longer sufficient to only focus on the physical interactions, especially when implementing cybersecurity mechanisms such as intrusion detection systems (IDSs) and mitigation schemes that need to access both cyber and physical data. This new landscape necessitates novel methods and technologies to successfully interact and understand the overall cyber-physical system. Specifically, this paper will investigate the need and definition of cyber-physical observability for the grid.
Reducing the risk of cyber-attacks that affect the confidentiality, integrity, and availability of distributed Photovoltaic (PV) inverters requires the implementation of an Intrusion Detection System (IDS) at the grid-edge. Often, IDSs use signature or behavior-based analytics to identify potentially harmful anomalies. In this work, the two approaches are deployed and tested on a small, single-board computer; the computer is setup to monitor and detect malevolent traffic in-between an aggregator and a single PV inverter. The Snort, signature-based, analysis tool detected three of the five attack scenarios. The behavior-based analysis, which used an Adaptive Resonance Theory Artificial Neural Network, successfully identified four out of the five attacks. Each of the approaches ran on the single-board computer and decreased the chances of an undetected breach in the PV inverters control system.
In recent years the use of security gateways (SG) located within the electrical grid distribution network has become pervasive. SGs in substations and renewable distributed energy resource aggregators (DERAs) protect power distribution control devices from cyber and cyber-physical attacks. When encrypted communications within a DER network is used, TCP/IP packet inspection is restricted to packet header behavioral analysis which in most cases only allows the SG to perform anomaly detection of blocks of time-series data (event windows). Packet header anomaly detection calculates the probability of the presence of a threat within an event window, but fails in such cases where the unreadable encrypted payload contains the attack content. The SG system log (syslog) is a time-series record of behavioral patterns of network users and processes accessing and transferring data through the SG network interfaces. Threatening behavioral pattern in the syslog are measurable using both anomaly detection and graph theory. In this paper it will be shown that it is possible to efficiently detect the presence of and classify a potential threat within an SG syslog using light-weight anomaly detection and graph theory.
Historically, control systems have primarily depended upon their isolation from the Internet and from traditional information technology (IT) networks as a means of maintaining secure operation in the face of potential remote attacks over computer networks. However, these networks are incrementally being upgraded and are becoming more interconnected with external networks so they can be effectively managed and configured remotely. Examples of control systems include the electrical power grid, smart grid networks, microgrid networks, oil and natural gas refineries, water pipelines, and nuclear power plants. Given that these systems are becoming increasingly connected, computer security is an essential requirement as compromises can result in consequences that translate into physical actions and significant economic impacts that threaten public health and safety. Moreover, because the potential consequences are so great and these systems are remotely accessible due to increased interconnectivity, they become attractive targets for adversaries to exploit via computer networks. Several examples of attacks on such systems that have received a significant amount of attention include the Stuxnet attack, the US-Canadian blackout of 2003, the Ukraine blackout in 2015, and attacks that target control system data itself. Improving the cybersecurity of electrical power grids is the focus of our research.
Energy resilience has emerged as a national security priority over the past fifteen years. Recent research efforts have aimed to develop metrics and analysis methods for energy resilience, but most of those efforts have focused on extreme weather and other natural hazards as the threat of interest. This paper introduces a novel set of resilience metrics and exemplifies how they can be applied to analyze resilience for growing concerns about cyber threats. The metrics are formally described with mathematical equations and demonstrated in a case study that evaluates the resilience benefits of a new moving target defense technology.
Proceedings - 17th IEEE International Conference on Trust, Security and Privacy in Computing and Communications and 12th IEEE International Conference on Big Data Science and Engineering, Trustcom/BigDataSE 2018
To ensure reliable and predictable service in the electrical grid it is important to gauge the level of trust present within critical components and substations. Although trust throughout a smart grid is temporal and dynamically varies according to measured states, it is possible to accurately formulate communications and service level strategies based on such trust measurements. Utilizing an effective set of machine learning and statistical methods, it is shown that establishment of trust levels between substations using behavioral pattern analysis is possible. It is also shown that the establishment of such trust can facilitate simple secure communications routing between substations.
Critical infrastructure systems continue to foster predictable communication patterns and static configurations over extended periods of time. The static nature of these systems eases the process of gathering reconnaissance information that can be used to design, develop, and launch attacks by adversaries. In this research effort, the early phases of an attack vector will be disrupted by randomizing application port numbers, IP addresses, and communication paths dynamically through the use of overlay networks within Industrial Control Systems (ICS). These protective measures convert static systems into "moving targets," adding an additional layer of defense. Additionally, we have developed a framework that automatically detects and defends against threats within these systems using an ensemble of machine learning algorithms that classify and categorize abnormal behavior. Our proof-of-concept has been demonstrated within a representative ICS environment. Performance metrics of our proof-of-concept have been captured with latency impacts of less than a millisecond, on average.
Conventional cyber defenses require continual maintenance: virus, firmware, and software updates; costly functional impact tests; and dedicated staff within a security operations center. The conventional defenses require access to external sources for the latest updates. The whitelisted system, however, is ideally a system that can sustain itself freed from external inputs. Cyber-Physical Systems (CPS), have the following unique traits: digital commands are physically observable and verifiable; possible combinations of commands are limited and finite. These CPS traits, combined with a trust anchor to secure an unclonable digital identity (i.e., digitally unclonable function [DUF] - Patent Application #15/183,454; CodeLock), offers an excellent opportunity to explore defenses built on whitelisting approach called 'Trustworthy Design Architecture (TDA).' There exist significant research challenges in defining what are the physically verifiable whitelists as well as the criteria for cyber-physical traits that can be used as the unclonable identity. One goal of the project is to identify a set of physical and/or digital characteristics that can uniquely identify an endpoint. The measurements must have the properties of being reliable, reproducible, and trustworthy. Given that adversaries naturally evolve with any defense, the adversary will have the goal of disrupting or spoofing this process. To protect against such disruptions, we provide a unique system engineering technique, when applied to CPSs (e.g., nuclear processing facilities, critical infrastructures), that will sustain a secure operational state without ever needing external information or active inputs from cybersecurity subject-matter experts (i.e., virus updates, IDS scans, patch management, vulnerability updates). We do this by eliminating system dependencies on external sources for protection. Instead, all internal communication is actively sealed and protected with integrity, authenticity and assurance checks that only cyber identities bound to the physical component can deliver. As CPSs continue to advance (i.e., IoTs, drones, ICSs), resilient-maintenance free solutions are needed to neutralize/reduce cyber risks. TDA is a conceptual system engineering framework specifically designed to address cyber-physical systems that can potentially be maintained and operated without the persistent need or demand for vulnerability or security patch updates.
The Department of Energy/Office of Electricity Delivery and Energy Reliability (DOE/OE) Cybersecurity for Energy Delivery Systems (CEDS) program is currently evaluating the Roadmap to Achieve Energy Delivery Systems Cybersecurity document that sets a vision and outlines a set of milestones. The milestones are divided into five strategic focus areas that include: 1. Build a Culture of Security; 2. Assess and Monitor Risk; 3. Develop and Implement New Protective Measures to Reduce Risk; 4. Manage Incidents; and 5. Sustain Security Improvements. The most current version of the roadmap was last updated in September of 2016. Sandia National Laboratories (SNL) has been tasked with revisiting the roadmap to update the current state of energy delivery systems cybersecurity protections. SNL is currently working with previous and current partners to provide feedback on which of the roadmap milestones have been met and to identify any preexisting or new gaps that are not addressed by the roadmap. The specific focus areas SNL was asked to evaluate are: 1. Develop and Implement New Protective Measures to Reduce Risk and 2. Sustain Security Improvements. SNL has formed an Industry Advisory Board (IAB) to assist in answering these questions. The IAB consists of previous partners on past CEDS funded efforts as well as new collaborators that have unique insights into the current state of cybersecurity within energy delivery systems. The IAB includes asset owners, utilities and vendors of control systems. SNL will continue to maintain regular communications with the IAB to provide various perspectives on potential future updates to further improve the breadth of cybersecurity coverage of the roadmap.
Critical infrastructure control systems continue to foster predictable communication paths and static configurations that allow easy access to our networked critical infrastructure around the world. This makes them attractive and easy targets for cyber-attack. We have developed technologies that address these attack vectors by automatically reconfiguring network settings. Applying these protective measures will convert control systems into «moving targets» that proactively defend themselves against attack. This «Moving Target Defense» (MTD) revolves about the movement of network reconfiguration, securely communicating reconfiguration specifications to other network nodes as required, and ensuring that connectivity between nodes is uninterrupted. Software-defined Networking (SDN) is leveraged to meet many of these goals. Our MTD approach eliminates adversaries targeting known static attributes of network devices and systems, and consists of the following three techniques: (1) Network Randomization for TCP/UDP Ports; (2) Network Randomization for IP Addresses; (3) Network Randomization for Network Paths In this paper, we describe the implementation of the aforementioned technologies. We also discuss the individual and collective successes for the techniques, challenges for deployment, constraints and assumptions, and the performance implications for each technique.
Critical Infrastructure control systems continue to foster predictable communication paths, static configurations, and unpatched systems that allow easy access to our nation's most critical assets. This makes them attractive targets for cyber intrusion. We seek to address these attack vectors by automatically randomizing network settings, randomizing applications on the end devices themselves, and dynamically defending these systems against active attacks. Applying these protective measures will convert control systems into moving targets that proactively defend themselves against attack. Sandia National Laboratories has led this effort by gathering operational and technical requirements from Tennessee Valley Authority (TVA) and performing research and development to create a proof-of-concept solution. Our proof-of-concept has been tested in a laboratory environment with over 300 nodes. The vision of this project is to enhance control system security by converting existing control systems into moving targets and building these security measures into future systems while meeting the unique constraints that control systems face.
Industrial control systems (ICSs) rely on embedded devices to control essential processes. State-of-the-art security solutions can't detect attacks on these devices at the hardware or firmware level. To improve ICS cybersecurity, defensive measures should focus on inspectability, trustworthiness, and diversity.
Process Control System (PCS) and Industrial Control System (ICS) security is critical to our national security. But there are a number of technological, economic, and educational impediments to PCS owners implementing effective security on their systems. Sandia National Laboratories has performed the research and development of the OPSAID (Open PCS Security Architecture for Interoperable Design), a project sponsored by the US Department of Energy Office of Electricity Delivery and Energy Reliability (DOE/OE), to address this issue. OPSAID is an open-source architecture for PCS/ICS security that provides a design basis for vendors to build add-on security devices for legacy systems, while providing a path forward for the development of inherently-secure PCS elements in the future. Using standardized hardware, a proof-of-concept prototype system was also developed. This report describes the improvements and capabilities that have been added to OPSAID since an initial report was released. Testing and validation of this architecture has been conducted in another project, Lemnos Interoperable Security Project, sponsored by DOE/OE and managed by the National Energy Technology Laboratory (NETL).
This paper describes a new hybrid modeling and simulation architecture developed at Sandia for understanding and developing protections against and mitigations for cyber threats upon control systems. It first outlines the challenges to PCS security that can be addressed using these technologies. The paper then describes Virtual Control System Environments (VCSE) that use this approach and briefly discusses security research that Sandia has performed using VCSE. It closes with recommendations to the control systems security community for applying this valuable technology.
Process Control System (PCS) security is critical to our national security. Yet, there are a number of technological, economic, and educational impediments to PCS owners implementing effective security on their systems. OPSAID (Open PCS Security Architecture for Interoperable Design), a project sponsored by the US Department of Energy's Office of Electricity Delivery and Reliability, aims to address this issue through developing and testing an open source architecture for PCS security. Sandia National Laboratories, along with a team of PCS vendors and owners, have developed and tested this PCS security architecture. This report describes their progress to date.2 AcknowledgementsThe authors acknowledge and thank their colleagues for their assistance with the OPSAID project.Sandia National Laboratories: Alex Berry, Charles Perine, Regis Cassidy, Bryan Richardson, Laurence PhillipsTeumim Technical, LLC: Dave TeumimIn addition, the authors are greatly indebted to the invaluable help of the members of the OPSAID Core Team. Their assistance has been critical to the success and industry acceptance of the OPSAID project.Schweitzer Engineering Laboratory: Rhett Smith, Ryan Bradetich, Dennis GammelTelTone: Ori Artman Entergy: Dave Norton, Leonard Chamberlin, Mark AllenThe authors would like to acknowledge that the work that produced the results presented in this paper was funded by the U.S. Department of Energy/Office of Electricity Delivery and Energy Reliability (DOE/OE) as part of the National SCADA Test Bed (NSTB) Program. Executive SummaryProcess control systems (PCS) are very important for critical infrastructure and manufacturing operations, yet cyber security technology in PCS is generally poor. The OPSAID (Open PCS (Process Control System) Security Architecture for Interoperable Design) program is intended to address these security shortcomings by accelerating the availability and deployment of comprehensive security technology for PCS, both for existing PCS and inherently secure PCS in the future. All activities are closely linked to industry outreach and advisory efforts.Generally speaking, the OPSAID project is focused on providing comprehensive security functionality to PCS that communicate using IP. This is done through creating an interoperable PCS security architecture and developing a reference implementation, which is tested extensively for performance and reliability.This report first provides background on the PCS security problem and OPSAID, followed by goals and objectives of the project. The report also includes an overview of the results, including the OPSAID architecture and testing activities, along with results from industry outreach activities. Conclusion and recommendation sections follow. Finally, a series of appendices provide more detailed information regarding architecture and testing activities.Summarizing the project results, the OPSAID architecture was defined, which includes modular security functionality and corresponding component modules. The reference implementation, which includes the collection of component modules, was tested extensively and proved to provide more than acceptable performance in a variety of test scenarios. The primary challenge in implementation and testing was correcting initial configuration errors.OPSAID industry outreach efforts were very successful. A small group of industry partners were extensively involved in both the design and testing of OPSAID. Conference presentations resulted in creating a larger group of potential industry partners.Based upon experience implementing and testing OPSAID, as well as through collecting industry feedback, the OPSAID project has done well and is well received. Recommendations for future work include further development of advanced functionality, refinement of interoperability guidance, additional laboratory and field testing, and industry outreach that includes PCS owner education. 4 5 --This page intentionally left blank --
Supervisory Control and Data Acquisition (SCADA) systems for automation are very important for critical infrastructure and manufacturing operations. They have been implemented to work in a number of physical environments using a variety of hardware, software, networking protocols, and communications technologies, often before security issues became of paramount concern. To offer solutions to security shortcomings in the short/medium term, this project was to identify technologies used to secure "traditional" IT networks and systems, and then assess their efficacy with respect to SCADA systems. These proposed solutions must be relatively simple to implement, reliable, and acceptable to SCADA owners and operators. 4This page intentionally left blank.