To: Members of the Castle Rock Water Commission
From:
Mark Marlowe, P.E., Director of Castle Rock Water
Title
Resolution Adopting the 2019 SCADA Capital Implementation Plan for Castle Rock Water
Body
________________________________________________________________________________
Executive Summary
Castle Rock Water seeks Town Council approval of a Resolution (Attachment A) adopting the 2019 Supervisory Control and Data Acquisition (SCADA) capital implementation plan. This capital plan was developed by staff from the 2019 SCADA Master Plan. The 2019 SCADA Master Plan was prepared with help from a consultant, Jacobs Engineering Group, Inc., and represents the starting point for an overhaul of the existing SCADA system that monitors and controls operation of the water and wastewater systems in Castle Rock. The SCADA Master Plan contains confidential and sensitive information about Castle Rock Water’s (CR Water’s) operations, and as such, it is not included in this memorandum. The current SCADA system has been in place for almost two decades and technology has evolved immensely over that period. Although some hardware components have been replaced and software upgrades have been performed over the years, it is time to migrate the control system over to a more robust operational control system that will address deficiencies within the existing system and bring the system into compliance with modern cybersecurity and equipment safety standards, reduce risk from aging equipment failure, reduce costs of support by utilizing current, non-obsolete hardware and software, and improve operator efficiency by standardizing policies and procedures.
Based on the SCADA Master Plan, staff developed a revised Capital Implementation Plan to migrate to a new SCADA system. The estimated cost for this migration is approximately $12M. Remaining with the existing system, would cost approximately $9.7M in terms of capital investment to replace and upgrade existing equipment. The life cycle cost of the existing platform is expected to be higher than the new platform and continuing with the existing system does not address the eventual obsolescence of the existing system. The CIP would implement the changes to the new system over the next 7 years. Castle Rock Water has about $4M in our current financial plan for rehabilitation and replacements of the SCADA system over the next 7 years. Additional capital investments are needed over and above the current financial plan regardless of whether Castle Rock Water stays with our current system or migrates to a new system. As such, implementation of this revised SCADA CIP and migration to the new system will increase our capital plan for SCADA by ~$2.3M over the next 7 years. A copy of the proposed CIP is included in Attachment B.
There also will be an increase in operational costs for SCADA as a result of the needs identified in the SCADA Master Plan, including one additional full time equivalent. This additional operational cost is recommended regardless of the platform or system type. CR Water worked closely with the Division of Innovation and Technology (DoIT) to develop the SCADA Master Plan and the revised CIP and will rely on DoIT to provide technical support for the SCADA system going forward. This will require the addition of at least one fulltime equivalent in DoIT to be fully funded by CR Water. In recognition of this additional support from DoIT, a Memorandum of Understanding between departments has been developed.
Notification and Outreach Efforts
This project is a behind the scenes effort that requires no public outreach. Implementation will be performed in a phased approach that will allow for maintenance of existing operations and no disruptions in service.
Discussion
Castle Rock Water uses Supervisory Control and Data Acquisition (SCADA) to monitor and control all processes in the water and wastewater systems for Castle Rock. The SCADA system has been in place and evolving with changes in the system since the mid-1990s. Since that time, CR Water has been using Bristol Babcock equipment and Iconics software to run the SCADA system. These products have served adequately over the years, but other technology has developed that has surpassed the performance of Bristol Babcock/Iconics. Given the complexity of CR Water’s system, the SCADA system is a crucial component for day to day operations as well as managing risk and resilience associated with process automation. To truly evaluate the direction of where CR Water should go with technology, CR Water issued a request for proposals (RFP) in February 2019 to solicit professional services to update the SCADA master plan. The scope of work included:
• Identifying SCADA system upgrades required to maintain the current water and sewer system facilities, and to maintain the SCADA system as new facilities are brought online;
• Reviewing options for utilizing other hardware and software systems, in addition to Bristol Babcock and Iconics;
• Designing a plan for implementing a cybersecurity program in accordance with Department of Homeland Security national standards;
• Developing a plan for enhancing, updating, or replacing the current SCADA radio communications system;
• Developing a plan for enhancing, updating, and/or replacing the current secondary backhaul network that supports video cameras and access control at remote sites;
• Identifying the hardware, software, human resource, training, and other recommended improvements needed to support the SCADA system;
• Determining the logical order in which the tasks identified in the master plan should be performed and providing estimates of the time needed to complete each major phase of the CR Water SCADA Master Plan; and
• Preparing budget estimates for the tasks, identified by year, to implement the described CR Water SCADA Master Plan.
Through a competitive RFP process, Jacobs Engineering Group, Inc. was selected as the recommended consultant to prepare the plan. Over the past ten months, Jacobs has worked closely with CR Water and DoIT to develop a comprehensive plan that provides direction for improving the SCADA system to ensure a robust control system for both the near term and long term future. The major goals for guiding the development of the SCADA master plan were:
• Ease of support and integration;
• Ease of operations and maintenance;
• Reliability and availability of hardware and software;
• Security; and
• Standardization.
Based on these goals, the recommendations in the master plan are:
1. Programmable Logic Controllers (PLCs) - proceed with Allen Bradley PLC hardware and Rockwell Automation Studio 5000 suite of products for PLC software updates with PlantPAX® PLC controller software;
2. Human-Machine Interface (HMI) - use Allen Bradley FactoryTalk View Site Edition (SE) for HMI software;
3. Operational Technology (OT) - upgrade OT architecture and equipment to accommodate Rockwell PlantPAX®;
4. Backhaul Network - proceed with a Cambium radio network for the existing, site-specific IT and video security needs where low latency traffic is a priority, and FluidMesh radio network for SCADA backhaul;
5. Telemetry - proceed with a General Electric MDSTM Orbit radio upgrade for the telemetry;
6. Cybersecurity - perform resiliency review and emergency response plan per risk analysis and management for critical asset protection; develop comprehensive Industrial Control System (ICS) cybersecurity policies and procedures based on National Institute of Standards and Technology cybersecurity framework;
7. Physical Security - ensure all critical components are in locked rooms or panels;
8. Staffing - continue to train existing SCADA/ICS staff in new PLC and HMI platforms; hire additional SCADA/ICS staff if needed during upgrades; hire network/server specialist engineer; train cybersecurity engineer to help manage SCADA system; and
9. Standards Development - develop control system hardware and software standards to address all major aspects of structured implementation under common rules of integration (PLC, HMI, OT, Backhaul, and Telemetry).
Based on these recommendations, staff worked with Jacobs to develop a new CIP for the SCADA system. The revised SCADA CIP was added to the 2019 rates and fees study and models. As a result, additional prioritization of capital expenditures was needed to identify projects which can move further out for implementation to allow the SCADA projects to proceed in the next seven years without impacting the current projected rates and fees. Replacement of Tank 6B had to be moved out past 2026, and staff also had to reduce the investments in pipeline replacement for several years from $1.5M to $0.6M to maintain the current projected rates and fees. Staff believes these changes to be the best course of action as the revisions to the SCADA CIP take precedence to Tank 6B and the pipeline replacements.
Budget Impact
Based upon a prioritized project plan, expenditures are projected as shown in Figure 1. This projection is preliminary and will be refined as more details are developed. Since the master plan was completed after the 2020 budget was prepared, the costs were not included in the 2020 budget or the 2020 through 2024 capital improvement plan. Projected expenditures will be included in the upcoming budget cycle. SCADA system upgrades will be funded through rates and a financial impact has been analyzed with respect to future rates. As noted above, several changes to the 2020 to 2024 capital improvement plan are needed to ensure that the revised SCADA CIP can proceed under the current rates and fees plan.
Staff Recommendation
Staff recommends Town Council approval of a Resolution adopting the 2019 SCADA CIP.
Proposed Motion
“I move to recommend to Town Council approval of Resolution as presented”
Attachments
Attachment A: Resolution
Attachment B: 2019 SCADA Capital Improvement Plan