To: Members of the Castle Rock Water Commission
From: Mark Marlowe, P.E., Director of Castle Rock Water
Shawn Griffith, Assistant Director of Operations
Nic Van Kooten, SCADA Superintendent
Title
Discussion/Direction related to item #3: Addition of Town New SCADA Positions, an Operational Technology Network Engineer and an Operational Technology Systems Administrator [Entire Castle Rock Service Area}
Town Council Agenda Date: February 18, 2025
Body
________________________________________________________________________________
Executive Summary
The SCADA Master Plan (MP) was approved by the Town Council on December 17, 2019. The plan was updated in the 2025-2029 SCADA Master Plan in 2024. Additional needed work, not previously identified in the original SCADA MP scope, was included in the updated MP. Revised estimated costs for full implementation were updated to $15M. This cost may go higher and will be reviewed in the 3rd Quarter of 2025 by an outside firm.
The following program description is supported and described in both the 2020-2024 and 2025-2029 SCADA MPs, finalized and accepted in January 2021 and September 2024, respectively:
“The Castle Rock Water (CRW) Supervisory Control and Data Acquisition (SCADA) Master Plan is the starting point for the development of the CRW SCADA system functional requirements, which for this Master Plan includes cybersecurity, operational technology (OT), telemetry, backhaul, programmable logic controllers (PLCs), and human-machine interface (HMI). During the master planning effort, investigations were performed to determine all desired functions, features, and requirements for each subsystem (PLC, HMI, OT, cybersecurity, telemetry, backhaul). This Master Planning effort provides an opportunity to identify deficiencies within the existing system, consider new technologies, and document present and future system requirements.”
As part of the 2025-2029 SCADA MP, finalized in September 2024, the document identified in Section 5.1, the need to hire an Operational Technology (OT) “Network Engineer”, as the individual who maintains the network and maintains CRW’s SCADA/OT servers as well as an OT “Systems Administrator”.
Discussion
The SCADA Master Plan outlines the need for cybersecurity as a way to protect CRW infrastructure and the Town’s water supply. In today’s fast-evolving technological landscape, maintaining current tools and architecture is vital. Utilities, particularly in the water and wastewater sectors, are frequent targets for cyber-attacks, often following the patterns established in the power sector. As a result, water utilities are adopting similar protocols and equipment to enhance their cybersecurity defenses. CRW has lacked upper-level staff with the specialty training and education required to oversee physical and cyber security issues.
The OT Network Engineer will be responsible for:
• Implementing and maintaining OT solutions based on proven security architectures, including virtualization, networks, security platforms, and various other OT technologies.
• Participating in the design, documentation, implementation, and maintenance of Industrial Control System (ICS) networks.
• Securing ICS projects and processes, including backup and disaster recovery, following industry best practices, NIST guidelines for ICS security, and Castle Rock Water requirements.
• Installation and configuration of network switches, routers, firewalls, virtualized servers, client workstations, and various cybersecurity platforms and tools.
• Providing support and troubleshooting for network communications, hardware, firmware, and security settings.
The OT Systems Administrator will be responsible for:
• Maintaining CRW’s SCADA/OT servers
• Managing the SCADA domain
• Overseeing Active Directory
• Ensuring the functionality and security of domain-linked computers, including client computers, and SCADA service laptops
• Manual updates, due to network isolation, for CRW’s servers, ASA switches, and computers, as they cannot receive automatic updates from the internet. The Systems Administrator will manage these manual updates to maintain a robust security posture and protect against potential cyber threats.
The need to provide staff that can utilize current tools and architecture in the fast-paced environment of technology is essential. Cyber hackers have consistently attacked Utilities over the past few years. Power is a primary target and water/wastewater is a very close second. Water utilities have been able to learn from Power utilities and are now implementing many of the same protocols and equipment that Power uses. The design, maintenance, and repair of a protected SCADA system is crucial to the safety of Town facilities and its residents.
For the past five years, CRW has contracted with firms that offer the services of Network Engineers, Systems Administrators, and Cyber Security experts. While this approach has had its successes, particularly in recommending specific devices and providing training on Best Management Practices, it has also revealed consistent drawbacks. The immediate availability of contractors has been a challenge, compounded by their lack of investment in CRW’s unique SCADA environment. While contractors meet their obligations, they often lack familiarity with Town assets, their locations, and their critical importance to the operation of CRW.
To fulfill our vision of robust cybersecurity and operational efficiency, an active, informed, and responsive OT Network Engineer and System Administrator with industry-specific technical knowledge is crucial for effective implementation and management. These staff members would not only address immediate technical needs but also ensure that our systems are safeguarded against evolving threats.
Budget Impact
These two positions will be paid from the SCADA Water, Water Resources, and Wastewater funds through a first-quarter budget amendment, costs shown in the chart below:
|
Budget Year |
2025 |
2026 |
2027 |
2028 |
2029 |
|
Annual Cost for two FTEs |
$341,198 |
$356,455 |
$372,450 |
$389,223 |
$406,818 |
Staff Recommendation
Staff Recommends Approving the request for two new full-time equivalents, the SCADA Network Engineer and the Systems Administrator with a vehicle.
Attachments
Attachment A: SCADA OT Two Positions-Vehicle Worksheet
Attachment B: Business Case 2025 OT Network Engineer
Attachment C: Business Case 2025 OT System Administrator