Having comprehensive list of security-related patches by using operational technology (OT)-specific patching tools to gather complete software, vulnerability, and security patching information sounds simple. Information technology (IT) security professionals will tell users to scan it with a vulnerability assessment tool. Or, if it’s an original equipment manufacturer (OEM)-vendor relationship, use their “approved” patch list. Or, check the vendor’s website for patches labeled “security”. Or manually or automatically identify common vulnerabilities and exposures (CVEs) for the software/firmware on the devices and access the patches that address those CVEs.
OT security patching management is far from simple
- Vulnerability scanning tools negatively impact operations. It is key to use an OT-specific vulnerability assessment platform to ensure deep visibility without risking the sensitive OT assets.
- In many cases, operators do not have a robust asset or software inventory. Some rely on manual spreadsheets, others on passive inventory tools which only see what’s on the wire. Deep visibility to every piece of software on these systems, the accurate patch status on each, and tying those to vulnerabilities is important.
- OEM-approved software patch lists often exclude patches for “critical” or “high” rated vulnerabilities. Without a comprehensive view of the vulnerabilities present and “relevant, if not approved” patches, operators are blind to their risk OEM-approved patch lists are NOT always inclusive of all third-party software deployed on OT devices. Users are not supposed to load this software on these types of systems, but based on Verve’s vulnerability assessments, we know this is quite often not the case. Adobe, Office, Java, and a range of more surprising software are often present on OT systems. If these applications are not required, remove them. If they can’t be removed, ensure users are independently checking these other applications the OEM vendor may never test.
- Often, OEM vendors’ labeling convention for their patches as “security” excludes patches that may have security-related content within them. Make sure to review not just those called “security” but all patch notes to capture all patches that have security-related content relevant to the environment.
- Without automated vulnerability management tools, a manual review of the CVE and other databases may lead to missed vulnerabilities if only searching for specific versions. CVEs are not always structured in a standard format. It is critical that automated tool or manual process has a deep view of the “details” in the CVE database and have wide logical search criteria to ensure users capture the full number of relevant vulnerabilities and patches for an environment.
- Finally, OT/ICS embedded devices are constructed from a range of underlying software packages. The Ripple20 and Urgent11 vulnerabilities were major industry milestones as they identified critical vulnerabilities in underlying software components. Unless the vendor has a robust software bill of materials (SBOMs) and active/aggressive vulnerability program, the asset that sits in the asset database may not appear in the vulnerability databases, even though the underlying components are vulnerable. Ensure the vulnerability management tool can identify potential “unseen” vulnerabilities in the SBOM.
Review security patching relevance to specific systems, status, and configurations
A simple vulnerability assessment suggests the need for a patch, even though the particular system may not require that patch for a variety of reasons. The patch may not be “relevant” for that machine as other patches supersede it. Or, the configuration of the device may make the patch irrelevant because the device is not set up to run certain services or capabilities. For instance, many networking device vulnerabilities would imply the need for a patch, but further reading would find that the patch is only necessary if the device has certain enabled configurations. Several key recommendations flow from this:
- Confirm vulnerability and patch tools see detailed patch status, not just an OS version. This is why Verve leverages deep insight into all patches deployed and which ones are relevant, not just by looking at OS version.
- Ensure the vulnerability tool considers configuration settings and other compensating controls to reduce the need and urgency for a particular patch.
- Look to mitigate the “Never” or “Next” vulnerabilities with alternative mechanisms that negate the need for a patch if certain configuration settings changed or services disabled.
Four issues to ensure safe, secure install path and process
Four issues must be considered in the patch management process:
- In many cases, ICS devices have not been patched in many years. New patches may require the operator to “walk” the patch path from earlier patches up to the most recent one. Deploying the most recent patch can cause operational errors and be difficult to reverse. Plan the patch walk carefully for each asset prior to beginning. An example would be a customer of ours with an OEM system that was never patched and had to walk through 10 years of patches, batch at a time.
- Make certain you have updated and tested backups. Prior to conducting any security patching activity, the operator should ensure that the device has a robust and recent backup. Verve provides regular updates on backup status so users know before they begin whether they have a quality backup available in case you need to restore in case of patch issues. We have seen OEM-approved patches destroy an OPC server used for collecting setpoint information and controlling the process and the backups were invaluable in getting the plant up and running again.
- Confirm HASH values of patches prior to deployment. The Dragonfly attack several years ago highlighted the risk of insecure patching. In the North American power industry, NERC now requires this as a step in all patching. This should, however, be common practice in all ICS patching to ensure the secure delivery of patch files.
- Test and confirm. No ICS patching should be “spray and pray”. Take time to test on a set of devices. Test operations after deployment. Let those devices soak for some period of time. Then move on to other, similar devices. It is not enough to just make sure the device reboots effectively. Proper patch deployment includes testing of the devices and services after the fact to ensure that all key operations processes are in working order.
When patching ICS/OT human-machine interfaces (HMIs), workstations, or servers, ensure “downstream” effects are identified. Control systems patching challenges do not stop at the need to reboot. In many cases, changes to software at one part of the process have implications on other parts of the process. Worse, a “Now” or “Next” device patch could force a “Never” device patch. Users need to know the implications of applying such patches to a control system.
This can become complex and require deep knowledge of the OEM patching process. For instance, an update to the OEM’s engineering software may force updates to controller firmware or I/O cards, making it impossible for the standard software to work again unless these devices are updated.
Controller firmware updates may not have been in the original plan or may require an outage, but now loading a single OEM-approved patch results in the loss of all engineering capability until the firmware updates are completed on the controller and IO cards.
This article originally appeared on Verve Industrial’s website. Verve Industrial is a CFE Media content partner.
Original content can be found at verveindustrial.com.
Do you have experience and expertise with the topics mentioned in this article? You should consider contributing content to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.
