Weaknesses in the software supply chain, coupled with reluctance from vendors to implement new changes or stray from familiar systems, have been major factors in allowing threat actors to wreak global havoc. When hackers exploit weak links in the supply chain, even major companies like SolarWinds and Kaseya are at a greater risk for attacks. CFE spoke with leading experts for this week’s Industrial Cybersecurity Pulse Roundtable discussion to further discuss supply chain risk and how to mitigate it.
Once again, joining Gary Cohen, senior editor of Industrial Cybersecurity Pulse, are Ron Brash, vice president of technical research and integrations at aDolus Technology; Eric Byres, chief technology officer (CTO) and a board member at aDolus Technology; and Dino Busalachi, CTO and co-founder of Velta Technology.
This discussion has been edited for clarity.
ICS Pulse: Attacks like SolarWinds and Kaseya really put an emphasis on the dangers of a weak supply chain, or at least a weak link in that chain. For attackers, software supply chain risk is a gold mine if they can get into one company to impact hundreds of companies. What do companies need to be doing going forward to try to cut off supply chain risk?
Ron Brash: Right now, probably the biggest problems that we have on the supply chain is the distribution and lack of a decentralized model for software. Think of open source. Is open source any more secure than closed source? No. For most cybersecurity in software, 80% of the bugs in software these days are related to the lack of proper engineering. If you combine that with the number of developers who do not think cybersecurity is their job, or it’s someone else’s job, or it’s a tool, or they’re to be told what to do, that gets compounded when packages start building off of other packages.
You wind up with an amplification effect of risk in supply chain and software. If you took a Rockwell, or Schneider, or GE or any one of those top-tier vendors — and it gets worse the farther you go from the top tiers, it’s an exponential increase of risk when you go to second and third tiers — but even then, they’re integrators. They are gluing together pieces of pieces inside of products. They’re going to some manufacturer that builds PCBs (printed circuit boards) and saying, “OK, I’d like a board that does this, this and this to start with. This is our basic requirement.” And they say, “Cool, I have something for you here. It’s got a board support package, which reduces your development time. It’s got SDKs (software development kits). It’s got sample code. You’ll get it up and running, and then you can start piling on your code.”
That’s where the problems start, [when] the SD key and the board support packages and all the software that goes into building that board, including binary blobs for FPGAs (field-programmable gate arrays) and for certain types of smart chips that do certain things, all [the problems] start right there. And there’s even suppliers to suppliers for whoever built that reference design there, too.
But the OEMs (original equipment manufacturers) are so much farther away from that. The supply chain, even if the OEM did care about cybersecurity, it’s only in their code that they generally care about it. It’s in their first-party code. The risks there are monumental. And because, as I said earlier, we’re getting to the place of systems, especially with software being multiple pieces, where the developer’s saying, “Oh, that’s cool. I want that.”
The software says, “Sure, press import,” and you get 50 new imports to some library for some high-level language. We are in a place where a device that should be running something that’s 10 megabytes of software is now 100, a gigabyte of software. No developer knows any of that anymore. The supply chain, in terms of software and then the engineering of the pieces that go together, we’re in for some eye openers, I think.
And the aviation industry has talked about this with the DO standards on certifying chips to only speak to each other in certain ways and components. We’re going to have to get that way for consumer electronics, for applications that run on Windows or in the cloud or IIOT (Industrial Internet of Things). We’re all in a very interesting place.
Dino Busalachi: The supply chain, just keeping an eye on the news, has already got plenty of problems as it is around the world, and we’re going to see that pretty quickly as we’re pricing. But [in regard] to the manufacturing environment and getting into the technology piece of it, I’m going to speak to it from more of an end-user perspective and having to deal with OEMs and SI’s (stem integrators) who bring forth these $8 to $10 million machine centers and skids that come into these manufacturing environments. They’re very stringent on making any changes to their stock. Even trying to get them to try a different remote access method is difficult. Forget patching machines. They’re not even going to allow that to be done for an asset that’s been bought that’s got 25 [to] 30 years of life to it.
It freezes in time. I was at a plant just recently, [a] Windows 10 machine, [an] engineering workstation still with the original CBEs (computer-based education systems) on it — 370 CBEs, and they didn’t even patch it at all to try to resolve anything. From the supply chain perspective, once it gets into the plant, that technology tends to freeze and it doesn’t move. Anything else coming in, as far as security in mind, to try to work on the supply chain, they just want those machines to make whatever it is they’re supposed to make as quickly as they can make it with as little disruption as possible.
ICSP: You both mentioned patching and how that can be an issue with the supply chain. But if you don’t know what’s in this piece of software that you have now integrated into your entire company, there’s no way you can really know all of your vulnerabilities.
Busalachi: It could be in the chips, too. It’s not just in the software; it could be in the chips side.
Eric Byres: This is why supply chain isn’t a local problem. It’s a software transparency problem. Nobody can solve the supply chain problem by themselves because it’s downstream, it’s upstream. That’s why I’m really excited about software bill of materials, not as a silver bullet that’s going to solve all the problems, but it’s the first step toward starting to understand, what is the real product I bought? What did I actually get? And with that product that I bought, what are the risks I’m facing, and which ones of those are critical to me that I need to figure out a way to address? It won’t be all of them, but there’ll be certain things that are really, really high risk for your organization, your operation.
That’s really the key part here is that we move away from just compliance. Listening to Dino and I, you might think we’re anti-patching, but we’re not. It’s just smart patching. Figure out what has to get patched, and if you don’t want to patch it, then figure out a mitigating control to get around it so that you’re still not [at risk]. You’re not exposed. But the important thing here is to be smart about this [and] figure out what your risks are, figure out what vulnerabilities are going to impact your risks and then figure out what you’re going to do about it. Be it patching, be it firewall rule changes [or] be it removing that piece of equipment.
Busalachi: The key is getting on a path to do something. Regardless of the technology, to get on a path. It’s not going to come find you. You’re going to have to go find it. And there’s no wrong answer. What you decide today compared to where you’ll be in five years, you’re not going to be at 100%, but you might be at 90% or 95%. I try to get clients to think about start[ing] down the path, and when you’re on a path, keep moving. You [have] to be able to get visibility into any framework that you go after. I don’t care if it’s NIST, MITRE ATT&CK [or] IEC 62443, take your pick. Or NERC-CIP — [it’s about] visibility.
First thing you’ve got to do is get asset inventory and get visibility into the things that you need to see that you don’t know anything about. You’ve got to do that first. And it could take you a year, two years, [to] do that depending on the size and scope of your facilities [and] the number of plants in your fleet to get to that data before you can even start a mitigation or mediation conversation on what you’re going to do next. That’s the biggest challenge that I see clients trying to grapple with. Even clients that have been going down this [path] for a couple of years already are frustrated because they were sold a technology that was just given to them from a transactional perspective without any support on how to deal with the volumes of information that come flying up out of that industrial environment.
It overwhelms them, and they don’t know how to organize it and calibrate, commission, operationalize, localize and make sense of it. They get frustrated quickly. There are even large enterprises that have gone down this path that are having to rethink what they’re doing. It requires professionals to come in and help them, just like they would hire a system integrator to help them build a packaging machine. They don’t build the packaging machine. They hire somebody to do it for them. Then it slowly gets turned over to them so they can start producing their goods. If they get in a jam, they need help or they want to make changes to it, they go back to the SI to help them. A lot of these cybersecurity tools [you] want to put in OT platforms, [they] need to go down that same machine center mentality path.
For more with Brash, Byres and Busalachi, check out their first roundtable discussion where they talk about the efficacy of government regulations regarding cybersecurity.
Do you have experience and expertise with the topics mentioned in this article? You should consider contributing content to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.
