When a cyber incident occurs, getting to the bottom of what happened quickly and accurately is not always simple. Things are often not what they seem, and it’s easy to over (or under) react in the heat of the moment. That’s where dedicated incident response professionals come in. These people can investigate what happened to help prevent future incidents and make sure your business can keep running as usual.
ICS Pulse talked to Lesley Carhart, director of industrial control system security incident response at Dragos, at the RSA Conference in San Francisco about working in incident response, a power plant that seemingly turned on by itself and the maturity journey of operational technology (OT) cybersecurity. Listen to the full podcast here.
The following has been edited for clarity.
ICS Pulse: Let’s talk incident response. I know you are here at RSA to give a talk on incident response. What is your talk covering?
Lesley Carhart: I will be talking with two other amazing women, Wendi Whitmore and Katie Nichols, and we will be moderated by Lily Hay Newman, the amazing journalist. We are going to be talking about cybersecurity war stories for incident response. So what’s going on in the incident response space this year and what’s trending, what’s changing and what types of cases we’re seeing. People always like stories, so we’re going to tell some stories about what’s going on in that space and what’s new and what people should be looking out for.
ICSP: I’m obviously going to have to ask for one of those stories. If you don’t mind giving us one, what is one of your incident response stories that you like to tell?
Carhart: My most wild recent one that my coworkers just love is the power plant that turned itself on. We got called in to investigate a peaker plant. Peaker plants are the power plants that you use during the summertime when there’s a bunch of air conditioning on, if you live somewhere where there’s multiple seasons. So you have extra power plants that when everybody uses a lot more power during the summer, you turn them on. But this was during the winter, and we got called by a customer who was like, “Our peaker plant just turned on by itself. It must be China.” And my colleagues we’re all like, “Wow, it must be China. It takes 12 buttons to turn on that plant.” So we go out to this remote plant in the dead of winter, and we do a really good forensic investigation and see no signs of evil.
There are two people there. They were looking at each other. There are security cameras. We do a full forensic rundown of the few computers in there. Everything looks fine. There’s no malware. There’s no sign of hacking. Nothing is even connected to the internet. And we’re like, “Yeah, this is really strange.” It all comes down to this computer that’s out in a shed by the equipment. It’s like one of those outdoor hardware store sheds. They go in there and there’s a bunch of weird stuff up on the computer screen, like command prompt and stuff. And they’re like, “It’s getting hacked. It’s getting hacked.” And, yeah, it does for a moment look like it’s getting hacked. But I look at it a little more, and I’m like, “Why would a bad person — I get command prompt, I get calculator — but why would they open the RDP GUI? Why would they open notepad?”
Like, all these Windows applications are open on it. I’m like, “Yeah, this looks kind of random.” And I look at it a little bit more, and I look at how long those windows had focus, how long somebody had clicked on it, and it was like 10 seconds. And I’m like, “It’s like somebody’s bonking Windows applications open on the computer.” I just don’t buy it’s a hacker, and everybody’s still really freaked out. So I’ve got to prove my gut feeling that this was not malicious. This wasn’t a hacker. And so I’m like, “It has to have been something just randomly clicking buttons on this computer.” So I tell my coworker to go out by the keyboard and mouse that are on there and freeze them and heat them up and see if that makes it happen. That didn’t work.
But then I start looking at a touchscreen on the computer, and I’m like, “Well, it’s kind of sensitive.” So I’m like, “OK, open up Paint and leave it on overnight. Let’s see what happens.” Sure enough, the next morning they come in, and there’s pretty little whirligigs all over this touchscreen on Paint. So what happened is you get this really bright screen in the dead of winter in a slightly climate-controlled outdoor shed that’s not fully sealed. You get this really bright screen in the dark, and all the bugs come and they land on this really bright screen. It was like 10,000 monkeys at 10,000 keyboards. That was the one night after however many years they hit the right sequence of buttons to turn on the power plant.
ICSP: Were people skeptical when you told them this? Like, “That can’t be right. There’s no way bugs could have done this.”
Carhart: I mean, I proved it really well. It sounds like I was just like, “Oh, just open Paint.” That’s how my coworkers tell it now. Lesley waltzed in and they were just like, “Oh, oh, just open Paint.” No, I spent hours and hours of doing forensics to look at what happened on the computer, and I could see things from a timeline perspective, from a narrative perspective on the computer happening. So we know the exact time the power plant turned on, and we know within a few seconds these other windows opened up in the computer, and nothing else happened on the computer. It didn’t connect anything. Their network was offline. Nothing else happened. But we see suddenly a bunch of windows that have buttons on the desktop get clicked. And then, of course, I proved it by having them open Paint, and then they saw the random clicks all over the screen the next day. It’s an interesting world, industrial incident response.
ICSP: That’s unbelievable. I would think with your engineer’s brain, which obviously you have, incident response is the perfect career for you.
Carhart: Oh, it is, but it’s also the engineer gut feeling. If you’re a good engineer, you also have a good gut feeling about things, and you have to have that kind of gut feeling over some years of experience to be really good at incident response. It’s like, “This doesn’t feel like a hacker, or this feels wrong and maybe it is something malicious.” So being able to kind of intuit as well as take things apart. That’s the interesting challenge that goes along with learning and being good at incident response.
ICSP: Dragos works primarily in OT cybersecurity. Do you find that some people are still reluctant to even attempt cybersecurity on the operational side of things, or do you think it’s getting more accepted now as, “Look, if we want to do business and keep the lines running, we’ve got to have OT cybersecurity”?
Carhart: It’s a mix. It varies vastly by region and by vertical. Some industry verticals are much farther ahead than others in industrial cybersecurity. They have more funding. They have more staff. They have more regulation even. And then others are still running on a tiny margin, and that makes it rather challenging for them to do any cybersecurity at all. Look at municipal utilities. Look at manufacturing. We see the whole spectrum of maturity and cybersecurity for OT.
Do you have experience and expertise with the topics mentioned in this article? You should consider contributing content to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.
