Recorded March 9, 2021
Industrial Control Systems (ICS) and Operational Technology (OT) have risks and consequences in the real world, such as the health and safety of people, but how those industries handle the potential cybersecurity risks varies greatly depending on the regulation that has been applied. The US Government has declared many different industries as critical infrastructures with different levels of prioritization placed on cybersecurity regulation.
Hey, welcome back to security and compliance weekly Episode 64. We’ve been talking to Jim Gilson of Dragos about all things security and compliance as they pertain to industrial control systems and critical infrastructure. We covered a lot of ground, I think in the first segment talking about the complexities and the difficulties from getting companies to, to pay attention to security, and so on and so forth. So we want to jump back into that discussion. And hopefully in the second segment, talk about some things that can be done to help companies along whether that’s regulatory compliance or not.
Before we jump back into the discussion to have a couple more announcements if you want to stay in the loop on all things security weekly related, you can do so by visiting securityweekly.com/subscribe, where you can subscribe on your favorite podcast catcher, or you can sign up to the YouTube channel. Or you can sign up for our mailing list for which is the most fun for us, join our Discord server and enjoy the conversation that goes on live while we broadcast. Also, of course, if you have any specific guest or topic suggestions you’d want to share with this, you can do so by submitting your suggestions to securityweekly.com/guests. There’s a form there, fill it out, we review it, and we’ll get back to you once we review.
All right. Let’s get back to the discussion and Jim Gilson. So, Jim, let me start off, get this out of the way, right at the outset, and then the counter can go wild for the rest of the discussion. But I was kind of curious, from a compliance perspective. I mean, you know, full disclosure, I think my belief is that a lot of these companies that are struggling with finding budget and the resources and all that, and especially the public sector ones where they got to get where it becomes political, and they got to get permission to my way of thinking these smaller companies, as you described, that sort of fallen below that cybersecurity poverty line, the only way that they get nudged in the right direction is through regulation and compliance. So, the one thing I wanted to ask you about is, and this is the serious PCI question is if there’s so many companies that are especially like utility companies, where I’m sure they let their consumers pay their bills, with credit cards. And therefore they in theory are subject to PCI. So the PCI data security standards is sort of in the ecosystem anyway. It’s is there? I guess, a Do you see that come up any B are there? Is there a way to connect the dots because some part of the company has to do PCI and therefore security, to some degree already? Is there a way to sort of bridge that and bring it over to the you know, production, industrial control area, operational security technology area of these organizations? and say, hey, look, this is something that you have to do anyway, it’s a pretty comprehensive framework for doing security things, why not? We why not apply this into this environment? Is that even feasible? Do you see that as something that could help things along?
So, it for the most part, these systems are separate. So the business systems are separated from the OT ICS systems are not going to have a direct relationship between the billing systems and the actual control systems themselves? Now that said there are plenty of cases where you may not have great segmentation between the networks. But they’re supposed to be separated. From a risk point of view. They’re supposed to separate out the business systems, from the OT ICS systems in some way.
For that white coffee maker, that shuts down entire factories or the cable that went across the hallway or the…
Yeah, yeah, no, I mean, I’m not gonna say that it doesn’t happen in the industry, there are plenty of cases where things are connected, or there are dual home systems, or there are back channels, or there’s open VPN connections or things like that. So there are plenty of cases where they are interconnected to some degree.
But I don’t want to belabor this Jim, but I do want to clarify, I’m not talking. Yeah, I understand that the from an architectural networking perspective, these systems are probably separated, segregated, segmented, not physically isolated. What I’m asking more for is if you look at the PCI data security standard as a framework with its six overall all goals 12 major requirements, things like, you know, build secure systems have some sort of hardening standard where you set systems up. Granted, they were put in 20-30 years ago, you know, the idea of keeping things secure patching, providing mitigations, or offsets for identified vulnerabilities, occasionally testing things, having logging enabled, testing logs, controlling access, implementing two factor or multi-factor authentication, all the things, you know, granted that you’re not going to be able to take advantage of things that you’ve done for the other networks and other areas. But something to use as a guide for what do we do here? Is that something where, because it’s in the environment already, you think there would be an advantage to say, Hey, why don’t? Or is it? PCI, in the in the mind of the world is so isolated to certain types of systems, it doesn’t make sense to and you’re treated as a security standard or framework and try to apply it elsewhere. That would just be too confusing, I’m willing for you to say, yeah, it would be too confusing, and we can move on.
I. So I actually did take a look at DSS just to see what it was what, was in there. And for the most part, I would say, a good 80% of the standard is probably consistent with just about every other cybersecurity standard that’s out there. So Josh is gonna think it’s a good thing I would do this. But so as the co-chair of the ISA99 committee that I’m at right, in right now in my, my exposure to the standards world. For the 62443 series, I might have also the liaison to a large number of other groups. So I’m actually liaison to the 27,000 series JTC, 1sc27, I’m liaison to the Industrial Internet Consortium, I have liaisons to some of the other, like the actual industrial, Ethernet vendors, vendor associations, there’s a lot of different things that I do. And so I’ve gotten a chance. Also, things like Josh and Scott all know CMMC, I’m actually doing some work with them. I we do work with this deal, EC 2m2, having looked at all those standards, if you look across all of them, including like CSF, and 800-2. And, and things like that.
The bulk of the requirements in those standards are very consistent. They all generally say about the same things. There are some specifics that are different, there’s different wording that’s used in the different standards, there’s different points of view, different guidance, material, but when we actually look at the requirements themselves, they are relatively consistent on asking for things like network segmentation, asking for things like access control, asset inventory, other thing hardening all these kinds of things are fairly consistent. So whether it’s PCI, or others, there is a lot of relationship between them that we can all use. If an organization has a really good program for PCI within its organization already, then one of our recommendations typically is to use what already exists, and adapt the other environment to use as much as you can beg, borrow and steal everything you can from what works in your organization, in the other parts of the organization. So in the ICSOT environment, which usually is much less mature, you can make use of of a large portion of what you’ve got in 27,000 program or something like that in the IT environment.
You know, it’s actually it’s funny, it’s funny that you talk about that because a lot of people don’t look at saying what can you beg, borrow and steal from one compliance regime to another, like, how does PCI? How does it commingle with HIPAA? How does it commingle with ISO? What are the bits and bobs that need to be brought across to fit the organization versus saying the organization must comply with x? You know?
Well, that’s not exactly true, Scott. I mean, it is to a degree but I have lots of customers that, you know, they’re like, Oh, you know, it’s time for the annual audit and we’re asking for evidences and artifacts. And they, it’s very common for them to ask well, we just put together all the evidence for “fill in the blank” audit we just did last month, not to mention the one that we just did three months ago and the other one we did six months ago, because we have to do 17 different audits throughout the year. That, you know, it’s very common to ask, Is there a way to get reuse out of the evidence in the artifacts?
crosswalk them? Yes.
Yeah, I think I think I think people are aware of it from that aspect. I wanted to echo what Jim was saying is, you know, the, the commonality across all the different frameworks and standards is, is very much there. So, to which my responses then, you know, we don’t really need a new framework or a different framework, we need to have the tailored customized “This is how this general generally agreed upon things you need to do to do security in your organization applies to your very unique and special environment and line of business that you’re in”. That’s the cut.
Yes, yes, and no, is sometimes you have to actually account for the people that are reading the standard. So a lot of the ways that we have to write, what we do, and in the 62443 series is very dependent on the readers that we’re trying to reach out. Now. Our series is actually broad, basically applying to everything that is considered sort of an automation system of some sort. And so we’ve actually been having to, like investigate things like IoT, and a lot of different systems, industry 4.0, and things like that recently, that are, have a different sort of world when you get out to actually doing cloud. But for the most part, we’re trying to actually relay the information in a way that makes sense to the people that are actually reading our standard. And how(to) understand how that actually works. And along with that, there’s a lot of tailoring of the guidance and recommendations that go along with it. The first time I wrote six, the two one standard 16 44321, we had about 40 pages of requirements and another 160 pages of guidance along with that. So
I apologize, I forgot that you sort of cut your teeth working at NIST, an organization they often like to scoff at. So I might be stepping on some toes. So apologies.
No, that’s fine. I mean, I helped some of the material that I worked from in the ISO 99 committee ended up basically becoming 82 of the 800-82 standard. And I also did provide input into the NIST CSF when that was getting going. So I do like, I like CSF as a good sort of general-purpose framework. But people try and actually turn it into considering a standard, saying that it’s actually got enough meat to it to be a standard, which I don’t like it. Why not? It’s a, it’s a framework, it’s a thing that you can hang your own requirements off of related to these particular topics. And you can assess against it as a general-purpose thing, but you have to do a lot of tailoring in order to actually come up with relevant questions and relevant sort of assessment criteria for how to actually do an assessment against that.
NIST CSF is a risk management framework, it is a framework where you can understand what things you need to work…
I thought the risk management framework was the risk management framework. I mean, there’s an RMF that NIST has as well.
Yes, as well, yeah, there is CSF, this CSF is a risk management framework, the RMF is also a risk management framework, they are similar, then you need a control framework or control set a set to work on to actually have something to measure against for those categories and subcategories in the CSF.
So which is where all the informative guidance references for that go and point to.
yeah, absolutely. We’re on the same page. Yeah, absolutely agree. It’s okay.
So if you’re not dealing with any of this, where do you start because like demystify all of this for people who don’t understand what we’re talking about.
So one of the big things I would do is if you don’t know where to start, I would highly suggest taking a look at CSF and the categories and subcategories they have in those are good areas to take a look at for trying to at least come up with a general sort of organization to start with. I would also take a look at something like mill one in the DOE C2M2, which basically has a lot of general-purpose, “These are some really simple things to do to start with, in order to build out a program”. Once you can do that and Come up with those at least initial, I hate the term cyber hygiene, things to do within your organization, then you can build on that in order to really improve your cybersecurity maturity. But those NIST CSF and the C2M2 really help you to sort of build-out that – “Here’s what I need to do if I don’t even know where to start” kind of things.
What do you mean C2m2? Are you talking about the….
Cybersecurity Capability Maturity Model, which is C2m2..
It’s the DOE’s version of it.
Yeah. Yeah, I just, um, and we get so used to throwing around acronyms, I just want to ensure that we’re trying to pull out and extract out as much as possible for people who listen to the show, but they may, they might not know what we’re talking about, you know,
There’s also an Australian version of the C2M2, that actually has what is called anti-patterns, which we, when I look at that, that really is one of the good things to add to that Maturity Model, if you’re looking to try and build it out. And what they do with anti-patterns is like, if you’ve done dual home that ends up touching the internet, or if you have an internet connection or something like that, you can’t basically achieve a higher maturity level because you’ve done a bad thing. And so they add these anti-patterns, which sort of negates you moving to a higher maturity. If you’ve done things that they consider sort of, like against standards and practices.
Is there any standard you’re not involved in?
that’s a great question.
When it comes to cybersecurity, for industrial, I tried to dabble in as much as I can, like internally with my company I’m trying to actually build out of risk framework to, or basically a framework to combine everything into something that we can train internally.
So it’s basically taking all of the standards and allowing us to create a single set of assessment questionnaire kind of things and policies and procedures that go along with that, that we can work with customers, instead of having to like train on just “okay, if you’re going to do a C2m2 assessment, you do these set of questions. And if you’re going to do CMMC, you do these, if you’re going to do 62443, you do these set”. I’m trying to actually look across all of them. So I actually spent time really learning as many of the standards as I could, including European and Middle Eastern standards. So the NIS standards over in Europe, the UNISA ones. And then things like, some of the ones in the Middle East, the Japanese are like the Far Eastern, they tend to use the ISO standards, so not as familiar with.
So it stands to reason that your clientele companies you’re engaged with because they’ve engaged with a company like Dragos already, they’re just displaying some sort of degree of maturity, or at least, you know, they’re trying to pursue doing something along the lines of security.
What about all the organizations out there and the various sectors within, let’s say, critical infrastructure, especially the ones that are, you know, publicly-traded small government-led? The ones that don’t have the budget, basically.
How do you how do you begin? Not you Dragos. But you, or we as security professionals, how do we begin to have the conversations with companies that just that, not that they don’t want to necessarily but they’re just so handcuffed into not doing the types of things that we would traditionally think they should be doing?
In at least in terms of the companies that fall under, say the NPPD, the different critical infrastructures, those it would be really good to basically get communication from the government oversight organizations. If we can’t directly reach out to those companies, sometimes it helps to just reach out to the government agencies that oversee them.
There’s different varying appetites for listening to cybersecurity, for those different organizations. Things like the water industry is being overseen by EPA. EPA is much more concerned about water quality and wastewater treatment and things that affect the environment, then they are cybersecurity. It’s not that cybersecurity doesn’t occur to them. It’s that the industries they’re dealing with the companies they’re dealing with don’t always have, like you said the budget to deal with this.
So it’s convincing them that cybersecurity can have implications for those outcomes, those consequences. And to some degree, it’s messaging, it’s getting that messaging out there not having the, sorry to say is it focused cybersecurity for cybersecurity sake, it’s that cybersecurity has reliability and performance issues, on the actual business implications for these companies, and being able to try and convey that in a way that actually makes sense to the organizations that they will listen to you, which is a really hard problem.
There’s so many people that are just trying to find the Blinky box to do the job so they don’t have to do it. And understanding that it’s actually a journey and trying to teach that and trying to get people to understand that they can do a lot with what they already have. But it means that they have to put in time to do it. So.
Wait, I have to work on this stuff. Gosh, man, you’re just vicious. You’re a horrible person, you want me to work on things Come on,
Don’t think, just pitch.
well, and like we talked about there, these people are already doing multiple jobs. And so if they have to spend time working on something else, and learning something that they’re not used to, or acting as shadow IT or something like that. It does actually cause them heartache.
So okay, so now that we’ve got a Wow, so we’ve basically become incredibly depressed. Now, let’s move on to fun things.
Why is ITS and OT always a depressing discuss?
Well, it’s that you know, he’s right. I mean, so Okay, in information security, we deal with having to go for budget to the bosses, right? in compliance, we deal with having to tell the bosses, hey, we have to do these things. And you need to give security more budget, so we can help drive for each other what’s going on, and we can help each other.
But when you start talking about OT, it’s more along the lines of “Hey, we’re a public utility unless our customers say yes, we’re screwed for budget”. And it’s, it’s a problem. So it seems like we should be talking about how do we make the CFOs more understanding of what’s going on.
So that’s why the business aspects of these, Jim, as you mentioned, are so incredibly important. But it’s, it’s, it’s a tough thing, you know, when the CFO looks at you and goes, we haven’t got anything unless you want to quit, and then I can have your salary to buy them a system, you know, which is kind of a self-defeating thing, right? So
Well, can’t they do the PCI like, you know, SAQD and say, Hey, we’re good to go.
Come on, it’s not like people lie on self-assessment surveys. They come on. Anyway. So um,
I didn’t hear what Jim said on well, Scott was while there, what is your comment, Jim?
So um, what I would say is that it’s a matter of, it’s changing the conversation. So having it actually change from the discussion of, like you said, the Blinky boxes and everything like that to the How can we do work? How can we do our job better? How can we produce better products? How can we be more reliable, have more uptime, all these kinds of things, changing that to the business-speak, will help people in the trenches get more traction.
Now, one of the things that I have found quite often is that when I, when we, come into companies, there are a lot of times where they bring us in, they know, 90 plus percent of all the findings that we’re going to actually come across as we do our job. And they just need the third-party validation to get the visibility at the upper levels.
Another thing I would say to do in order to really get people to understand things, is to get the IT people to get the managers actually to walk the floor, walk the line, have them actually go out get put on their PPE (personal protective equipment for people that don’t know), like put on the steel-toed shoes put on the…
Thats a term that we have become familiar with over the past year.
Of all the acronyms that you’ve thrown out this episode.
Do the steel-toed shoes fit over my Gucci loafer?
They actually do have metal caps to go over to go over shoes that if you want them. And I have seen them used in like manufacturing environments where people are in like full suits and everything, or you’ll end up with like ladies in like a skirt suit. And then they’re in high heels, but they’ve got this big, huge metal, like thing that goes over the top of their shoes. So it looks ridiculous. That’s why I’m saying get them actually in like real clothing to walk the line to understand what it is to be in the environment and realize just how dirty and nasty these places can actually be.
So I’ve got two questions really quickly. One is, instead of just walk the line, what would you say to having them work the line for a day, once a quarter actually work the job? Because that’ll tell you what, why don’t we have this? We don’t have it? If you want it buy it for us, you know?
And the second question would be tell us some of the worst things you’ve ever seen.
So, um, so in terms of getting them to actually work the line, I don’t know if that might work. But one thing I will say when I have dealt with actually doing, one of the things that we typically do as we do assessments is we will do walk downs. And when I go into walk-downs, I make a point of trying to actually talk to the operators, talk to the technicians that are working in the field, talk to the actual operations and maintenance techs, the instrument techs, people like that, and find out what they do day to day and talk to them like real people, bring them a box of doughnuts and a box of coffee to the break room and have a sit-down and actually spend a few minutes talking to them like real people, and find out what they do day to day and find out what makes their job more difficult.
People, a lot of people like to really talk about what they do. And they’re excited when someone’s actually sitting there listening to them talk about it. And so they will tell you the things that do and don’t work. So if you have an IT policy that’s set up to say they can’t use USB is to do this, or to move files between things. But you find out that somehow they have to get this report out of this machine over to their other machine in order to actually go and do their daily reporting. But they don’t have a way to communicate between the two, they have to then handwrite it, which makes their job a lot more difficult. So if you can find a secure way to have that communications, to do little, those little things for them, they will actually work for you to help you actually make the systems more secure. Instead of trying to find ways around your system, which may go against your IT policies.
Well, that is so true. And that’s not even ICS. I mean, I’ve had that experience many times over the years in consulting. And you win so much trust and you, when you negate the adversarial nature of the relationship. So often by just a, like you said, acting like you care, I mean caring, like oh, somebody who’s really paying attention, you know, encouraging them that they know their systems better than you’re gonna ever learn. You’re not out to get them and give them a win. That’s so huge.
And a quick kind of off-the-wall question because you mentioned something about, you know, 80 or 90% of the time. You’re not how did you phrase it? You know, they already know what’s wrong, what you say is the third party independent, it’s not news to them. It’s just the validation from the people with the magic capes kind of thing. Which made me think of pen testers like yeah, if only people would believe the pen testers, so you wouldn’t have to spend all the money on the pen test. And then this is my question.
Do you find it hard to find pen test companies that are qualified to work and actually do good jobs in these peculiar non-standard types of environments from a technology perspective?
Absolutely. And you actually companies have to really pay attention and be careful who they do hire to do a pen test within their environment.
So before I joined Dragos, I had done a couple pen tests on environments, but they were only lab systems, I never actually touched production. There are cases where you can touch production in with a pen test, or something like that. But usually, it’s going to be during a, what’s called a turnaround, which is basically a downtime where they actually take the system’s down and do refits and, and make sure it’s these kinds of things like happened once, maybe twice a year, sometimes every other year, or things like that, where they will actually shut down the system stop production, and do what they have to do maintenance wise.
And so sometimes during those kinds of periods, you may get the chance to pen test the production systems, or actually, like actively run network vulnerability scanners or things like that, on the production systems. Most time you’re going to run on a test system, or some sort of like a production dev system or something like that. If you get companies that aren’t experienced with the risks associated with production, coming in, they may have come from an IT and oh, this is it’s all just technology. So it doesn’t matter if it’s OT or IT. They don’t fully understand the risks.
There have been some cases where they’ve taken production down, they’ve done some, had some consequences. I mean, you run the risk of having production consequences anytime you touch a production network. In one of the nuclear sites I was working in one of the systems was one of the network switches was so overloaded with traffic that even when we turned on a spam court, it over it, like it actually shut down the switch because the CPU was so overloaded. And it was such an old switch, that it didn’t have the extra power to handle it.
This wasn’t Chernobyl was it?
No, this wasn’t.
Just checking. Alright, so yeah, our time is winding down. To try to recap what I think we’ve been hearing is, you know, in terms of how do we solve the problems, how do we get past the problems, education, awareness, trying to, you know, get organizations to realize the synergies and the overlaps, try to reduce the duplication of efforts, try to get them to, you know, put “like” things together, so to speak, as best as possible. Work with them win trust, build trust, get them to pay attention, because it’s important. Am I missing anything?
You know, any, any final thoughts, Jim? In terms of Yeah, what can we do on a positive note to try to help this along?
So a lot of really basic things can help to secure OT and a lot of it is just like you said, it’s those interactions, understanding the environment, understanding the risks associated with the environment, learning how to use everything you can for building out an OT cybersecurity program and then doing the basics, right? If you can do the basics, right? That gets you to a level of maturity that you can build upon to really do things do better things. There’s always going to be special cases where things don’t work, right. And you have to figure solutions that can actually work within the bounds that you’ve got to deal with.
Scott, Josh, any final thoughts or questions?
Do the fundamentals come on? What do you listen to the show or something?
Yeah. I mean, I would be bad to not say that monitoring your environment, the assumed breach, because you are dealing with systems that are legacy. Having that assume breach mentality and looking for bad things on your network is probably a good thing at this point, given the fact that there are a lot of different ways into these networks. And so having that assume breach mentality and working to monitor your systems is probably a really good thing too.
No, I just want to say thanks for coming on the show. It’s been an absolute pleasure speaking with you.
Thank you for having me. It’s been really good.
I can certainly really echoed that sentiment. Appreciate you coming on, Jim, and appreciate your contributions to the discussion, most weeks over on the discord server.
That’s going to wrap us today. We will be back next week where I think we will be having a topical discussion as a yet-to-be-named topic. So stay tuned on the social media channels for what we might be talking about. I’ve got – I had an idea, but based on our discussion today, I’ve got another idea. So it’s all up to me, and we’ll see you next week.