cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
lavancini
Newcomer I

Water/Wastewater Sector

Hey everyone, I just started as an IT Security Manager for a water utilities company, which never had a dedicated security section before.  Since this is a brand new position, I have to basically create the entire system from scratch.  I was hoping that I connect with other security professionals who work in the water/wastewater sector or any other utility sector. Thank you,

 

Larry

6 Replies
CISOScott
Community Champion

I'm not in water/sewer but I have come in to places that didn't have IT security before so I will offer this advice.

1) Before you go suggesting changes, understand WHY they do what they are doing. I once was at a place that did not have Anti-virus software installed on their servers. Their reasons? It slows down the servers, we have a micro segmented network and our workers don't do "daily work" on the servers. Well A/V might have slowed the servers down 20-30 years ago but a virus infection usually slows down servers way more than A/V software or does more damage. Just because your network is segmented doesn't mean it can't be traversed. Plus if I've already hacked a workstation and can get to your server without A/V protection, guess where I'm going to store my tools? On your server. Just because your policy is not to do "daily work" on them, I am sure that some people break with policy and do it to "get the job done". I could have said "You Idiots! Not having A/V on your servers is the stupidest thing I've ever seen." but that would not have earned me any friends and would have built up resistance to my security initiatives being accepted as the "new security guy". I simply pointed out that if we were ever breached, senior management and the legal system could very easily understand the optics of having unprotected servers versus understanding the complexities of a mis-configured firewall. They would surely point the finger of blame at the CIO and demand he be fired for being so incompetent, even if there were compensating controls. The optics of it just looked bad. Understanding why they were doing helped formulate my response to it and earned me some friends along the way.

 

2) Try to make friends/allies across multiple departments. One of the ways I did this was to prove my competency in an area that affected those departments. In the past at this workplace, someone in IT would review Internet logs and provide a one page printout to the HR and legal departments as a reason to terminate an employee for Internet misuse. Neither department wanted to, nor could they act on such inefficient evidence of misuse. Having had experience in investigations and going to court I prepared an inch thick summary of their Internet misuse (pornography) and posited why they were a risk to the company. I showed how their Internet use could be traced back to our company, how the company's reputation would be damaged if this information was to be made public, how they were a risk to our IT network as viruses/malware love to hide in pornographic websites, and how the type of pornographic stuff he was looking at (Sadomasochism, bondage, torture, etc.) could be indicative of a situation for potential client abuse (our company helped a lot of single mothers and other vulnerable women). I prepared the evidence as if we would go to court and told both HR and legal depts that I would be willing to go to court if this person sued us after being terminated. Once they saw that I knew my stuff (SME), had their back, and had provided clear and undeniable evidence of risk to the agency, they were able to act. They felt confident in me and had my back in several senior level meetings. Word spread quickly that I was good at my job. I did this with other depts as well.

 

3) Be approachable. Don't be the "security jerk". Look for ways to help people become more secure instead of being a roadblock to what they want to do. Be available to answer questions, even if they seem dumb or simple to you. You want people to be able to come to you with the small stuff so they are not afraid to come to you if something big happens.

 

4) Make a list of everything that needs to done, but don't overwhelm your bosses. When you get this list try to group it together in logical groups (Access control, Policy written or changed, IT architecture, etc.) Then see if you can partner with the people to get things fixed.

 

5) If you need a staff of 5 people but you have none, realize that you may have to do the work of those positions until you can prove clear value/need for those people. I have been able to build my staffing levels at every place I was a leader at by using metrics or other methods to prove the need and then proving my ability to do the work until someone could be hired to do it. I have seen other people fail because they said " I need a staff of 10 people!" but never proved it, just kept blaming stuff not getting done on the lack of resources. 

 

These should be some good starting points to get you started and P.S. Don't be a security jerk!

lavancini
Newcomer I

CISOScott,

 

Thank you for your very informative and well put together reply.  The hardest aspect of this position for me so far is that I am coming from the DoD where change happens quickly and sometimes forcibly.  I am learning the hard way that the water utilities sector moves like frozen water; however, it does change.  You made a lot of great points, especially producing metrics and justifications for adding more personnel.  Right now it is just me, but I do have intentions of requesting 1-2 more additional personnel at a later date.  Until then, what are your thoughts about sharing the wealth among the various IT people in regards to some of the security functions? For example, account auditing.  Being that it is just me, I do not want to get to the point that I am drowning because I consolidated all of the functions into my section, which is just me.  Finally, here is my itemized list of projects that I want to accomplish in order, let me know if it looks good to you:

 

1.  Produce, revise, update policies and procedures.

2.  Produce highly detailed and thorough network diagrams, both virtual and physical (will contract out).

3.  Implement an IT Asset Management program (reviewing programs right now).

4.  Change existing password reset procedures (Right now people just contact the IT section).

5.  Create and implement a Cybersecurity awareness and training program.

6.  Institute a multi-factor login option (Looking at smart card technology).  

7. Implementing some type of data monitoring system (have a request for a quote from Splunk).

8.  Institute a daily log review schedule. 

 

These are just some of the things I identified in the first few days that I have been here. Again thank you for taking the time to reply and I am trying very hard not to be a security jerk, especially since I am a former Criminal Investigator!

 

CISOScott
Community Champion

One of the things I try to do is to think about where I want my department to be next year. I always like to build up my threat hunting capabilities. To do this you will need some security analysts. The ideal situation is to have both Senior and Junior analysts so that you have 1) a career path Jr>Sr>ISSO/CISO and you have built in training potential. Sr teaches Jr how to do it better.

 

Sounds like you are off to a good start. One of the biggest challenges you will face around policy is going from Federal government to this agency. You didn't mention if it was state or local government but usually a service like water/wastewater is done at a local or state level. Could be some federal ones too, but I am guessing more local/state. I had over 25 years in federal government. The big advantage of the feds is ready made documentation and policies, NIST 800-53, etc.. You have frameworks and standards a plenty in federal gov. Not so much on the local/state levels. So coming up with policy is a good idea, but may not be your most pressing issue. 

 

I think you are on the right track with the review of who does what in IT. I would expand your list by finding out what everyone can do in IT. In most agencies/companies I have been with (about 10 so far) I find that the line between IT and security is very blurry with folks operating on both sides of the lines. So find out what security items are being done and by whom. Then you can help lay out your plan to help the company separate them by showing that these security items should really be done by people who do not have the rights to do them (separate the fox from the hen house). I was able to "pry" some of my security staff away from the IT department this way. If you have an audit dept, try to lean on them to do some of the auditing items, if they can. If they deny you, use that as a reason to have more staff. Lean on the Least Privilege Principle. As a former investigator you have probably seen the damage caused by insider threat. Show how having people that are able to make changes, with no one to audit their actions, and not enough people watching all of the items that need tracking could be bad for the company. Look at rights management to see if people have too many rights (i.e. IT made them a global or domain admin because it was easier than trying to figure it out on a granular level.)

 

One of the quickest ways to gain staff is to frame your findings in how it relates to risk to the agency.

1) Not having a policy in place means it is harder to prosecute individuals for wrongdoing.

2) Having people with too many rights raises the risk of insider threat or even incompetent admins deleting whole Active Directory OU's, yes I saw that happen.

3) Once you perform an activity, create Standard Operating Procedures (SOP's) documentation for them. This will help you document what you do in an efficient manner, it will help with policy or procedure creation, and it will prepare you to be able to hand that duty off to a staff member when you get one.

4) If you start getting overwhelmed you can look at this: Here's what I do, here's what I could be doing, and here is where I would like to go in the future. Look at the current list of tasks and prioritize them according to what you see as important. Include the risks for each item. For example; Not dealing with people clicking on links in email is more of a threat to the network than not having an email policy. Having a policy makes it easier for people to understand the expected behavior and prosecute them for violations of it. So maybe you set up a phishing response security awareness course or even just an email explaining phishing and then have a way for people to send in questionable items for review. Then write the policy.

 

One of my early successes at one job was shutting off USB access for 85% of the organization of over 5000 users. They had PII and other stuff being transported around on USB drives and were bringing in viruses on them. Once I proved to the CIO there was a risk to his network from viruses and had also explained the potential "lost" PII risks to him and senior management, they went with my recommendation to block them and have people that had a valid business need submit a request. Then I updated the policies around use of portable media. Then I went on to inappropriate use of company email to reduce/close that source of risk.

 

Also it is going to be key to understand the unspoken organizational culture. If everyone is operating out of a fear mentality, they may be reluctant to do any improvements out of fear of being scolded or punished. If they operate out of an "I don't care" attitude you might have to explain the risks in a way they can understand., etc.

 

And remember, security should be more like a speed bump than a stop sign. Unless they are headed for a cliff.......

Caute_cautim
Community Champion

@CISOScott @lavancini Given the previous responses, I would openly seek to understand their current OT engineers, and how they use such systems and control them.   IoT, and automation of control systems is key to understanding how their control systems communicate, many of which will have long distance monitoring going on via Microwave, UHF and potentially direct line of sight communications using WiFI techniques. 

 

When speaking to the OT engineers, take the tie, off, don the safety boots, get into the one piece engineering and safety helmet, and actively listen, and understand their issues, and how they deal with situations.    You wil gain their respect, and therefore subsequently any decisions will be coordinated and tested with them, before being pushed up to the Governance layer. 

 

Or do they have a mature approach towards control systems, integration of IT systems, or do they keep themselves separate from the normal IT Operations personnel? 

 

Regards

 

Caute_Cautim

lavancini
Newcomer I

CISOScott,

 

I apologize for not replying to your post sooner, got caught up at work with some hot topics.  We are a local water/wastewater service provider which is actually a private organization.  As with your first reply, you really provided me with a lot of useful information, in fact, I will be printing out both replies and adding them to the continuity book I am creating ( I will redact your screen name).  

 

One of my mid-term objectives is to bring on 1-2 analysts in order to perform the continuous monitoring function of the security program; however, because of how we are funded (we have to increase service cost to add new personnel) I am looking at hiring from within.  We are evaluating cloud options for our servers, etc., so if we go that route then some of the IT personnel will be phased out. 

 

Your suggestion on a company-wide email in regards to phishing is an awesome idea, I will propose this option this week to the Director of IT to get his approval and send this out by the end of the week!  Again thank you for giving me great advice and helping me find a direction, which as been kind of difficult since there are so many things I need to do to get security where it needs to be.  

 

Larry

 

 

lavancini
Newcomer I

Caute_Cautim,

 

Thank you for taking the time to respond.  The SCADA team actually has a pretty good security system in-place.  They are very security-minded and aware of the need for security their SCADA/ICS, their not specifically trained in IT security so it needs some work.  The great thing about water is that if the systems go down, then everything still can be performed by personnel.  

 

I am glad that you stated that I needed to "take the tie, off, don the safety boots, get into the one piece engineering and safety helmet" because they were surprised when I asked for my safety boots voucher since no one in IT has asked for them before.  I never thought about that this would gain their respect though, more that I needed to understand their systems better before I submitted policies for approval that would affect their abilities.  Thank you for providing that input.

 

Larry