Choice of utilities for emergency management ...

[rslade]

OK, that's a nice, neutral subject line, isn't it?  I mean, it's not as if I'm bashing anyone for repealing net neutrality or anything.

Speaking of neutral and neutrality, were you worried about it?  Did you wonder how net non-neutrality was going to work out?

Well, ask the guys on the wildfire lines.  Fire officials in Santa Clara county noticed that the cell service based Internet access they used for co-ordination slowed to a crawl.  They contacted Verizon, their provider.

Verizon told them to purchase an upgrade.

Can you say "blackmail"?  (I knew you could.)

(I wonder if there are any Verizon buildings threatened by fire anywhere?)

Verizon acknowledged having made a "mistake."

"In this situation, we should have lifted the speed restriction when our customer reached out to us. This was a customer support mistake."

(Can you say "PR disaster"?  Can you say "Streisand effect"?  I knew you could.)

"In light of our experience," Santa Clara County Fire Chief Anthony Bowden wrote, "County Fire believes it is likely that Verizon will continue to use the exigent nature of public safety emergencies and catastrophic events to coerce public agencies into higher cost plans ultimately paying significantly more for mission critical service — even if that means risking harm to public safety during negotiations."

[CISOScott]

I once worked with a colleague whose nickname was "Flaming Amy". Amy's favorite phrase was " I don't do stupid." Which translates into I am not going to support any idea whose premise is not logical. We were at this national government agency and they had built a multi-million dollar datacenter that was greater than 100 miles out of the "blast zone". They held the ribbon cutting ceremony and all the big-wigs came down to celebrate and cut the ribbon. They had built it according to "all of the best disaster recovery ideas out there" according to the ribbon cutting speech. In front of the gathered crowd she said out loud " You know this datacenter is on the same power grid as the main office. If the main office's grid goes down, so does this datacenter." You could have heard a pin drop. Lot's of red faced big wigs started questioning each other "How could this happen?" "Who didn't do their research?" etc., etc.. 2 million dollars later they had a power junction run to the next power grid over.

When I read your title that's what popped into my mind.

[denbesten]

@CISOScott wrote:

In front of the gathered crowd she said out loud " You know this datacenter is on the same power grid as the main office...."

Cute story.  Thanks for sharing.  I'm surprised that the Data Center manager didn't take the opportunity to direct attention towards the generator building and discuss its capabilities.

It appears that Flaming Amy does do stupid:

1. Blindsiding your chain of management is not a smart career move.
2. Major power failures don't tend to align with the 9 regional interconnects nor the 500 supplier boundaries that comprise the North American power grid, making it very difficult to determine the necessary destination for the second interconnect.
3. Shifting loads tends to cause cascading failures, meaning that their own attempts to regain power are likely to make the problem worse for both themselves and others.  
4. Any decent data center has on-site power generation.

In my case, Utility-power failure is the one "big" thing that we can easily address without any impact  -- at least as long as the credit card allows us to keep purchasing fuel tankers. Generally, I only know if we are on reserve power if operations tells me or if hear the gen-set (which is hard to miss) as I head to the parking lot.  In my case, DR scenarios are for recovering from fire, weather damage, hazmat, crime-scene and backhoe events.

[CISOScott]

Well the data center manager was going on and on about how they could survive the "big one". How they were ready for "anything". Yes, they had generator capabilities. Look at what happened in the US with hurricane Katrina. There was all the help in the world available but no one could get into the area fast enough to replenish the supplies or help the people. Yes the tankers and other trucks loaded with supplies were purchased and lined up but they couldn't get into the area! It was a logistical nightmare. In a true regional disaster you cannot count on the infrastructure to be as you planned it. I have been at 2 agencies where the "perfect storm" happened and the generators did not kick on as planned/expected and they experienced data loss and downtime. In one case they had enough fuel on site but one of the batteries that was supposed to be supporting the detection switchover circuit became corroded and failed, not allowing the generator to turn on. So the generator was ready, but the device to detect when to turn it on failed. In another the generator was fine and fueled up ready to go but no one noticed when they built the generator monitoring/control room that there was a large water pipe running in the overhead drop-down ceiling. When it burst, it destroyed the controls to turn the generators on.

Disaster plans are nice but you can't plan for everything and even when you do, you find out that something as simple as a battery malfunction can ruin your plans. So this person said they had planned for everything and that's when Amy spoke up.

[rslade]

> CISOScott (Contributor III) posted a new reply in Industry News on 08-24-2018

> Well the data center manager was going on and on about how they could survive
> the "big one". How they were ready for "anything".

Recall a story, semi-major airline. CEO in the data centre with the head IT and
BCP guys, they saying prepared for anything. CEO reaches over and pushes Big
Red Button and says, "Prove it."

They weren't. Cost $20 million.

====================== (quote inserted randomly by Pegasus Mailer)
rslade@vcn.bc.ca slade@victoria.tc.ca rslade@computercrime.org
There's no reason to have costly security when costs are easily
offset to the rest of the world in the form of user acceptance of
undetermined risks.
victoria.tc.ca/techrev/rms.htm http://www.infosecbc.org/links
http://blogs.securiteam.com/index.php/archives/author/p1/
http://twitter.com/rslade