ALL THINGS CrowdStrike - July 2024 Incident

[Kaity]

Hi all! There are so many great discussions about CrowdStrike going on in this Community, but we want to bring them together in one place, so that folks can share and discuss efficiently! 

[ericgeater]

Yay!  So there were three big stories this weekend:

• something something Biden
• Crowdstrike
• many, many, many news sources chose to run a highly erroneous story about Southwest's infrastructure

(found the link: Delta Air Lines struggles to recover from global IT outage | AP News)

[Caute_cautim]

Hi All

Ask yourselves why was China not impacted?

• The global IT outage on Friday was caused by a single update to CrowdStrike software that runs on Microsoft Windows.
• Far fewer businesses in Asia use CrowdStrike services, meaning the consequences there were minimal.
• Chinese cybersecurity firms say the CrowdStrike outage highlights the superiority of their products over US technology.

Who do you believe?

https://www.abc.net.au/news/2024-07-23/why-asia-was-less-crippled-by-the-crowdstrike-outage/10412659...

Regards

Caute_Cautim

 

[AndreaMoore]

After the CrowdStrike Outage: What Can We Learn?

The CrowdStrike software update issue highlighted the importance of robust software testing and the software development lifecycle, domain elements of the CSSLP certification. Following on from our look at dealing with IT outages, we consider how this incident can serve as a learning opportunity for those producing and deploying software.

Read more in ISC2 Insights: https://www.isc2.org/Insights/2024/07/After-the-CrowdStrike-Outage-What-Can-We-Learn

Keep the conversation going here in this thread. How did this impact you/your organization? What are the lessons you've learned? 

[dcontesti]

@Caute_cautim  posted from an article:

• Chinese cybersecurity firms say the CrowdStrike outage highlights the superiority of their products over US technology.

I wish I could call Bull dodo here but the news will print anything.

Any cybersecurity firm COULD and MAY still cause the same issue unless their Change Management processes are 1000% effective.

Again this was a failure on CrowdStrike's part in their Change processes.  Did they not follow them?  Were they flawed?  Was someone too tired and let something fly.

Even we folks in Security understand the process (Development/ Testing / QA  /Deployment.....end of story.

It will affect the Availability of systems as well as a Security attack.

@AndreaMoore is correct, folks need to look at the CSSLP and the domains that governed this.

And the folks in China need to understand the difference between a Security incident and a Change incident.

-----okay off soap box and yes finally home.............

d

[ericgeater]

From the article @Caute_cautim shared: "Only international hotel chains and other foreign businesses in China reported major issues."  A little further down: "CrowdStrike's customers are mainly in Western countries"

I would wager that, in the realm of cybersecurity for Chinese-borne companies, there's probably a distinct preference for Chinese security vendors instead of protection over an American company.  And from what I've read about Crowdstrike SLAs which were shared to Twitter, it seems that they're not inclined to provide guarantees, and limit their liability in spite of the protections they're providing. 

Occam's Razor applies here.  Asia was not affected because they use Windows 3.1 because they don't use Crowdstrike products.

[dcontesti]

Okay guys, I am hoping this is also "Fake" but .......a $10 card?

https://techcrunch.com/2024/07/24/crowdstrike-offers-a-10-apology-gift-card-to-say-sorry-for-outage/

Where's mine?  I got stranded  (okay the airline did give me stuff but only because I am a Top-tier Member....in other words I fly too much) and they could not get me anywhere near my home for two days.  The guy behind me got zreo, zip, zilch because he was a normal Joe.

Someone please tell they did not just do this.

[ericgeater]

It's funny, @dcontesti, that some folks received cards, while Kevin Benacci of Crowdstrike sent me an email, asking me to buy some Uber Eats gift cards, then send them to him.

I think I'm getting the short end of the stick on this deal.

</sarcasm>

[dcontesti]

@ericgeater  LOL

[Caute_cautim]

Hi All

A CrowdStrike software update that crashed computers globally last week hitting services from aviation to banking and healthcare was caused by a bug in the U.S. cybersecurity firm’s quality control mechanism, the company said on Wednesday.

Friday’s outage happened because CrowdStrike’s Falcon Sensor, an advanced platform that protects systems from malicious software and hackers, contained a fault that forced computers running Microsoft

’s  Windows operating system to crash and show the “Blue Screen of Death.”

“Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data,” CrowdStrike said in a statement, referring to the failure of an internal quality control mechanism that allowed the problematic data to slip through the company’s own safety checks.

https://www.cnbc.com/2024/07/24/crowdstrike-says-bug-in-quality-control-process-led-to-botched-updat...

Regards

Caute_Cautim

[denbesten]

It sounds like they have a decent release process.... 

The sensor release process ... staged sensor rollout process that starts with dogfooding internally at CrowdStrike, followed by early adopters. It is then made generally available to customers. [cite]

---

BUT, they do not follow it everywhere....

[Rapid Response Content updates] ....Template Validator..... performs validation checks on the content before it is published   [cite]

---

The good news is that their lightbulb is starting to flicker on here...

Refined Deployment Strategy
● Adopt a staggered deployment strategy, starting with a canary deployment to a small
subset of systems before a further staged rollout.
● Enhance monitoring of sensor and system performance during the staggered content
deployment to identify and mitigate issues promptly.
● Provide customers with greater control over the delivery of Rapid Response Content
updates by allowing granular selection of when and where these updates are deployed.
● Provide notifications of content updates and timing.

[cite]

---