ALL THINGS CrowdStrike - July 2024 Incident

[Kaity]

Hi all! There are so many great discussions about CrowdStrike going on in this Community, but we want to bring them together in one place, so that folks can share and discuss efficiently! 

[EchelonVigil]

Just to bring everyone up to speed - I was in the dark myself until I started receiving numerous inquiries about the disruptions in their systems. It appears that an outage instigated by CrowdStrike had a ripple effect, impacting airlines, public transit, and healthcare, among other sectors.

They attributed the incident to a flawed update and assured that there was no malicious activity involved. However, I can't shake off a sense of unease about the whole situation.

I'm eager to know your thoughts on this. What's your take?

[dcontesti]

Ever sit in an airport and see the screens go blank and then hear that MULTIPLE flights are delayed or cancelled?  That's how I spent my morning.......LOL not a fun way to end a work week.

Now that I am back in the hotel and reading all the explanations, https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/ and claims that this was not a Security breach, I call "Bullxxxx"....can't use the word that I want to here.

This affected the Availability of many systems globally.  I really question where their Change Management was....no testing? What happened to the Development/Testing/QA/ Production?  A classic fail on their part.

Many organisations were able to recover Manually but the cost of this outage will probably never be known.

In my case, the airline is providing a hotel room for two nights and two food vouchers.  Imagine that cost multiplied by 100 or better 1000 

At least I got stranded some place fun.

d

[dcontesti]

Thanks to a friend on LinkedIn:

CrowdStrike are a large security company, based in Austin Texas who provide security software to a great many large enterprises. This morning they released an automatic upgrade to one of their programs called Falcon Sensor. Falcon Sensor is a kind of multi-purpose anti-malware, anti-intrusion system. Unfortunately the upgrade this morning had a serious flaw that caused it to crash computers using Microsoft Windows 10.

Falcon Sensor caused what is known as a “kernel panic” state in Windows, which is every bit as bad as it sounds. It immediately caused Windows 10 computers to “blue screen” (crash) and when the computers tried to restart, caused them to crash again. As you might expect, this has caused considerable mayhem.

Luckily CrowdStrike and Microsoft can up with a fix very quickly. Microsoft has a “safe boot” mode that allows you to start a Windows computer in a way that does not load Falcon Sensor. This allows the IT guys to fix the problem without the Falcon Sensor system crashing the computer before they can get to it.

Manually fixing thousands of computers is not going to be fun. Many IT people are currently cursing CrowdStrike for releasing their upgrade on a Friday. Many people’s weekends have been trashed. But that is not the worst of it.

Some versions of Windows, particularly virtual machines running on cloud services like, for example, Amazon Web Services cannot be put into safe mode. This means to be fixed they have to be transferred to a server where they can be fixed then moved back to their original location. This is nowhere near as easy as it sounds. And that is still not the biggest problem.

The problem is that many of the companies that use CrowdStrike are very security conscious. This means their servers are encrypted. To put those servers into safe mode you have to release the encryption using what are known as recovery keys. The problem is, many companies will have their recovery keys stored on servers affected by the problem, leaving them with a potentially difficult to solve Catch 22.

In the next few hours and days we’re going to find out which companies have really thought through their disaster recovery measures. Those who have not are going to be rebuilding a lot of servers from scratch.

Anyway, that’s why your IT guy is very unhappy today (or indulging in a bit of schadenfreude.) The ironic thing is that companies who are worst affected by this are probably the most security conscious, which is why we’re seeing a lot of banks and airlines in trouble.

[EchelonVigil]

You hit the nail on the head of everything I was thinking, thank you.

I'm sorry you had to experience that.

I was asked if I thought this was a cyberattack. My guess, this was either an insider threat or someone was rushing and clicked on something "phishy".

[Caute_cautim]

Hi All,

Well many of you have felt the impact of the CrowdStrike issue and it is massive!

A technical issue, related to a US-based cybersecurity firm named CrowdStrike, caused computers running Microsoft software across Australia and abroad to glitch on Friday.

The global outage impacted a raft of Australian companies and government agencies, causing many computers to attempt to restart and display a blue-screen error message.

Here's what we know so far.

What is CrowdStrike?

CrowdStrike is a US-based American cybersecurity firm that helps companies manage their security in "IT environments" - that is, everything they use an internet connection to access.

Its primary function is to protect companies and stop data breaches, ransomware and cyber attacks.

It includes among its main customers global investment banks, universities and even the Australian betting agency TAB Corp.

The cybersecurity environment has changed rapidly in recent years due to the increased presence of threat actors targeting big business, including Ticketmaster, Medibank and Optus.

As a result, more and more companies are turning towards firms like CrowdStrike to protect their customers' information.

https://www.rnz.co.nz/news/national/521777/new-digital-framework-tackles-trust-issues

Regards

Caute_Cautim

[Caute_cautim]

@dcontesti. A great explanation, by the way it is not a cyber security incident:

CrowdStrike outage shows New Zealand's critical technology dependencies 
The New Zealand Internet Task Force is tonight reminding people that the basics are what keep our online lives safe after the wide scale CrowdStrike outage has impacted services nationally and internationally.
“While there is no indication that there is anything malicious behind the outages that kiwis are experiencing to services tonight, it’s a solid reminder that our lives are firmly intertwined with online services” says Tandi McCarthy, New Zealand Internet Task Force spokesperson.
The outage is the result of an update to CrowdStrike Falcon, security software that protects systems from viruses and other threats. Affected organisations and their IT specialists should make sure that they are connected with Crowdstrike through the formal support channels to receive the correct fix and guidance.
“It’s scary how business as usual changes can take whole systems offline. We are seeing the wide scale and potentially physically harmful effects that a big outage can have. Our peers in Australia and further afield are seeing outages impact healthcare and transport, among other industries.
“There is a fix in place from CrowdStrike, but it’ll take time for organisations to work through and implement it and this will be different for every organisation. People are going to be working really hard, and likely throughout the night and weekend, to get this sorted.”
Incidents like this are a reminder that organisations should understand and document their dependencies on systems and how to get help when something goes wrong.
ENDS
The New Zealand Internet Task Force (NZITF) is a non-profit organisation with the mission of improving the cyber security posture of New Zealand. Our members are IT security professionals who work together through trusted forums to make the Internet safer for all New Zealanders.

Regards

Caute_Cautim

[Caute_cautim]

Hi All

Reminder: While there’s no indication that there is anything malicious behind the outages, this incident is already being exploited for phishing and malware delivery.

Regards

Caute_Cautim

[Caute_cautim]

Hi All

For those thinking about our dependence on key systems as a society in the aftermath of the Crowdstrike deployment failure, can I suggest that you read E.M. Forster's short story, the Machine Stops, written in 1909.

In a future version of planet Earth, most of the human population doesn’t venture above ground. Rarely do they even leave their own rooms, in which all of their needs are met by the Machine.

The Machine allows the humans to communicate “ideas” with one another, which is essentially their only activity. It doesn’t stop them from leaving their rooms, but they have little desire to do so anyway. They’ve started to believe the Machine is omnipotent and omniscient, not to be questioned. And when it begins to malfunction, they trust that it knows what it’s doing—forgetting they invented it in the first place . . .

https://www.amazon.com/Machine-Stops-M-Forster-ebook/dp/B085G4HBRX

Regards

Caute_Cautim

[Caute_cautim]

HI All

On July 19, 2024, an issue present in a single content update for the CrowdStrike Falcon® sensor impacting Windows operating systems was identified, and a fix was deployed.1

CrowdStrike Intelligence has monitored for malicious activity leveraging the event as a lure theme and received reports that threat actors are conducting the following activity:

• Sending phishing emails posing as CrowdStrike support to customers
• Impersonating CrowdStrike staff in phone calls
• Posing as independent researchers, claiming to have evidence the technical issue is linked to a cyberattack and offering remediation insights
• Selling scripts purporting to automate recovery from the content update issue

https://www.crowdstrike.com/blog/falcon-sensor-issue-use-to-target-crowdstrike-customers/

Regards

Caute_Cautim