topic CrowdStrike glitch: What caused the global cyber outage? in Industry News

ALL THINGS CrowdStrike - July 2024 Incident

Kaity — Mon, 22 Jul 2024 15:24:39 GMT

Hi all! There are so many great discussions about CrowdStrike going on in this Community, but we want to bring them together in one place, so that folks can share and discuss efficiently!

More Than an Outage

EchelonVigil — Fri, 19 Jul 2024 21:02:10 GMT

Just to bring everyone up to speed - I was in the dark myself until I started receiving numerous inquiries about the disruptions in their systems. It appears that an outage instigated by CrowdStrike had a ripple effect, impacting airlines, public transit, and healthcare, among other sectors.

They attributed the incident to a flawed update and assured that there was no malicious activity involved. However, I can't shake off a sense of unease about the whole situation.

I'm eager to know your thoughts on this. What's your take?

Re: More Than an Outage

dcontesti — Fri, 19 Jul 2024 22:10:46 GMT

Ever sit in an airport and see the screens go blank and then hear that MULTIPLE flights are delayed or cancelled? That's how I spent my morning.......LOL not a fun way to end a work week.

Now that I am back in the hotel and reading all the explanations, https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/ and claims that this was not a Security breach, I call "Bullxxxx"....can't use the word that I want to here.

This affected the Availability of many systems globally. I really question where their Change Management was....no testing? What happened to the Development/Testing/QA/ Production? A classic fail on their part.

Many organisations were able to recover Manually but the cost of this outage will probably never be known.

In my case, the airline is providing a hotel room for two nights and two food vouchers. Imagine that cost multiplied by 100 or better 1000

At least I got stranded some place fun.

A great write-up (non-technical) on what happened with Crowdstrike this morning (July 19th)

dcontesti — Fri, 19 Jul 2024 22:45:14 GMT

Thanks to a friend on LinkedIn:

CrowdStrike are a large security company, based in Austin Texas who provide security software to a great many large enterprises. This morning they released an automatic upgrade to one of their programs called Falcon Sensor. Falcon Sensor is a kind of multi-purpose anti-malware, anti-intrusion system. Unfortunately the upgrade this morning had a serious flaw that caused it to crash computers using Microsoft Windows 10.

Falcon Sensor caused what is known as a “kernel panic” state in Windows, which is every bit as bad as it sounds. It immediately caused Windows 10 computers to “blue screen” (crash) and when the computers tried to restart, caused them to crash again. As you might expect, this has caused considerable mayhem.

Luckily CrowdStrike and Microsoft can up with a fix very quickly. Microsoft has a “safe boot” mode that allows you to start a Windows computer in a way that does not load Falcon Sensor. This allows the IT guys to fix the problem without the Falcon Sensor system crashing the computer before they can get to it.

Manually fixing thousands of computers is not going to be fun. Many IT people are currently cursing CrowdStrike for releasing their upgrade on a Friday. Many people’s weekends have been trashed. But that is not the worst of it.

Some versions of Windows, particularly virtual machines running on cloud services like, for example, Amazon Web Services cannot be put into safe mode. This means to be fixed they have to be transferred to a server where they can be fixed then moved back to their original location. This is nowhere near as easy as it sounds. And that is still not the biggest problem.

The problem is that many of the companies that use CrowdStrike are very security conscious. This means their servers are encrypted. To put those servers into safe mode you have to release the encryption using what are known as recovery keys. The problem is, many companies will have their recovery keys stored on servers affected by the problem, leaving them with a potentially difficult to solve Catch 22.

In the next few hours and days we’re going to find out which companies have really thought through their disaster recovery measures. Those who have not are going to be rebuilding a lot of servers from scratch.

Anyway, that’s why your IT guy is very unhappy today (or indulging in a bit of schadenfreude.) The ironic thing is that companies who are worst affected by this are probably the most security conscious, which is why we’re seeing a lot of banks and airlines in trouble.

Re: More Than an Outage

EchelonVigil — Fri, 19 Jul 2024 23:27:02 GMT

You hit the nail on the head of everything I was thinking, thank you.

I'm sorry you had to experience that.

I was asked if I thought this was a cyberattack. My guess, this was either an insider threat or someone was rushing and clicked on something "phishy".

CrowdStrike glitch: What caused the global cyber outage?

Caute_cautim — Sat, 20 Jul 2024 00:53:04 GMT

Hi All,

Well many of you have felt the impact of the CrowdStrike issue and it is massive!

A technical issue, related to a US-based cybersecurity firm named CrowdStrike, caused computers running Microsoft software across Australia and abroad to glitch on Friday.

The global outage impacted a raft of Australian companies and government agencies, causing many computers to attempt to restart and display a blue-screen error message.

Here's what we know so far.

What is CrowdStrike?

CrowdStrike is a US-based American cybersecurity firm that helps companies manage their security in "IT environments" - that is, everything they use an internet connection to access.

Its primary function is to protect companies and stop data breaches, ransomware and cyber attacks.

It includes among its main customers global investment banks, universities and even the Australian betting agency TAB Corp.

The cybersecurity environment has changed rapidly in recent years due to the increased presence of threat actors targeting big business, including Ticketmaster, Medibank and Optus.

As a result, more and more companies are turning towards firms like CrowdStrike to protect their customers' information.

https://www.rnz.co.nz/news/national/521777/new-digital-framework-tackles-trust-issues

Regards

Caute_Cautim

Re: A great write-up (non-technical) on what happened with Crowdstrike this morning (July 19th)

Caute_cautim — Sat, 20 Jul 2024 00:58:47 GMT

@dcontesti. A great explanation, by the way it is not a cyber security incident:

CrowdStrike outage shows New Zealand's critical technology dependencies
The New Zealand Internet Task Force is tonight reminding people that the basics are what keep our online lives safe after the wide scale CrowdStrike outage has impacted services nationally and internationally.
“While there is no indication that there is anything malicious behind the outages that kiwis are experiencing to services tonight, it’s a solid reminder that our lives are firmly intertwined with online services” says Tandi McCarthy, New Zealand Internet Task Force spokesperson.
The outage is the result of an update to CrowdStrike Falcon, security software that protects systems from viruses and other threats. Affected organisations and their IT specialists should make sure that they are connected with Crowdstrike through the formal support channels to receive the correct fix and guidance.
“It’s scary how business as usual changes can take whole systems offline. We are seeing the wide scale and potentially physically harmful effects that a big outage can have. Our peers in Australia and further afield are seeing outages impact healthcare and transport, among other industries.
“There is a fix in place from CrowdStrike, but it’ll take time for organisations to work through and implement it and this will be different for every organisation. People are going to be working really hard, and likely throughout the night and weekend, to get this sorted.”
Incidents like this are a reminder that organisations should understand and document their dependencies on systems and how to get help when something goes wrong.
ENDS
The New Zealand Internet Task Force (NZITF) is a non-profit organisation with the mission of improving the cyber security posture of New Zealand. Our members are IT security professionals who work together through trusted forums to make the Internet safer for all New Zealanders.

Regards

Caute_Cautim

Re: A great write-up (non-technical) on what happened with Crowdstrike this morning (July 19th)

Caute_cautim — Sat, 20 Jul 2024 01:00:23 GMT

Hi All

Reminder: While there’s no indication that there is anything malicious behind the outages, this incident is already being exploited for phishing and malware delivery.

Regards

Caute_Cautim

Re: CrowdStrike glitch: What caused the global cyber outage?

Caute_cautim — Sat, 20 Jul 2024 01:40:06 GMT

Hi All

For those thinking about our dependence on key systems as a society in the aftermath of the Crowdstrike deployment failure, can I suggest that you read E.M. Forster's short story, the Machine Stops, written in 1909.

In a future version of planet Earth, most of the human population doesn’t venture above ground. Rarely do they even leave their own rooms, in which all of their needs are met by the Machine.

The Machine allows the humans to communicate “ideas” with one another, which is essentially their only activity. It doesn’t stop them from leaving their rooms, but they have little desire to do so anyway. They’ve started to believe the Machine is omnipotent and omniscient, not to be questioned. And when it begins to malfunction, they trust that it knows what it’s doing—forgetting they invented it in the first place . . .

https://www.amazon.com/Machine-Stops-M-Forster-ebook/dp/B085G4HBRX

Regards

Caute_Cautim

Falcon Sensor Content Issue from July 19, 2024, Likely Used to Target CrowdStrike Customers

Caute_cautim — Sat, 20 Jul 2024 01:46:40 GMT

HI All

On July 19, 2024, an issue present in a single content update for the CrowdStrike Falcon® sensor impacting Windows operating systems was identified, and a fix was deployed.1

CrowdStrike Intelligence has monitored for malicious activity leveraging the event as a lure theme and received reports that threat actors are conducting the following activity:

Sending phishing emails posing as CrowdStrike support to customers
Impersonating CrowdStrike staff in phone calls
Posing as independent researchers, claiming to have evidence the technical issue is linked to a cyberattack and offering remediation insights
Selling scripts purporting to automate recovery from the content update issue

https://www.crowdstrike.com/blog/falcon-sensor-issue-use-to-target-crowdstrike-customers/

Regards

Caute_Cautim

Re: A great write-up (non-technical) on what happened with Crowdstrike this morning (July 19th)

JoePete — Sat, 20 Jul 2024 01:59:16 GMT

@Caute_cautim wrote:
A great explanation, by the way it is not a cyber security incident:

I would prefer wording like "Not a malicious security incident." It is a cybersecurity incident; checks, balances, and testing that should have been in place weren't there to guard against a catastrophic failure.

Re: More Than an Outage

JoePete — Sat, 20 Jul 2024 02:07:30 GMT

@dcontesti wrote:
This affected the Availability of many systems globally. I really question where their Change Management was....no testing? What happened to the Development/Testing/QA/ Production? A classic fail on their part.

Absolutely. It may not have been malicious, but it most certainly is a security incident. Otherwise, 90 percent of the CVE "isn't a security incident."

Part of this is the hazard of uniformity in systems, but this also highlights the hazard of supply-chain attacks. Think about it; you have all these Windows systems bricked by a third-party application. Isn't this what everyone is squealing about these days with "zero trust?"

Re: A great write-up (non-technical) on what happened with Crowdstrike this morning (July 19th)

dcontesti — Sat, 20 Jul 2024 09:19:31 GMT

@Caute_cautim @JoePete

So my take, while maybe not malicious, I tend to think of this in terms of a Security Breach. The basis for most things we do in Security is CIA, although of late, Integrity and Availability are not discussed as much as Confidentiality. Also when computers start blue screening for no apparent reason, Security always get tapped to work on the issue. From past experiences, most people think that seeing the Blue Screen indicates they have been hacked.

I believe this is a colossal failure in their (CrowdStrike) Change Management processes (development/testing/QA and finally deployment).

I also question the contracts in place with CrowdStrike that allows them to automatically push a patch/fix to systems without (so it seems) proper notification. Not sure about others but we do not even let M$ push patches without our knowledge. Whilst they may be downloaded automatically, they are not deployed until they are tested in house.

Total failure; Management 101 gets an F- for CrowdStrike. To those that have cntracts with CrowdStrike also an F-.

Re: CrowdStrike glitch: What caused the global cyber outage?

Kyaw_Myo_Oo — Sat, 20 Jul 2024 07:10:11 GMT

Thanks for sharing this information with us @Caute_cautim.

Another outage caused by CrowdStrike (no one noticed)

dcontesti — Sat, 20 Jul 2024 23:53:52 GMT

CrowdStrike took down Debian and Rocky Linux a few months ago and no one noticed

Despite being a leading cybersecurity firm, CrowdStrike’s approach to pushing updates without extensive testing across all configurations is troubling.

https://stackdiary.com/crowdstrike-took-down-debian-and-rocky-linux-a-few-months-ago-and-no-one-noticed/

I agree with the author 100%, total failure on CrowdStrike's part in Management 101.

Birdstrike - How Crowdstrike smashed much of our IT infrastructure.

Caute_cautim — Sun, 21 Jul 2024 03:36:41 GMT

Hi All

Ah, birds and windows… avians and glass panels… locked in an eternal battle that, to be fair, the window wins 99% of the time. Not in this instance though.

Silly metaphor aside, this recent Crowdstrike vs. Windows debacle is serious, in fact I’d say that this is the incident Y2K wishes it was.

Want to know what happened? Grab yourself a cup of tea and read on.

https://www.linkedin.com/pulse/birdstrike-how-crowdstrike-smashed-much-our-liam-sutton-fczkc/?trackingId=ILqAFgOyZqu6jTNNttU2BQ%3D%3D

Regards

Caute_Cautim

Re: Another outage caused by CrowdStrike (no one noticed)

Caute_cautim — Sun, 21 Jul 2024 03:39:26 GMT

@dcontestiThe mainframe users are glad they were not affected at all....

Regards

Caute_Cautim

Breaking! Hackers Started Exploiting CrowdStrike Issue in Cyber Attacks - Beware

dcontesti — Sun, 21 Jul 2024 07:56:01 GMT

Cybersecurity experts have uncovered a concerning development following the recent CrowdStrike Falcon sensor issue that affected Windows systems on July 19, 2024. Threat actors are now actively exploiting this incident to target CrowdStrike customers through various malicious activities.

https://www.linkedin.com/pulse/breaking-hackers-started-exploiting-crowdstrike-issue-ie2ec/?trackingId=4kOMsknazEHDymL8Oo%2BQhQ%3D%3D

Well we knew it was only a matter of time...............

Re: Another outage caused by CrowdStrike (no one noticed)

dcontesti — Sun, 21 Jul 2024 07:58:26 GMT

Smiles. When I first started at my company (too many years ago)......the mainframe was going away.............Its still there and working.

Re: A great write-up (non-technical) on what happened with Crowdstrike this morning (July 19th)

JMan1 — Sun, 21 Jul 2024 17:22:37 GMT

This, is definitely a security incident. It violates, one of the core security principles… Availability.