A CrowdStrike update is breaking computers running Windows, causing them to crash and display the blue screen of death. Across industries, companies around the world haven’t been able to reboot, according to reports. Firms affected by the outage include Sky News, which has been unable to broadcast.
Concerned users have taken to forums such as Reddit to report the issue, with one user saying: “Wow, stuck in a boot loop, and entire org taken out.”
So if you got into work this morning and were met by, frankly, carnage, know that you are not alone. Here’s what happened and what to do next.
What Happened
As you might have gathered, an issue with CrowdStrike cybersecurity software is causing the widespread global issue. Engineers at the company said they are working on the issue, which affects its Falcon Sensor product. CrowdStrike calls Falcon “the CrowdStrike platform purpose-built to stop breaches via a unified set of cloud-delivered technologies that prevent all types of attacks—including malware and much more.”
The IT outage has affected airports, businesses and broadcasters, according to the Sky News website. Planes have been grounded in the U.S., trains in the U.K. are impacted, as well as boarding scanners at Edinburgh airport in Scotland.
MORE FOR YOU
Microsoft said it is taking “mitigation actions” after service issues it said started at about 6 p.m. Eastern Time. The company says it is investigating issues with cloud services in the U.S. and “an issue impacting several of its apps and services,” Sky News reported.
“We are aware of a scenario in which customers experience issues with their machines causing a bug check (blue screen) due to a recent CrowdStrike update,” a Microsoft spokesperson said. “We recommend customers to follow guidance provided by CrowdStrike.”
While initial reports focused on a dodgy update, a user named Brody, who is director of CrowdStrike Overwatch, posted on X (formerly Twitter) that it is “a faulty channel file, so not quite an update.”
There is a workaround, he added.
1. Boot Windows into Safe Mode or WRE.
2. Go to C:\Windows\System32\drivers\CrowdStrike
3. Locate and delete file matching "C-00000291*.sys"
4. Boot normally.
At 5:45 a.m. ET, CrowdStrike CEO George Kurtz posted on X, confirming the issue is not a cyberattack and was caused by a botched update.
“CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack,” Kurz wrote, adding that the issue has been “identified, isolated and a fix has been deployed.”
“We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website,” he added. “We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.”
What To Do
It’s not easy to say what to do next. While there is a workaround, it’s not scalable, as it would need to be applied manually, system by system. In a large company, this could could take hours or more to get back up and running.
By its nature, the issue is going to be very hard to resolve once systems are in a reboot loop, says Adam Harrison, managing director at FTI Cybersecurity. “Manual fixes are going to take time for system admins to apply: CrowdStrike can't push a new update remotely to fix. It's going to need manual intervention on each system.”
You might be lucky and be able to roll back to known good states, but the majority won't have anything that supports doing that, says Harrison. “The fix itself is quick to perform, but when you scale that up to thousands of servers and/or thousands of workstations, it's going to be a bad day in the office for lots of folks.”
It’s also going to be a bad day for CrowdStrike. What can the firm do to help people?
“They can only communicate that fix as quickly and widely as they can,” says Harrison. “My assumption would be that the update is already down, so any systems which hadn't updated for any reasons shouldn't still get pushed a bad update.”
Ian Thornton-Trump, CISO at Cyjax, says CrowdStrike “will certainly do their very best to pull the update and instruct the old agents not to update till they can get it sorted.”
However, he says, “what has been done can not be undone for those blue screen machines. If the machines can be booted in safe mode they may be able to issue an out of band update or patch. That’s time consuming—if the machines are critical, they might actually consider restoring from backup or a shadow copy (a built in MSFT recovery feature). Whatever path they have, they will try and fix as quickly as possible.”
CrowdStrike might be able to put a tool together that would apply the fix at the disk level, such as bootable media, says Harrison. “This would maybe help some people out who have thousands of systems to fix. It’s still not a solution that solves the problem fully remotely or at huge scale, but it could bring recovery times down.”
This is a breaking story. Keep your eyes peeled and check back to my Forbes page for updates.
