Cydea’s risk advisories are intended for senior management to aid their understanding of current events and the cyber risk posed to their organisations.
UPDATED 22/07/24 to include additional information on the cause, how the risk may evolve, the remediation steps to take, and other pertinent information for management.
What has happened?
Overnight into Friday 19th July, reports began circulating of mass IT outages worldwide, with airports, rail networks, media outlets and banks all being impacted. Affected organisations reported a “blue screen” error on Microsoft systems, effectively locking users out of Windows-based computers.
Two incidents occurred around the same time. Neither were the result of a cyber-attack:
- Microsoft suffered service degradation to their Microsoft 365 platform caused by a “configuration change” in their Azure backend that caused connection issues between their app servers and data storage;
- CrowdStrike deployed an update to their endpoint security solution that caused some Windows hosts to ‘blue screen’ and become inoperable.
Microsoft’s outage affected some of their Microsoft 365 services, such as Teams, Sharepoint and OneDrive and was largely resolved by mid-morning.
CrowdStrike’s botched update resulted in computers running Microsoft Windows crashing and requiring manual intervention to get them working again, causing prolonged impact. Devices that connected to CrowdStrike for updates between Friday, July 19, 2024 04:09 UTC and Friday, July 19, 2024 05:27 UTC will have received the file causing the problem. Microsoft estimates that around 8.5 million Windows computers were affected.
The endpoint security market is concentrated around a few key players: Microsoft has over 25% market share, CrowdStrike just under 20%, with the rest of the ‘top five’ clocking in around 5% each. Because CrowdStrike has focussed on advanced persistent threats (APTs) and nation state attacks, they have built up a significant enterprise customer base within critical infrastructure sectors. This concentration – of suppliers and their target customers – exacerbated the impact.
This risk advisory focuses on the CrowdStrike incident.
What is the risk?
For individual organisations
Customers of CrowdStrike may be directly impacted. However, there were indirect impacts for companies whose third-party suppliers or service providers used CrowdStrike and were similarly unable to operate.
Source:
Risk events:
- Availability interruption (denial of service)
Consequences:
- Operations (Business disruption)
- Financial (Loss of income, Unplanned response costs)
How may the risk evolve?
Cybercriminal activity
With the outage garnering so much attention it is likely that cybercriminals will try to cash in on the attention. It’s likely that they will impersonate CrowdStrike or their competitors with approaches providing ‘emergency updates’ and requesting users to login or install software.
Insurer and regulator action
We expect that both insurers and regulators will look at this incident with interest. Insurers worry about accumulation risk of ‘systemic threats’, while regulators are increasingly concerned with resilience of their sectors.
Organisations may be contacted by these parties to understand their exposure.
Official information should be obtained from CrowdStrike directly.
For CrowdStrike (and other tech companies with agents)
For CrowdStrike themselves, and if you too are a technology company with an agent-type deployment model, we see this risk as:
Source:
- Accidental (non-malicious)
Risk events:
- Process failure (to duly test an update before distribution)
Consequences:
- Strategic (Embarrassing reporting, damaged reputation)
- Operations (Business disruption)
- Financial (Unplanned costs)
- Compliance (Legal challenge)
What action is required?
This was the result of human error and not a cyber-attack. CrowdStrike promptly identified and fixed the offending update. However IT action is required where customer Windows devices downloaded the update.
It will be immediately obvious if your systems are affected as you will start seeing ‘blue screen’ error messages on your Windows servers and endpoints.
Option 1, from CrowdStrike
We recommend you follow the workaround actions as released by Crowdstrike:
- Boot Windows into Safe Mode or the Windows Recovery Environment
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Locate the file matching “C-00000291*.sys”, and delete it.
- Boot the host normally.
More information on the incident for technology teams, including adviceon recovering BitLocker kieys and restoring cloud environments is available on a special page on the CrowdStrike website.
Option 2, anecdotal reports
CrowdStrike fixed the update promptly (within 90 minutes) and so if affected devices can boot up and connect for long enough they may be able to download the fixed update.
Some users have reported that by repeatedly rebooting affected devices (up to fifteen times) that the devices have successfully downloaded the update and self-corrected.
This method may be useful as a first attempt in circumstances where the affected device is remote from IT support resources.
Longer-term procurement considerations
Competitors of CrowdStrike will highlight this during sales conversations while trying to paint themselves in a more favourable light.
IT, security and procurement teams would be wise to question all endpoint vendors on the controls they have in place to prevent such events from occurring and what their organisations learned from the outage.
For further information or assistance in understanding or measuring this risk to your organisation please contact us for a session with one of our cyber risk consultants.
Cydea uses the Open Information Security Risk Universe (OISRU) as a framework and taxonomy for describing information security risks independently of models or methods of analysing risks. Find out more about our contribution to the project on our cydea.tools site.
