CrowdStrike-like outages will turn into an epidemic and become more frequent. Here's why

In recent months, the digital world has been rocked by significant IT outages, with one of the most notable incidents tied to CrowdStrike’s software update.

However, this incident is just one part of a larger pattern of failures that have disrupted major platforms like Google, X (formerly Twitter), and other essential services. These widespread disruptions have triggered a critical conversation about the vulnerabilities and challenges inherent in our increasingly interconnected digital infrastructure.

The CrowdStrike Conundrum: What Went Wrong?
CrowdStrike, a leading provider of endpoint protection, found itself at the epicentre of a crisis when an update pushed to all its users simultaneously caused widespread disruptions. This incident disrupted one in four Fortune 500 companies and resulted in significant financial losses, highlighting a series of missteps that have left many questioning CrowdStrike’s approach.

The crux of the problem lies in the decision to implement a mass update without a staged rollout. In a staged rollout, updates are deployed in small increments, allowing for the detection of issues before they can escalate.

By updating only a small group initially, any catastrophic failures can be identified and contained, preventing a broader impact. However, CrowdStrike opted for an all-at-once approach, which meant that by the time the issue was noticed, it was too late to mitigate the damage.

Lack of Thorough Testing
Compounding the problem was an apparent lack of thorough pre-release testing. Effective testing requires simulating a wide range of hardware, software configurations, and user requirements to catch potential issues.

In this case, the update caused a 100 per cent failure rate, rendering systems inoperable until a manual fix could be applied. Such a catastrophic outcome indicates that the update was grossly under-tested. Given the critical nature of cybersecurity updates, it’s imperative that any new release undergoes rigorous testing to ensure it does not disrupt the very systems it is designed to protect.

CrowdStrike’s primary service involves providing “endpoint protection,” a comprehensive defence against malware and other cyber threats for corporate clients.

Unlike consumer antivirus software, CrowdStrike’s solutions are tailored to safeguard large networks of corporate devices, preventing them from becoming entry points for broader network attacks.

This service includes daily updates to address the latest threats, a necessity in a landscape where new vulnerabilities emerge constantly. However, the speed at which these updates are deployed can sometimes come at the cost of thorough testing and controlled rollout.

Updates in the fast lane
In cybersecurity, speed is often crucial. The rapid spread of ransomware like WannaCry and NotPetya highlighted the devastating potential of unchecked malware. These incidents underscore the need for quick updates to neutralize threats before they can cause widespread damage.

CrowdStrike operates on this principle, pushing frequent updates to ensure its clients remain protected. However, this urgency should not overshadow the necessity of ensuring these updates do not introduce new vulnerabilities or operational issues.

CrowdStrike’s incident was not an isolated event. Around the same time, Google experienced significant outages affecting its suite of services, from Gmail to Google Drive, causing widespread inconvenience and disruptions in business operations.

Similarly, X (formerly Twitter) faced an outage that left millions unable to access the platform, raising concerns about the stability of these major digital services.

These outages, though caused by different underlying issues, share common themes: the challenges of maintaining stability in complex systems and the consequences of rapid, sometimes insufficiently tested updates. Google’s outage was linked to a misconfiguration in its network infrastructure, while X’s was reportedly due to a failed server upgrade.

Both incidents underscore the delicate balance between innovation, speed, and reliability in maintaining global digital services.

The fallout
The fallout from these incidents has been significant. For CrowdStrike, the update aimed to enhance the system’s ability to detect specific cyber-attacks. Instead, it introduced a logic error that caused operating system crashes.

While the exact technical details remain unclear, the outcome was a severe disruption across numerous systems. CrowdStrike’s failure to maintain system availability—a core component of information security—compromised the operational integrity of its clients.

For Google and X, the outages sparked a reevaluation of their infrastructure resilience and update protocols. Users expressed frustration, and businesses relying on these platforms for critical operations faced significant setbacks. The financial impact, though not fully quantified, is expected to be substantial, adding pressure on these companies to prevent future occurrences.

Lessons Learned and Future Directions

Several theories have emerged in the aftermath, from the concentration of power in a few tech companies to regulatory failures. However, none fully encapsulate the core issue: the balance between rapid deployment and system stability. CrowdStrike’s incident, along with the outages at Google and X, serve as reminders that even in a fast-paced digital environment, there must be robust mechanisms to ensure updates do not cause more harm than the threats they aim to neutralize.

Moving forward, the takeaway from these debacles is the inevitability of such incidents in a highly interconnected and technologically dependent world. As IT systems grow more complex and the threats they face become more sophisticated, the potential for unexpected failures increases.

Companies like CrowdStrike, Google, and X must refine their processes, perhaps incorporating more rigorous testing and staged rollouts, to safeguard against future disruptions. The incidents also highlight the need for preparedness in handling such crises, ensuring that recovery is swift and comprehensive.

The recent global IT outages underscore a critical balance in digital services: the need for rapid response to threats must be carefully weighed against the imperative to maintain system stability and reliability. As the digital landscape continues to evolve, the lessons learned from these incidents will be crucial in shaping the future of IT management and cybersecurity.