Microsoft/CrowdStrike Outage: Lessons Learned

The world as we know it continues to rely more and more heavily on digital connectivity in order to function. For the most part, we as consumers do not notice the function of the technology that we deal with day-to-day until something occurs to disrupt it. The global outage on July 19th illustrated the stark reminder that we rely heavily on technology, and the consequences that come when our technology fails. 

Contrary to what many initially thought, this outage was not caused by a cyberattack, but the deployment of a single faulty software update provided by the cybersecurity firm, CrowdStrike. Many of us experienced the ‘blue screen of death’ when we logged into our Microsoft systems on Friday morning; globally, this update had a disastrous impact that affected airlines, media outlets, banks, and retailers that used Microsoft Windows operating systems.

This incident, which may in fact be the largest IT outage in history, serves as a reminder of the extensive connectivity of our IT infrastructure, and the potential for devastating and far-reaching consequences when something goes awry.

Why Were So Many People Affected by the Outage?

The Global Impact of Microsoft Services

With the outage affecting Microsoft services, the outage had a global impact. Office 365, Azure, and Teams are all used in the daily operations of businesses worldwide. Many of us rely on these services daily for collaboration, communication, and cloud computing. Any disruption to these services will inevitably have a massive impact on productivity.

A Growing Dependency on Cloud

With the global transition to cloud computing, the vast majority of business’ data and critical operations are now in the hands of a few service providers. Thus, if a market leader like Microsoft faces an outage, the ripple effect is felt globally.

Highly Interconnected IT Systems

Integrating CrowdStrike’s cybersecurity measures with Microsoft’s infrastructure indicates a greater dependency on interlinked systems. A failure in one can propagate to the other, magnifying the impact. In the case of this outage, Crowdstrike’s update directly impacted Microsoft’s systems causing the disruption of essential services.

Who Suffered Significant Losses from the Outage?

Healthcare

The healthcare industry relies on cloud services to maintain and access communications, patient records, and operational information. Disruptions in these services could result in delays or even cancellations of operations, treatments and appointments, accounting for an estimated loss of millions in the sector.

Financial Institutions

Financial institutions such as banks faced delays in processing trades due to employees being unable to log into their systems. Many financial institutions rely heavily on cloud services for daily transactions and data management, and downtime can cost them millions of dollars.

Retailers

Many retailers use cloud services both for their payment systems and for their e-commerce platforms. Microsoft systems are used for inventory management, point-of-sale, and customer service. This outage affected numerous vital systems, preventing essential operations for many retailers. 

Airlines

Over 2000 flights have been canceled worldwide so far, with passengers facing longer wait-times, delays, and manual check-in processes. Airports in major cities across the globe were affected, with Sydney, London, Seoul, and Washington D.C. being particularly hard-hit. 

Manufacturers

Factories and supply chains use cloud systems for planning, logistics, and operations. The outage could have delayed operations for OT machines more significantly, due to the longer process for patching and manually fixing the faulty update. The incident is likely to have cost hundreds of millions globally.

Many other sectors were significantly affected, and the impact of such an outage was felt across the world. 

Lessons Learned

There are many key takeaways that companies should gather from the IT outage on July 19th:

Rigorous Testing 

Before automatically accepting software updates across an entire organization or in a critical environment, testing the updates in a sandbox environment, or a smaller less-critical environment can be the key to preventing significant downtime. 

Implementing an Effective Patch Management Strategy

Patching during a staged rollout, or rolling out patches in a predefined window of  time can help to create a balance between patch rollout and uptime. Strategic planning, effective communication, and leveraging AI technologies can help ensure smooth updates and reduce disruptions during the patch process. 

Backup & Redundancy Plans

Organizations must have robust backup systems and redundancy plans to maintain operations during IT failures. Ensuring critical systems have redundant backups and performing full system backups before patching are essential for continuity. 

Third-Party Risk Management

Third-party vendors should be regularly assessed and monitored for compliance with security practices. This can help identify and mitigate risks associated with external software and services.

Contingency Planning and Disaster Recovery

In the case of an outage like this one, companies should have a disaster recovery plan in place to quickly and effectively get their systems back up and running. Whether it be an unexpected outage of critical systems like Microsoft services, or a data breach, a disaster recovery plan can improve internal collaboration and communication, and minimize the impact of such an event. 

Wrapping Up

While this incident may not have been easily prevented at the organizational level, it highlights the complexity of maintaining strong IT policies in a globalized, interconnected IT landscape.  This incident should prompt organizations to reassess their IT strategies to enhance resilience against future disruptions. The Microsoft & CrowdStrike outage highlights the vulnerabilities in an increasingly cloud-based world. Understanding the reasons behind the incident, its impact, and what we can learn from it can help us prevent similar issues from occurring in the future.