A recent IT outage caused by the failure of CrowdStrike’s Falcon cybersecurity product has had widespread ramifications for the global business community, affecting the operations of banks, airlines, hospitals, critical infrastructure providers and other businesses worldwide. It serves as an important reminder for companies to be prepared for all kinds of IT incidents to reduce the likelihood of significant business disruption.

What happened?

CrowdStrike’s Falcon software is a product that helps to monitor for malware and other potential cybersecurity threats. Falcon implements frequent, automatic “Rapid Response Content” updates so that the product can constantly evolve to address new cybersecurity threats.

On 19 July a routine update was automatically roled out to approximately 8.5 million devices concurrently.

Unfortunately, this update contained a defect that was not detected during Crowdstrike’s internal validation checks.[¹] When the update was deployed, the defect resulted in Windows system crashes and the “blue screen of death” appearing on impacted devices.

Even though the problem was quickly identified, and most systems were back online after a few hours, many businesses experienced significant and extended disruption, leading to huge financial losses.

The incident is an important reminder that IT product-related issues can be just as disruptive to businesses as large-scale cyberattacks. Businesses should take this opportunity to consider whether their arrangements with IT providers contain appropriate measures to prepare for and mitigate these types of issues and their potential impacts.

Business continuity arrangements

Businesses should review their business continuity plan and arrangements and refresh them if needed to help ensure the business can continue to operate in an event like this, or at least minimise the impacts.  

One useful mechanism is to create a designated crisis team (that includes IT experts) that can step in quickly when a problem occurs and to have an easily accessible communications plan so the team can be quickly mobilised.

Contract protections

Businesses should consider how their contracts for key IT systems address business disruption events and the consequences that flow from them. Some important issues to consider include:

• Software updates and upgrades, including: 

• • providing for staggered deployment of updates to enable parties to verify whether the update works correctly on a smaller number of devices before rolling it out more widely;
  • specifying how updates are implemented, including whether updates are implemented automatically or whether the customer has a right to refuse to implement updates for a period of time. Going forward, CrowdStrike has said it will allow customers more control over when and where Rapid Response Content updates are deployed; 
  • obligations on suppliers to ensure adequate testing of updates before they are rolled out to users; 
  • requirements to rollback to previous versions of the software where updates fail to implement correctly; and 
  • avoiding updates at the start of a weekend or holiday period, unless you have pre-arranged for your IT team to be on-board or on-call to address any unexpected issues.

• BC/DR planning – requirements on the supplier to have in place appropriate business continuity and disaster recovery plans that can be quickly implemented on the occurrence of a business disruption event; 
• Communications during outages – requirements for immediate notification of major incidents and outages, and regular communication during incidents;
• Actions during and after - requirements for the supplier to take actions during the event and post-event, including resolution times and post-incident reviews; 
• Audit – rights to audit the supplier’s compliance with contractual requirements, including BC/DR requirements;  
• Liability – the supplier’s liability for losses resulting from a business disruption event, including what disclaimers, exclusions and liability caps apply and any supplier indemnities; and
• Protection of data – contractual provisions to protect your business’ data – see our earlier article about this here.

Due diligence and audit

As the CrowdStrike case illustrates, the consequences of an IT outage can be very significant. The case is a timely reminder of the importance of undertaking appropriate due diligence on suppliers of key IT systems and conducting regular audits on those IT suppliers, to help provide comfort that your IT suppliers can deliver on their contractual promises.

If you have any questions on anything discussed in this article, please get in touch.

Special thanks to Sam Chaytor-Waddy for his assistance in writing this article.