The Aftereffects of the CrowdStrike Outage (2024)

On Friday, July 19th, 2024, CrowdStrike had a major outage. Many businesses and organizations came to an abrupt halt as all systems using CrowdStrike received the infamous blue screen of death. All because of one faulty update, mass amounts of flights were delayed and canceled, businesses such as banks or news broadcasts were forced to temporarily close, and most importantly, vital operations such as government agencies, emergency services, and healthcare organizations were also impacted. After such a major and widespread incident, changes need to be made.

We reached out to our incredibly brilliant Healthcare IT Today Community to ask them several questions about what those changes should be: What are the biggest lessons learned that healthcare organizations should apply from the CrowdStrike outage? What best practices should healthcare providers adopt to secure their systems against similar business continuity issues in the future? What changes will this outage spur from vendors and from healthcare organizations? What other downtime issues should we be working on to prevent a similar widespread event across healthcare? and What places do you think healthcare is vulnerable to downtime? The following are their answers.

Richard Bird, Chief Security Officer at Traceable AI
The CrowdStrike event serves as an incredibly important reminder that the digital world is nothing more than a tool that serves the human world. The ultimate receiver and victim of bad supply chain management and security is the consumer. How many lives were negatively impacted, and how many unnecessary and non-trivial tragedies and dramas were caused by this outage? Corporations need to remember that technology exists to serve human beings.

Also, Microsoft’s archaic directory structure and architecture are as much to blame for this global outage as CrowdStrike’s actions were. Technologists have been fighting the negative consequence of this folder-based system for decades and Microsoft has done little to mitigate the risks that one day that structure would result in an incident just like this. CrowdStrike will take the heat but Microsoft started the fire years ago. No one is isolated from the bad choices and sins of the past, not even technology companies.

Mehdi Daoudi, Founder and CEO at Catchpoint Systems, Inc.
Consider this historic IT outage a warning for the fragility of third-party dependencies. We’ve seen major disruptions — surgeries postponed, flights grounded, 911 services inaccessible — because of such dependencies. A healthcare data scientist told me that he’d never heard of CrowdStrike before Friday. But now, this will be the one thing he remembers them by as multiple surgeons and patients alike couldn’t confirm their appointments and his company was caught in the fray.

So, what lies ahead? Look to SolarWinds. The fallout from their 2020 breach was catastrophic – as it will be for CrowdStrike – U.S. regulators sued for fraud, citing neglected cybersecurity and severe vulnerabilities. The SolarWinds breach infiltrated U.S. government agencies and over 100 private companies. Moreover, the company faced relentless scrutiny, and CrowdStrike should brace for the same. As a security company, they must go the extra mile to prove their reliability. CrowdStrike needs to prepare for rigorous examination, rebuild trust, and ensure resilience moving forward.

Comment from CrowdStrike: Channel File 291 was not a breach or neglected cybersecurity, it was an outage caused by a logic error in a routine update.

Anand Naik, Co-Founder and CEO at SEQURETEK
Healthcare organizations primarily focus on adopting a multi-faceted system security and continuity approach. Key best practices include implementing robust and frequent backup processes vital for safeguarding data integrity. Testing and sandbox environments are essential before the full-scale deployment of any updates, ensuring that issues are identified and addressed early. Additionally, strict internal testing policies for updates should be enforced to prevent unforeseen disruptions. Real-time monitoring tools are indispensable for detecting and responding to issues promptly. Healthcare’s vulnerability to downtime spans several critical areas. Electronic Health Records (EHR) systems are central to patient care, and their downtime can have severe repercussions on treatment and operational efficiency. Network-connected medical devices are crucial for continuous patient monitoring, making them a significant point of vulnerability.

The increasing reliance on telehealth underscores the need for stable connectivity to avoid disrupting patient consultations. Pharmacy systems are another critical area where downtime can impede medication dispensing and prescription management. Laboratory Information Systems are essential for timely test results and any delays can affect critical diagnostics. Additionally, disruptions in supply chain systems can impact the availability of medical supplies, while failures in internal communication systems can hinder emergency and routine care coordination.

Jonathan Burk, Software Engineering Director at Full Spectrum
This is a good example of how the concept of diversity of defense can protect organizations from a single security vendor having an outsize impact on operations. Diversity of Defense suggests that organizations should employ multiple security vendors to ensure that an issue with one vendor does not impact all systems. This does not eliminate the kind of outage we saw with CrowdStrike, but it does reduce the cost of recovery by reducing the footprint. Along with using multiple security vendors, healthcare organizations should increase redundancy in critical systems. Backup systems protected by an alternate vendor would likely remain unaffected during this kind of outage.

Every healthcare organization has a disaster recovery plan already. But when these plans are not sufficiently tested, recovery can be time-consuming and costly. Healthcare IT should conduct regular disaster recovery drills to ensure they are able to switch to redundant systems and restore data in line with recovery time objectives.

Security vendors are tasked with balancing speed with quality. Security updates are often urgent, and delayed distribution could leave customers exposed to critical vulnerabilities. And while the exact root cause analysis of the CrowdStrike outage is not yet available (Note: CrowdStrike did release the Root Cause Analysis on August 6, 2024 after this quote was collected and before this article published), this incident serves as an example of how inadequate testing and controls could have serious consequences. Many healthcare organizations will need to rethink their approach to “evergreen” software and automatic updates. This, too, is a balancing act. Testing patches is time-consuming and slows the adoption of critical updates. However, testing in a non-production environment ensures the safety of an update when pushed to critical systems.

Many healthcare organizations have focused on threats that directly target the organization, such as the wave of ransomware attacks that have made headlines repeatedly. The CrowdStrike outage is a reminder that reliance on external partners – even highly reputable experts – is a potentially weak link in the chain. Disaster recovery plans need to include not just the organization’s own resources, but mission-critical third-party services as well. Both security vendors and IT departments will take steps to avoid another CrowdStrike outage. However, all organizations must keep in mind that IT outages are a fact of life, and it is impossible to say what the nature of the next serious incident will be. A forward-thinking organization embraces the fact that systems will fail, and builds recovery mechanisms, monitoring, and testing to minimize the impact of such failures.

Kevin Heineman, Chief Information Security Officer at Lyric
Organizations are more dependent on technology than ever before, and the technology is ever-increasing in complexity and the number of integration points. This has elevated the risk that a failure in a critical part of the infrastructure will result in an outage.

Vendors will need to assess their technology infrastructure from both a risk and threat perspective to determine potential single points of failure and the risk these create.

I believe the interconnectivity that supply chain risk brings to both individual organizations, as well as the industry as a whole creates new challenges for healthcare organizations. Business Continuity plans should evolve to consider how this risk can impact both the organization and the industry.

Jim Ducharme, CTO at ClearDATA
Business resiliency can be impacted by many things. We typically think of outages being caused by malicious attackers finding ways to penetrate the many security layers we have put into place to keep our applications and data secure and compliant. But in this case, the very technology meant to help keep us safe and secure caused a significant resiliency problem. Certainly, this wasn’t anything malicious but much like the old data centers and someone tripping over a power cord and causing an outage the inadvertent cutting of fiber optic cables in the Red Sea back in March of 2024, just about anything imaginable can have an impact on the resiliency of our business.

There are many lessons here even for those fortunate enough to not be impacted by this event or the vendor at the center of it. First, for everyone, the importance of having business continuity and disaster recovery (BCDR) plans is essential, and that when incidents like this occur we test our BCDR plans against these scenarios, whether we were impacted or not, to understand any risk exposure and react accordingly. Second, for those deploying software or infrastructure, resilient change management processes are also essential. From the “Mr. Obvious” department, quality control is of course important but once again there is always the unintended impact of change that we can’t always anticipate or replicate in a testing lab. Companies should look at the maturity of their change processes and for those critical systems implement progressive rollout strategies coupled with tight monitoring of the infrastructure to methodically introduce change into their environment and watch for any adverse reaction. These processes may not prevent issues from happening but can certainly minimize their impact.

Per the above, ensuring they have, and regularly test their BCDR plans against these real-world instances when they occur as well as have progressive rollout strategies for any change into critical environments or infrastructure to help minimize the impact when change introduces unintended consequences.

Beyond implementing these best practices, I think in certain industries we will see heightened regulatory requirements around resiliency of infrastructure. Healthcare organizations have already had plenty of advanced notice around issues like this due to the prominence of ransomware attacks that hold their systems hostage and stress even the most rigorous BCDR plans. Vendors will be under even more scrutiny around not only their security posture but now in their resiliency posture and change management processes. I’m sure third-party risk evaluations will now contain more requests for information on how changes to their services are rolled out to their customers.

Healthcare organizations, and really any business for that matter, should take a look at their business processes and the dependencies on IT infrastructure to execute them successfully. Something as simple as a scheduling application can shut down a business. In the case of Delta Airlines in this incident, the system responsible for contacting their flight crews was impacted so their ability to connect with their employees and address contingency plans was impacted because they couldn’t even effectively communicate with their employees to change their operating plans. Elective surgeries in many hospitals had to be canceled, not because the operating room wasn’t ready but because scheduling systems and EHR systems were not available. Many times there are systems we don’t think of as “business critical” when looked at in isolation but when looked at from its dependency on key business processes or even in the context of BCDR suddenly these seemingly simple systems become business critical.

Lathe Bigler, GM at FDB Vela
The recent CrowdStrike/Windows chaos that knocked out systems for transportation, delivery, and healthcare brings to the forefront the importance of redundancy for high-use healthcare technology—especially with it occurring on the heels of the recent Change Healthcare crisis. Redundancy—which is a system design in which a component is intentionally duplicated so there is backup during a failure—is crucial for patient access and safety.

Take, for example, ePrescribing: EHRs, pharmacies, PBMs, health systems, and technology solutions that rely on ePrescribing networks need to ensure that they have backup if their primary network goes down completely or partially. That way, they can still access eligibility and medication history data needed to write prescriptions and route them electronically to the appropriate pharmacy. Lack of access to prescription information when needed can range from a minor inconvenience to a life-altering or life-threatening scenario. Forewarned is forearmed: With only two ePrescribing networks available in the United States, healthcare executives have a simple decision to make, as we all hope for the best but prepare for the worst.

Kiran Chinnagangannagari, Co-Founder, CPO, and CTO at Securin
The CrowdStrike outage serves as a wake-up call for healthcare organizations to prioritize resiliency and preparedness. Healthcare providers should walk away with multiple lessons from this outage, but the one that should be the key focus is resiliency. Have a handy inventory of critical systems, processes, and personnel to deal with an unplanned event to minimize downtime. Also, the BitLocker issue emphasizes the need for readily accessible and thoroughly tested recovery methods, especially for critical systems.

The CrowdStrike outage also highlights the importance of testing software before deploying it on all systems at once – techniques like canary deployments, where updates are rolled out to a small group first, could have minimized the impact. Healthcare providers should invest in redundant systems and reliable backup solutions to ensure quick recovery in case of outages, having clear communication plans to keep staff, patients, and stakeholders informed during outages is also paramount. Despite the disruption, regular updates should remain essential tech practices as they often contain crucial security patches that protect against cyber threats. Not only do these recovery plans help restore systems to an operable state prior to a faulty update pushed by a software provider, but they also help protect against cyber threats. Without these updates, healthcare systems become significantly more vulnerable to attacks from cybercriminals, risking data breaches and patient safety.

This outage may be a catalyst for significant changes and potentially accelerate the adoption of “secure by design products” principles. Healthcare organizations will likely increase scrutiny of vendors, demanding better security and testing. Organizations will be a lot more demanding of more rigorous testing procedures and written warranties to protect themselves financially in case of outages. Vendors, on the other hand, might invest in more robust testing infrastructures and explore alternative deployment models to minimize downtime risks.

Looking beyond this specific incident, healthcare faces various downtime threats, particularly from cyberattacks like ransomware or data breaches that can disrupt normal operations and compromise patient care. Medical devices and equipment are increasingly vulnerable, requiring proactive cybersecurity measures such as routine penetration testing, vulnerability patching, and adopting security by design principles. A Securin security report with Finite State and Health-ISAC found there was a sharp 59% increase in vulnerabilities from 2022 across medical hardware, operating systems, and software applications. Many medical devices (such as infusion pumps, pacemakers, and monitoring systems) rely on software applications, and a cyberattack can lead to delayed treatments or compromised functionality, potentially endangering patients’ lives. Many of these devices, which we fail to recognize as potential targets for hacking, are at the same risk as our computers and networks.

Scott Stuewe, President and CEO at DirectTrust
Last Friday, airlines, banks, and businesses of all kinds suffered an outage that affected 8.5 million Microsoft Windows Computers after an update from Microsoft partner CrowdStrike caused a critical error resulting in the dreaded “blue screen of death” on machines worldwide. Since the fix sometimes requires being able to be at the keyboards of these machines, it is expected that full recovery may be weeks to months away. While the CrowdStrike incident wasn’t an attack by malicious actors, it tells us a lot about the importance of third-party risk management and what happens when a company, or even when society, relies on relatively few companies to supply technology to businesses.

For Microsoft, it’s the wake-up call they need to hold partners like CrowdStrike to higher standards. For all the players who experienced outages, the message is the core “zero trust” mantra — set up processes and procedures to test changes to infrastructure that are initiated from outside your walls. Trust nothing without verification. Develop contingency plans including well-documented manual workarounds and disaster recovery scenarios. Make sure to consider utilizing disparate technology stacks when constructing backup approaches. For society, we need to ask if the technology market has become so consolidated that all the primary players are too material to fail. If the free market is to address this issue, there needs to be more open competition. Otherwise, we will have to depend upon the government to regulate these highly consolidated industries.

Comment from CrowdStrike: As mentioned in the RCA, CrowdStrike has engaged two independent third-party software security vendors to conduct further review of the Falcon sensor code and end-to-end quality control and release processes. This work has begun and will be ongoing as part of our focus on security and resilience by design.

Katie Paxton-Fear, Security Researcher at Traceable AI
This entire situation really demonstrates just how interconnected global IT systems really are. CrowdStrike is considered the best money can buy for security solutions, promising security with no compromises only to bring the world’s computer systems to their knees. CrowdStrike is a popular security tool to help organizations keep their systems secure. This issue was caused by a faulty update which led to Windows crashing.

The user could then restart the computer, only for Windows to try to load it again and the computer to restart. So surely all they need to do is push another update with a fixed version? In theory yes, and this is what they’ve done! But this is what makes this outage so disastrous because the computer can’t boot up without crashing. The automatic update doesn’t load. The fix involves booting the computer in Safe Mode, which doesn’t load the faulty files, deleting the broken file, and then letting it auto-update. Seems easy enough? Well just one problem, this has to be done manually, for every single affected computer, you can’t automatically apply it. What we’re seeing organizations do is triage, focus on critical systems first, manually fix them as quickly as they can, and move on to the next. This may be one of the worst outages in history.

Jason Griffin, MBA, CISM, Managing Director of Digital Health/IT Strategy & Cybersecurity at Nordic Global
The CrowdStrike outage underscores the importance of having well-documented, established, and effective Business Continuity Planning (BCP), Disaster Recovery (DR), and incident response plans. It is crucial businesses allocate an appropriate budget to support developing and implementing these plans and dedicate time for regular tabletop exercises to prepare for a variety of scenarios and understand the different roles and responses in an emergency. This outage should be a key scenario in future tests to ensure organizations are prepared for similar incidents.

To mitigate the risk of similar business continuity issues, hospitals and health systems should adopt several best practices. First, develop and maintain a comprehensive action plan for activating coordinated teams that can address every machine on the network if needed. Organizations should also have a well-defined communication plan that ensures clearly defined and integrated messaging and strategies, led by incident commanders who have the authority to activate plans promptly. Lastly, conduct regular “disaster drills” to familiarize healthcare workers with actions they should take during an outage, ensuring they can respond quickly in real-life scenarios. Even individual providers who may have fewer resources available should be prepared with an outreach plan and disaster drills to ensure their office staff can quickly mobilize to inform patients and other stakeholders of any interruptions to service.

In response to the CrowdStrike outage, vendors will re-evaluate their Software Development Lifecycle (SDLC) and change/patch management processes, particularly focusing on the testing phase before releasing new patches or updates. Healthcare organizations should also proactively engage with their vendors to ensure measures are in place to prevent issues in the future. This includes demanding transparency in their vendors’ software update and distribution processes and implementing robust incident response protocols. By learning from this incident and applying these best practices and changes, healthcare organizations can better prepare for disruptions that are inevitable in today’s electronic world.

Steve Odd, Director of IT & Information Security Officer at ABOUT Healthcare
The CrowdStrike outage offers invaluable lessons in cybersecurity and resilience, especially for healthcare vendors and organizations. It highlights the critical importance of a robust incident response plan and the need for a multi-layered security framework, like HITRUST, to ensure continuity. Regular backups and thorough vendor risk management have become non-negotiable practices, and continuous monitoring, employee training, and regulatory compliance are essential to mitigate risks. Implementing these measures helps us protect our systems, data, and clients’ patients effectively.

Lance Reid, CEO at Telcion
The CrowdStrike event was an unfortunate mishap with a large impact. We are constantly preaching to our clients about having multiple layers of security to protect yourself from outside threats. You never imagine the threat will come from your security vendor. Unfortunately, this won’t be the last time this happens. Technology is constantly changing, software is always being updated, and so there will always be some risk. So what do we advise you to do to combat such events?

Comment from CrowdStrike: This incident was not acyberattack or a cyberthreat, but an outage.

In this case, it comes down to disaster recovery and continuity planning. You must have a well-prepared action plan that is rehearsed and can be executed quickly so your people know immediately what they need to do to keep operations moving. Sometimes the event is short-lived. The plan can include immediate actions, with additional actions if the event is drawn out. For the IT folks, the Disaster Recovery plan includes having accessible and current backups and snapshots. This may seem obvious, but many organizations can not roll back their systems a few hours. The use of snapshots enables this to happen so that when an event occurs, the system(s) in question can be rolled back to a time prior to the event, and corrective action can take place. This can be done very quickly across many servers. Getting your servers back online, and your data back online is the number one priority.

Zack Tisch, Senior Vice President, Healthcare Services at Pivot Point Consulting
A key learning from the unfortunate CrowdStrike outage is that many healthcare organizations now realize how vulnerable and ‘single-threaded’ some of their systems and processes may be as they begin to work upstream with third-party vendors and their third-party partners. As part of business continuity plans, the infrastructure world has kept a close finger on the pulse of this, especially in light of upstream outages such as large-scale power outages or academic medical centers that may receive their internet services from their University. As more systems, access, and processes — particularly around cybersecurity — are managed by a variety of software applications and/or AI, a best practice for healthcare organizations is to map their vulnerabilities and include their vendors (and those vendors ancillary to their direct vendor partners) in their business continuity plan.

A key change I see as a result of the CrowdStrike global IT outage is that technology vendor partners to healthcare organizations will need to be more transparent about their own vendor landscape and their own business continuity plans. We actively advise healthcare organizations to update their vendor agreements to include a broader approach to sharing details for mission-critical systems, software, and infrastructure so that there are contractual clauses and contractual services that deliver pathways with multiple redundancies to limit the impact and the likelihood of an enterprise outage. There is also an opportunity to educate IT, clinical, and operational staff on all types of potential outages — legacy and new — an organization might encounter, how to identify and report them, and business continuity workflows to address each outage type. What we have seen is that there are a variety of system downtime causes and that it can be challenging to implement technical solutions for each and even more of a challenge to educate and train staff on all the differences.

Organizations that keep their downtime processes simple tend to see better and more accurate utilization of downtime workflows during common downtime events, but they tend to also be more vulnerable to the less-common outage types or may have to fall back to paper charting in those moments. Organizations that invest heavily in a variety of downtime tools and processes in an attempt to cover every outage situation possible often have the opposite challenge. Because they have so many downtime options – and workflow solutions – these organizations tend to struggle in identifying what type of downtime they are in and how to communicate the appropriate process to users. Often healthcare organizations do not have the capacity to address business continuity concerns. Engaging an experienced third-party consulting organization can help set them up for success by addressing their tech vendor landscape and business continuity plans for outages.

Ravi Soin, Chief Information and Security Officer at Edifecs
As an engineer, reflecting on this incident, it is evident that more stringent integration and QA testing processes are needed in all critical infrastructure software. While no software is entirely immune to bugs, incidents like these underscore the importance of two key lessons:

1. Collaboration and Unity: Witnessing the industry unite in forums and Slack channels during late-night hours as things unfolded to brainstorm solutions reinstated the true essence of collaboration. The tireless efforts of CISOs, CIOs, and their teams across aviation, banking, clinical, and retail to implement fixes and restore functionality deserve our utmost gratitude.

2. Complex Technological Environment: Thankfully, this was an explained outage. This incident serves as a reminder of the critical nature of our interconnected systems.

The importance of resilience and preparedness in the face of potential infrastructure disruptions cannot be overstated. Standing together as a community amplifies our collective strength. Moving forward, it is clear that teamwork and resiliency are essential in today’s interconnected world.

Will Cantrell, Director of Product Solutions at InteliChart
Business continuity plans are something that deserve serious consideration and review. BCPs are frequently put together with the minimum consideration and without the input of key stakeholders in an organization. Not only is it important to have a plan but it should be a plan that everyone in the organization is aware of and supports. During an active outage is the wrong time to be debating whether or not your plan is the right one.

You’re never going to avoid these issues altogether. You control what you can and prepare for what you can’t. Even if your systems are up the chances are that not every third-party software or technology that you’ve selected is up as well. There will always be something that impacts your business at some point for some reason which is why having plans that help you mitigate the impact of disruptions is so important.

Obviously, if the frequency of issues you’re having with a technology gets above your tolerance level you should seek a replacement. CrowdStrike isn’t the first technology to ever fail and it certainly won’t be the last. I think the smartest organizations are reviewing their risk opportunities and developing plans to mitigate those risks. With technology being such a large part of the healthcare industry there are going to be many potential points of failure for any organization that will need to be accounted for.

When you have dominant technologies in the space I think you’re more prone to widespread problems. It was security infrastructure this time but we’ve seen widespread outages from hosting services as well. The recent problems from Change Healthcare were felt across the board as well. Even simpler infrastructure components like ISPs could impact huge swaths of the country. It would be better for everyone if the tools and infrastructure used in the industry were more diversified but they’re not and unfortunately, the likelihood of that changing is small. Which takes me back to my primary take-away – identify your risks and plan for them.

So many good insights here! Huge thank you to Richard Bird, Chief Security Officer at Traceable AI, Katie Paxton-Fear, Security Researcher at Traceable AI, Mehdi Daoudi, Founder and CEO at Catchpoint Systems, Inc., Anand Naik, Co-Founder and CEO at SEQURETEK, Jonathan Burk, Software Engineering Director at Full Spectrum, Kevin Heineman, Chief Information Security Officer at Lyric, Jim Ducharme, CTO at ClearDATA, Lathe Bigler, GM at FDB Vela, Kiran Chinnagangannagari, Co-Founder, CPO, and CTO at Securin, Scott Stuewe, President and CEO at DirectTrust, Jason Griffin, MBA, CISM, Managing Director of Digital Health/IT Strategy & Cybersecurity at Nordic Global, Steve Odd, Director of IT & Information Security Officer at ABOUT Healthcare, Lance Reid, CEO at Telcion, Zack Tisch, Senior Vice President, Healthcare Services at Pivot Point Consulting, Ravi Soin, Chief Information and Security Officer at Edifecs, and Will Cantrell, Director of Product Solutions at InteliChart for taking the time out of your day to submit a quote in to us! And thank you to all of you for taking the time out of your day to read this article! We could not do this without all of your support.

What doyouthink are the biggest lessons learned that healthcare organizations should apply from the CrowdStrike outage? What best practices do you think healthcare providers should adopt to secure their systems against similar business continuity issues in the future? What changes do you think this outage will spur from vendors and from healthcare organizations? What other downtime issues do you think we should be working on to prevent a similar widespread event across healthcare? And what places do you think healthcare is vulnerable to downtime? Let us know your thoughts on these questions either in the comments down below or over on social media. We’d love to hear from all of you!