By Steven Ramirez, Change Healthcare
In today’s realm Business Continuity and Disaster Recovery (BCDR) planning, it isn’t every day that a disaster “makes a reservation.” As a society, we have learned a lot from recent storm events, from Hurricane Katrina to Super Storm Sandy. Those lessons learned have helped strengthen BCDR planning across the country. With that said, according to an August 28th New York Times article “sometimes even the soundest plans have been foiled.”
Now, let’s look at the recent events of Hurricane Harvey that ravaged the state of Texas (and Gulf Coast), and the lessons learned from this event. While everyone “knew” this hurricane was coming, no one had an idea of the impact it would bring to the state of Texas. We have now seen that Hurricane Harvey surpassed the “100 year” flood mark (safe zone) previously set in many Texas counties.
When it comes to hospitals and healthcare in general, rain, sleet, snow, or hail will never diminish the necessity of medical care for the communities they serve. Healthcare has come a long way as it relates to its “culture” of preparedness and lessons learned, that can all relate to the preparedness initiatives and regulations implemented over the past decade.
To give a little history on the evolution of the healthcare industry over the past decade, until around 2010, many healthcare organizations were primarily paper based organizations, meaning that all patient medical records, orders, notes, etc. were done on good old fashion paper. While this “worked” for the industry, healthcare found itself lagging far behind other industries in means of technology utilization. This soon would change with the implementation of The Affordable Care Act (ACA), (which mandated that all healthcare organizations implement Electronic Health Records (EMR) as a means to improve the quality of care, information sharing, population health improvement, amongst other things).
Now where we are today, with the passage of the ACA, we have seen the dependency grow more and more into the necessity of having these systems “always available” to ensure quality patient care. The risk tolerance level of many healthcare organizations, especially for emergent care sites, i.e. “trauma 1 facilities,” stroke centers, and critical care units (burn, pediatric) has continued to decrease. Hospitals are so dependent on system availability that many have come to adopt “Internal Disaster – IT Outage” as a part of their hospital incident management process. Can patient care continue without technology? The answer is yes, however, it is so ingrained into our culture and processes it has a significant impact.
The means how a hospital/healthcare organization handles disaster/disruptions all relates back to the organization’s risk management and BCDR maturity level. An organization’s risk appetite varies by location, threats, vulnerabilities, management support, and of course, funding. With that said, just because an organization is a large multimillion dollar revenue generating machine, doesn’t necessarily mean it is “more” prepared than a rural critical access hospital. Bigger isn’t always better.
Now to tie this back into healthcare organizations’ dependence on technology. So, what are the risks of system or network outages on healthcare organizations? Today, technology is ingrained in all elements of a hospital’s ecosystem. From the medical imaging systems, to patient care records, registration, and insurance verification/billing, any impact to these systems can leave many organizations at a stand-still, (with impacts varying from patient care/safety, to hospital operations) requiring them to cancel any non-emergent cases or put their emergency department on diversion.
To highlight the impact of medical imaging systems, data Change Healthcare Medical Imaging Consulting has gathered shows that medical imaging accounts for roughly 40% of a healthcare organization’s revenue stream. Medical imaging systems are also a critical care component for physicians to diagnosis and treat emergent trauma or other health related issues. These systems alone emphasize the importance for a robust BCDR plan.
So how do hospitals mitigate these risk to their IT systems? The Center for Medicare and Medicaid Services (CMS) has mandated that hospital take several preparedness means to protect healthcare facilities against disasters/disruptions. These vary from generator requirements, to emergency response/preparedness and contingency plans. These requirements have evolved to encompass technology as well. The Health Insurance Portability and Accountability Act (HIPAA) Security Rule has been the most impactful regulatory means to address healthcare IT systems. Specifically, under the HIPAA Security Rule (Contingency Plan Standard 164.308(a)(7)) Administrative Safeguards, the standard requires that organizations “establish (and implement as needed)” methodologies for recovering systems containing protected health information: (Source HHS.gov).
So, what did Harvey teach us from a IT Disaster Recover Perspective? Focusing on the impact to healthcare organizations IT assets, Harvey has emphasized some previous lessons learned as well as brought to light some other factors to consider:
1. Dust Off That DR Plan and Test It
Have a DR Plan and test it frequently. Having a well developed and tested DR plan will help ensure your organization is properly prepared to recover your IT systems in a timely fashion. What makes up a good DR plan? As a baseline, it is important to include steps and responsibilities associated with recovery of the system. Testing the restoration of a system (and documenting the recovery steps it in the process, especially if no plan exists) will help identify any gaps and enable you to fine tune the plan. Also, it is important to note upstream and downstream applications that are essential to the application, to ensure the systems also have recovery plans. Testing a DR Plan (even as a tabletop) once to twice a year well help an organization institute a more effective and efficient recovery. Those organizations that had robust BCDR Plans fared much better with mitigating the risks of Harvey.
2. Embrace Hosted/Cloud Storage
The best way to protect and recover your data is to have a data protection strategy that aligns to the organization’s. Having data backups offsite is the best model to align with a minimal to no data loss, in a disaster like Harvey (or anything impacting your production data center). The best way to achieve this is with a hosted/cloud storage solution. Hybrid solutions like Datatility incorporate “on-premises” devices that allow for expedited retrieval, as well as adhere to IT security requirements. This means of storage/archival protect organizations from internal and external threats by housing their data offsite and enabling expedited recoveries at DR and co-location sites.
3. DR Sites and Co-locations Are Too Close in Proximity
Many healthcare organizations have taken big steps to enhance their BCDR posture over the years, with a lot of this due to the large investments into new electronic heath records and the mandates on recovery. Due to latency (network slowness), costs, and an organization’s regional footprint, many organizations have chosen to “stay local” or “regional.” While this may not seem as an alarming statistic, when events like Harvey roll into town and bring with them flood waters that hit historic flood zone marks, there is reason for concern. While many of these co-location and DR sites have a lot of redundancies and safeguards in place, they can still be impacted and leave an organization without a recovery site.
For example, in Houston, there is a large tech presence in the Katy Texas area which is roughly 30 miles away (and at the 100-year flood zone; note Harvey hit the 300-year mark), that poses risks to those sites. To remediate this risk this is where cloud/hosted storage and disaster recovery as a services (DRaaS) can be utilized as well as DR/Co-location sites much further away. It is important to understand what disasters your region is prone to when making these decisions to ensure a low risk solution is chosen.
4. Institute Disaster Avoidance
Technology has come a long way over the past decade. With the emergence and transition from physical of virtual environments (hardware) there are many more options for recovery solutions organizations can utilize. New failover technologies VMware bring to the table, allow for organizations to institute “disaster avoidance.” What this means is organizations can failover systems when they know events like Harvey are coming, to mitigate the risks of an outage at their primary data center (most of which are housed at the hospitals).
We learned a lot from the events of Harvey that we in the BCDR community will be able to take back and use to help better prepare for future catastrophic events.
About the author: Steven Ramirez is a Sr. IT Risk/Security Consultant with Change Healthcare.
