How a California county health system took a quantum leap in disaster preparedness and business continuity through the use of hybrid cloud and network fabric technologies?

By Amanpreet Kaur, Information System Analyst IV, San Joaquin General Hospital and Mark Thomas, CIO, San Joaquin General Hospital

Introduction

Until 2018, San Joaquin General Hospital and County Clinics had a relatively immature digital infrastructure. Its core healthcare processes were documented in paper charts, and numerous isolated systems supported the core administrative functions of patient registration, billing, general ledger, and human resources management. In March of that year, the health system successfully pulled off a ‘big-bang’ implementation of monolithic and integrated Electronic Medical Record and Enterprise Resource Planning systems. While this effort yielded a transformation in the organization’s clinical and business processes, it significantly increased its reliance upon Information Technology.  Extended outages were no longer acceptable and a major disruption in IT services became a high risk to patient care and business operations. With the full support of executive leadership, the Information Systems Department initiated a transformation of the organization’s network infrastructure and disaster recovery capabilities to meet these new availability and resiliency goals.

Hybrid-Cloud

For experienced IT professionals, the very phrase “Disaster Recovery Solution” conjures up thoughts of distant co-location facilities, idle hardware, expensive circuits, a large capital investment, and infrequent and often unsuccessful Disaster Recovery Testing events. Today, however, hybrid-cloud solutions offer organizations like ours the ability to deliver superior functionality with less capital investment and a lower total cost of ownership. Hybrid-cloud solutions extend modern data center capabilities into the cloud and enable the organization to leverage its existing on-premise investment.

Nutanix Xi Leap

Since our data center already used a Nutanix Hyperconverged Infrastructure, we implemented the Nutanix Xi Leap Disaster Recovery as a service (DRaaS) to avoid a large DR capital investment and extend our existing data center’s capabilities into the cloud. We partnered with Nutanix professional services to protect our most crucial business systems, as identified in our Business Impact Analysis (BIA). The entire configuration was done from a single ‘pane of glass’, Prism Central.  Nutanix Xi Leap offered us a variety of recovery plans and protection policies to meet the Recovery Point Objectives (RPOs) and Recovery Time Objectives (RTOs) established by executive leadership. Xi Leap offered us a variety of service levels to select from, including full network and partial network failover recovery plans.

Network Fabric Technology

Readers who have struggled with complex network architecture challenges will be thrilled to read what network fabric technology allowed us to accomplish with our infrastructure. 

Legacy Network

The health system had a traditional and complex network design which lacked dynamic routing, relied upon spanning tree for loop detection, and required manual provisioning. Network switches were statically interconnected with single fiber paths, virtual networks were provisioned with a rudimentary loop prevention mechanism, and the network design was flat and riddled with single points of failure.

Extreme Fabric

We partnered with Extreme professional services to implement Extreme’s Network Fabric to transform our legacy network infrastructure into a private network cloud. Extreme Fabric technology offered us extremely fast convergence times by allowing all links to be active and simplifying the extension of services across multiple sites. Fully redundant and active LAN and WAN fiber paths transformed our infrastructure, and the flexibility of the fabric allowed us to meet our challenging disaster recovery requirements.

Disaster Recovery Testing  

As many IT professionals know, Disaster Recovery testing events are often horror stories. Millions of dollars may have been spent to create a DR infrastructure that is only tested infrequently and that otherwise sits idle. Lengthy and complex DR runbooks are dusted off and staff members attempt to meet the RTOs and RPOs that their managers have committed to the business. The resulting complexity turns testing events into labor-intensive, exhausting, and thankless ordeals. The flexibility and ease of using Nutanix’s Xi Cloud allowed us to rewrite this tale completely.

Armed with a highly capable infrastructure, we designed disaster recovery procedures for multiple systems, simply and in little time. As one example, we created a DR recovery plan for a pneumatic tube system powered by several network-attached controllers and managed by a central server. P-tube systems are critical healthcare infrastructure that shuttle blood and tissue samples between nursing units and the clinical laboratory. We utilized a full network recovery plan for this system, which allowed us to recover the P-Tube central server in Xi Cloud. After a few readiness checks, we  recovered the central server in the Xi Cloud while maintaining its original IP address. The P-Tube system was operational within minutes without making configuration changes to P-tube controllers. The Xi Leap failback feature allowed us to restore services to the on-premise infrastructure while replicating back all system logs and changes.

Conclusion

After our DR journey, our organization has a network and disaster recovery infrastructure that is simpler to manage but significantly more capable. We are well on the road to meeting all the recovery objectives for our critical business systems. Our executive team has greater confidence in our ability to sustain operations in the event of a catastrophe.

Our journey to meet demanding resiliency and availability goals was at times perplexing and time-consuming, but as Audrey Hepburn once said, “Nothing is impossible, the word itself says ‘I’m possible’!”