Taos is a managed IT services provider with primary facilities in the Silicon Valley region of California and in Boise, Idaho. Its three divisions are consulting, project advisory and managed services. The third division, launched in 2009, provides 24/7/365 remote managed services, including inbound and technical support and client application support. As the managed services division expanded, Taos needed to fortify and improve its Disaster Recovery and Business Continuity Planning capacities in order to meet the expectations of potential Fortune 500 clients and avoid crippling the company’s growth.
As a 100% United States based technical support center, Taos now handles: 20,000+ end user help desk tickets per month, 25,000+ Infrastructure monitoring alerts per month and 250+ critical infrastructure incidents per month. Taos serves clients in the following industries: high-tech manufacturing, biopharmaceuticals, clean technology, government, software, retail, wireless, publishing and automobile.
The inability to offer 100% SLA for power availability threatened to hold Taos back. Yet the infrastructure investment required to guarantee 24/7/365 uptime was prohibitive. Ryan Munson, VP of Infrastructure, has been with the Company since its early days. “When we got started, Taos didn’t have the need for sophisticated disaster recovery planning, but within the first year or two we started to be approached by clients who were interested in our managed services but had much steeper disaster recovery and continuity requirements than we had. We were talking to very mature Fortune 500 companies with established DR/BCP. They couldn’t roll back their standards of care to take on a new provider. We needed to mature with DR and BCP, otherwise we were going to cripple our ability to grow.”
Taos engaged Involta to formulate a cost-effective strategy for significantly improving its DR/BCP. Brainstorming between Involta and Taos infrastructure engineers yielded a key insight. Munson explains, “We came to Involta and realized that for the services we needed, for us it’s not about getting a server running in a different location, it’s about getting people running.”
One of Involta’s data centers is located in the same city as the Taos inbound call center. The Involta Data Center has fully redundant power, cooling and Internet connectivity, making it a convenient and technically feasible disaster recovery site for Taos.
Involta created a custom Business Continuity Suite within the Boise Idaho Data Center. The Continuity Suite is a call-center-grade secondary site outfitted to Taos’ specifications.
Collaborative ingenuity also solved the problem of speeding the RTO (Recovery Time Objective) from several hours to 30 minutes. All of the technical work of cabling up connections, power and solving for the VPLS Taos’ Boise facility to interface with its Silicon Valley facility was done in advance. The PCs at the Continuity Suite are now actively plugged into the VPLS, receiving patches and updates. “That creative brainstorming radically shortened the RTO. Without that, it wouldn’t have been a 30 minute exercise, it would’ve been a 3-hour exercise.”
With Involta’s encouragement, Taos conducted several full-scale tests of its DR/ BCP despite the logistical challenges (given its 24/7/365 operations). In both tests, Taos successfully transitioned operations to the Involta Business Continuity Suite in under 30 minutes.
On May 1, 2014, an isolated power-grid failure at Taos’ primary facility in Boise put the Involta DR/BCP solution to the test. Munson happened to arrive at the facility when it occurred. “My building was pitch black. Totally dead. It seemed like it was literally only our building.” A series of phone calls with Idaho Power yielded the unwelcome news that this outage was due to a nontrivial fault. That is, the power had gone out to a patch of about 45 commercial and residential customers, but Idaho Power didn’t know why. Estimated downtime duration: 6-8 hours. The Taos executive team had no other option but to declare disaster.
The extensive advance preparations and multiple exercises paid off. Though there were hiccups, the Business Continuity Plan procedures were executed as planned. Taos clients didn’t realize there had been a phone system outage until Taos informed them. Ultimately, the Taos inbound call center in Boise Idaho was officially out of power for a total of 2 hours and 39 minutes. But thanks to Involta’s custom DR/BCP solution, Taos’s 100% SLA was not impacted.
Within the first 30 minutes of declaring disaster:
With the ability to offer 100% SLA and the reassurance of the DR/BCP’s seamless performance when tested, Taos now supports 37 enterprise clients with operations in the United States, South America, Asia/Pacific, EMEA and India.
They have achieved astonishing 29% per quarter growth. “Involta takes a lot of care to honor this service agreement and keeps this in place so it looks and performs the same way every time we show up. As far as a service offering, Involta’s DR/BCP solutions are fantastic. I think they should appeal to a lot of people who should be thinking about this stuff but instead are enduring the downtime.”
An isolated power-grid failure at Taos’ primary facility in Boise put the Involta DR/BCP solution to the test and it passed – maintaining Taos’ 100% SLA offering.
