I was recently collaborating with a group about creating a fault tolerant environment for their applications. None of their systems have any sort of immediate failover implementation. I wonder how common this situation is across companies worldwide. For one reason or another they are not using cloud solutions with failover built in. Be aware that all cloud solutions don’t necessarily have failover build it, but many do. Companies are self hosting with a limited DRP (Disaster Recovery Plan).
This conversation reminded me about a blog I read some time ago from Denny Cherry about recovery. He had gone through the experience of losing a complete hosting site for his database servers, and had to recover to a secondary sight using database backups not located at the primary location.
Could your company do that? If you lost your prime location, do you have the capability of bringing your databases up on a secondary location? Sure, there are many different ways to do this kind of action. Some are real time such as SAN replication. Others have latency such as making backups to tape, and shipping those to the secondary location when required. Log Shipping, replication, Always on. There are many different ways to get this done.
So, you are still feeling pretty good about your implementation. That is a great feeling. You have a DRP implementation that covers just about any contingency. Or, maybe, you think you do. Have you ever tested your DRP? When it’s time to execute a DRP it is too late to find that there is something you missed, or that some important file is corrupted. Perhaps the secondary hardware is not powerful enough because your load has increased since your secondary site was established. An untested DRP may work perfectly. Are you willing to bet your job or reputation on that?
Here’s a scenario I sometimes consider. What if your data becomes corrupted for some reason? Is that being replicated to your secondary site? Could you restore you database to a point prior to the corruption?
There are any number of scenarios that may require execution of your DRP. Make a good one. In my opinion, humble or not, a good DRP is one that has been tested and maintained.
Cheers,
Ben