I turned my phone on as my flight landed at the Tokyo-Narita airport for a quick layover. That’s when I saw all of the Facebook messages that my kids sent me. My wife was rushed to the hospital just a few hours before due to a mild stroke. Before I even called my kids, I immediately went on to Expedia to make modifications to my travel arrangements. Besides, I still have another 15 hours to go before I make it to JFK airport. Forget about the original train ride from NYC to Ottawa, I’m booking a new itinerary once my flight lands in NYC. I promised my kids to be at home before they wake up the next morning.

Good news. My flight landed at the JFK airport with enough time for me to go thru US immigration, transfer to LGA and catch my flight to Toronto. Bad news. The flight to Toronto from LGA was 3 hours delayed and the next flight from Toronto to Ottawa won’t be until the next morning. I needed to decide fast. When my flight made it to Toronto, I rushed to the nearest rental car terminal to book a one-way rental. It was 20 minutes before their office will close for the day. With a four-and-a-half hour drive, I managed to make it home before sunrise, kiss my kids while they were still asleep and snuck in a comfortable bed to get some rest before heading to the hospital. That, so far, was the longest and most stressful travel experience I’ve had ever since I started traveling.

Responding To An Unexpected Disaster

Large organizations whose business operations rely on IT assets already have a disaster recovery (DR) plan in place. Some of them rehearse the DR plan on a regular basis as part of their compliance requirements. Unfortunately, most of them only have it on paper. When I used to work as a data center engineer, we performed our DR plan every six months and updated our processes and documentation after the exercise. A business continuity plan (BCP) exercise was also conducted regularly to ensure that every team member will be prepared to respond when a real disaster strikes.

But regardless of the amount of preparation, we won’t be able to account for every single factor that can contribute to a disaster. Because there is no such thing as an ideal disaster. We can only prepare for what we are aware of. Plus, DR plan exercises are done under favorable conditions. John Pavley, CTO of the Huffington Post, once commented after Hurricane Sandy, “We test at night when traffic is low, or we test when the key IT people aren’t stressed out.”  Even the emotional and psychological state of the team members are different during DR plan exercises – they know its just an exercise; there is nothing at stake other than making sure that the processes are followed and that the service level agreements are met. Real disasters change our emotional and psychological state especially when our lives and our loved ones are at stake. And I believe that is the most important thing that we need to prepare for when a real disaster strikes – ourselves.

1. Don’t panic. This is easier said than done. Our natural reaction when things don’t go as planned is to panic. And, unfortunately, you won’t see this statement in any DR plan. Ask the fire fighters, the soldiers, law enforcement personnel, emergency response team members. They train for the worse and prepare for it. But they are also trained to stay calm under pressure. Panicking can cause anxiety that will affect your way of thinking. It may cause more harm than good, especially in an emergency situation. When I heard about what happened to my wife, I had to stop for a while and gather my thoughts. Which led to . . .
2. Identify what’s really important. When I do my high availability and disaster recovery (HA/DR) workshops, I emphasize the value of defining what’s really important. Organizations have a different definition of what’s important to them and are very different from what’s important to us. The one thing that I tell people about dealing with real disasters is this: “I doubt that you will be concerned about your database servers when a strong earthquake hits.” As someone who has experienced typhoons, earthquakes and volcanic eruptions in my lifetime, I can absolutely tell you that life is more important than anything else. I’m 100% sure that you will think about your family first – not your database servers – when a real disaster strikes. Which is why preparation is important. In an IT organization, every asset needs to be categorized according to importance and criticality because not all IT assets can be considered equal. When I heard about what happened to my wife, I know that my priority is to make it back home safely to be with my kids as soon as possible.
3. Decide based on #2. When a real disaster strikes and you can’t go to your physical office, the HR system will probably not be the focus of your efforts (unless you’re a software-as-a-service company providing online HR systems.) The focus of your efforts will be on what has the greatest impact to the organization. This is why you need a business impact analysis. A business impact analysis (BIA) answers the question , “What are the risks of losing a specific process in the business?” The team needs to know “what is the priority“, not “what are the priorities.” Unity and a single-minded purpose can help make better decisions and initiate quick actions. Because I needed to make it back home safely ASAP, I had to forget about my train ride from NYC to Ottawa and arrange for flights instead. When I couldn’t make it in time for my flight to Ottawa, I decided to drive. Decisions and actions were all based on what was important at that time.
4. Remember the team. Whether you’re a database administrator, a developer, a manager or the CIO, you are and will be working with a team – people like you who have concerns, emotions and priorities. It doesn’t matter whether your official job title signifies authority. What matters is that you have influence. While our natural tendency is to take care of ourselves first during a real disaster, it is important to watch out for the other members of the team. Ask around if anybody needs assistance with anything. See if anybody would need a word of encouragement. Be inspiring, be positive and be encouraging. Even highly-skilled individuals feel discouraged sometimes. People will perform better when they feel good about themselves. As I was talking to my kids on the phone before boarding my flight to NYC, I tried to listen to their voices. I can’t do anything for them at that moment but I did offer some comforting words. I heard the positive change in their tone as they enumerated the tasks that they need to do before I even got home. They’re excited to see daddy and even more excited to visit mommy at the hospital the next day.

DR plans focus mostly on tasks but rarely on people. But we miss the point when we don’t prepare the team responsible for responding during real disasters. That’s why it is important to focus on the (P)eople, (P)rocess and (T)echnology aspects of any HA/DR solution.

Question: Have you faced an unexpected disaster in the past? How did you and your team respond to it? You can leave a comment by clicking here.