Initial Customer Contact
You’re customer is upset. You’ve promised to provide some service that is now failing, and they are likely loosing money because of your outage. They’ve called your support line, or sent an email, and they are looking for a response. What do you do?
Give your self a second
Outages happen on their own schedules, and you may be at a movie, sleeping, the gym, or eating dinner at the French Laundry for example. You need to give you’re self 2-3 minute to compose yourself, find internet access, and call the customer back. If you have an answering service you’ve likely met the terms of your SLA, if you don’t figure out how much time you can take. I think this is a better option than voicemail, since it handles any issues you may have communicating with a customer in the first few minutes of the call. They may even be able to file a ticket for you with the basic information you need. This can cost a fair bit of money, and if this option is too pricey for your service, consider a voicemail number that will page your on-call team. It gives your team a small buffer, but they have to be prepared to talk to the customer quickly since this may add up to 5 minutes between the initial call and page. As the last resort, have your customer support number dial someone who is on-call. If you have the time and resources, make the email address you use for outage reports follow the same workflow as calls, so you don’t need a second process.
Promises Can Be Hard to Keep
Track your customer’s complaint; make sure its recorded in your ticketing system. You want to start keeping a record from the moment they called you, and be able to reconstruct the incident later. This will also help you determine a start time for any damages clause that may be in your SLA. I’d make sure the following things are done:
- Get a call back number.
- Let them know you are looking into the issue.
- Let them know when you expect to call them back.
- Let them know the ticket / incident number you are using to track the issue.
- And most importantly, don’t promise anything that you can’t guarantee happens.
Have you met the terms of your SLA?
You only have one SLA agreement, right? If not, hopefully the basics are the same. Keep in mind what you’ve agreed to with your customers, and as early as possible identify if you’ve not met the terms of the service agreement. This is really just for tracking, but it can be useful if you have to involve an account manager and discuss any damage claims.
Houston, we don’t have a problem.
You’ve talked with the customer, you’ve created a ticket, you’ve managed expectations, now its time to figure out if there is an issue.
- Check your internal monitoring systems.
- Check your external monitoring systems.
- Check you logging.
- Check your traffic.
- Give our customer’s use-case a try?
Does your service look ok, or do you see a problem? At this point you want to figure out if you have an issue, or not. If you can’t figure it out quickly, you need to escalate the issue to someone who can. If you don’t have an issue, call the customer and see if they still have any issues, and if they’ll agree to close the issue. If they are still having issues escalate, and if you have doubts as to wether your service is working, escalate. If you know you have an issue, its time to move on to resolving it.
Who Needs to Know?
Its important to let everyone on your team know your service is having issues. Before anything happens, you should know who you need to contact when there is an issue. This will save time, and help minimize duplication of work(in larger organizations, two people may be receiving calls about the same issue). A mail group, or centralized chat server is an ideal solution since it fairly low latency, and you can record the communication that can be review later. You should be clear as to what the problem is, and provide a link to the ticket.
Who has your back?
The next thing you should be working out is who do you need to solve your issue. You product could be simple, or fairly complex. You may be the right person to address the problem, or you may need to call for backup. If you have an idea of who you need get in-touch with them now. Get them ready to help you solve your problem. It takes quite a bit of time to get people online, so if you possibly need their help its better to call them sooner than later.
Herding Cats
Finally, now that you’ve let everyone know, and you have a team assembled to solve the issue, figure out how you’re going to communicate. The method should be low latency, and low effort. I prefer conference calls, but a chat server can work just as well plus you can cut and paste errors into the chat. You should have this figured out well in advance of an incident.
Come on you tiny horses!
You’re ready to fix the problem. Just a few more things your should have figured out:
- Who is doing the work?
- Who is communicating with your customer?
- Who is documenting the changes made?
- Who will gather any additional people needed to resolve the issues?
This could be an easy answer if you only have one person, but working through almost any issue is much easier with two people. Ideally one person will act at the project manager, getting extra help, talking to the customer, while the other types furiously in a terminal to bring the service back up. I fyou have this worked out beforehand you’ll save some time, but if you don’t, come to an agreement quickly, and stick to your roles. You don’t need 2 people talking to your customer, telling them different things, or worse two people bringing up an down a service.
So you’re finally back up…
Great only a few more things to do.
Open a ticket for the post-mortem. Link it to your outage ticket, and begin filling in any information that might be helpful. Be as detailed as possible, and even if its inconvenient take a little time to document the issue and resolution. You should also schedule a meeting immediately for the post-mortem that takes place in the next 24 hours. People are beginning to forget what they did, and you need to capture as much of it as possible.
Once you’ve completed your meeting, produce a document explaining the outage. This should be as brief as possible with very little internal information included. Document the timeline leading to the issue, how the issue was discovered, and how it was resolved. Also, build a list of future actions to prevent a repeat of the incident. If your customer asks for it, or their SLA includes language promising it, send them the document to explain the outage.
So, spend time thinking about talking to your customer when your down. Think through the process, so when they call you won’t have to make it up. I’ve setup several of these processes, and I’ve found that these are the issues that always need to be looked at. It worth the planning, and its always important to look at what happened, so that you can improve the process.