Designing From the Ground Up

If you're designing a new center, consider developing your disaster plan from the start. Paul Marshall, senior director customer support for Yellow Freight System (Overland Park, Kan.), the nation's largest less-than-truckload carrier, has been with the company for 15 years, originally as a branch manager of terminals. "As a less-than-truckload carrier," he explains, "we carry everything bigger than a FedEx or UPS package-general commodities of various sizes. There might be 10 shipments in one trailer and 30 in another." Yellow Freight's 25,000 employees work for a Fortune 500 operation that enjoyed 1998 revenues of $2.5 billion on its way to being voted one of the top 100 technology companies this fall by CIO magazine.

In 1994, Marshall was part of a team that moved the decentralized customer services into two call centers totaling 500 employees. The Des Moines, Iowa, center opened in 1995, and the Sioux Falls, S.D., center opened five months later. Both centers are operational 24x7.

As the centers were designed, Marshall kept redundancies in mind and ensured that there were emergency backup generators, a network of multiple fiber optic lines and back up routers. "We built in as many redundancies as possible," Marshall says, "but it does get kind of expensive." When the centers first opened they had no mission-critical applications. "I'd say the worst situation we could have faced at that time would have been a great inconvenience, but not catastrophic." However, the centers have since become the central places to call for pickups. "If a center goes down and people can't call for pickups, we could lose millions of dollars in a matter of hours," says Marshall. "That realization took us to the next level in terms of disaster recovery."

"We used to be able to live without providing service for 24 hours if we had to, but customer service is now so important that we can't live without it for one."

Short-Term Crises

Not everyone is so lucky to plan a disaster recovery when the center is being built. However, most companies can restructure their existing systems to build in more redundancies, such as increasing the number of servers and other equipment that are on standby. As Marshall says, "We used to be able to live without providing service for 24 hours if we had to, but customer service is now so important that we can't live without it for one."

Lovvorn's AT&T facility has three categories of disaster plans in place: short-, medium- and long-term. His short-term plan covers disruptions ranging from systems problems, power outages, fire alarms and bomb threats that might force short-lived down times or evacuations. In the latter event, he says, "First we get people out of the facility," he says, "then route the calls to a sister facility in Sacramento, Calif. We also have a panic button in several places throughout the facility. If they're activated for any reason, the calls are routed outside."

Mid-Range Problem

A mid-range problem disrupts or closes a center for one or two days, usually from a weather-related occurrence like heavy snowstorms. AT&T has contacted local hotels to see how much advance notice they would require for the company to book a block of rooms in the event of inclement weather. "At that point," says Lovvorn, "we solicit volunteers to come to work prepared to stay in a hotel if necessary, so that they could make it back to work rather than be stranded at home. This way, at least we have partial staffs at all times. In order to operate such a system, however, we need to take into consideration the availability of restaurants or caterers for our people."

Even a weather problem that doesn't directly hit a center can seriously overload and strain its resources. Prior to his job at AT&T, Lovvorn spent a decade managing reservations at the Sheraton Hotel in Raleigh, S.C. He says that when Hurricane Andrew hit Florida, there was a tremendous flow of people coming inland and the overload of calls to the hotel's CIC had to be reallocated to different parts of the country.

If one of Yellow Freight's centers goes down, the second location takes over as much of the workload as possible. Marshall's group worked with Yellow Services, the company's tech team, to develop a voice recognition message that could explain to callers why a center was inoperative, that their calls were being rerouted and there might be an unusual delay. Database routing would then determine which calls would go directly to a representative at the other center and which would be put in a queue.

Yellow Freight is also working with a vendor to develop a third electronic site that could handle calls using an automatic voice response system. Calls can be routed to the appropriate people following a voice prompt, such as "1" for pickup routes. "We're well prepared if something happens to one site," says Marshall. "Fortunately, it's usually only a network problem, something going bad in the equipment."

Yet the sites have had their share of non-network problems. In 1997, tornadoes forced the Des Moines center to be evacuated several times during one day. In another instance, the same center closed in the early afternoon because of a blizzard. "The transition to the other site was seamless to the customers except that we lost the ability to answers calls as quickly as we'd like," says Marshall.

"Using an outsourcer to keep calls answered can be a solution for emergencies. But don't wait until the disaster strikes to form the necessary relationship with such a firm."

Long-Range Problem

In this age of immediate gratification, being down for two or three days can qualify as a long-term disaster. If forced to, Yellow Freight is capable of opening another center outside Kansas City, Kan., and making it operational within 24 to 48 hours. Most of their systems are proprietary and can be reconstructed by the company's IT team.

When Hurricane Floyd hit the East Coast in September, the AT&T Orlando, Fla., site was closed for one day. The New Jersey site was also heavily flooded and the network sustained some damage. The AT&T network diverted calls to centers in the western United states, but Lovvorn warns that this only works well if the various sites have linked databases. "You must ensure that the company has access to customer databases in other parts of the country," he says.

A disaster planning group should maintain a list of employee names, addresses and phone numbers that is updated at least every quarter, preferably every month. In the case of Hurricane Floyd, although the New Jersey center thought the Friday closure would be a one-day event, the team made calls during the weekend to schedule people for overtime on Monday. The center wasn't even able to reopen that fast, but had it been able to, those staffers who didn't have personal disaster issues-or who had other family members who could care for things while they worked-would have been ready to man the phone lines.

When Hurricane Fran hit North Carolina several years ago, many Sheraton employees needed personal time off to take care of their own needs. As things started to settle down, but not quite to normal levels, people brought family members to work with them, including young children who needed to stay in the temporary day care facility set up on site. "The company tries to help its employees with their personal lives," says Lovvorn. "There was a break room where people could go watch TV, and employees could get water and ice from us. Because of all the flooding and power outages, ice was at a premium."

AT&T CIC employees have an internal IVR number to call for information to find out if a facility is open, if people need to report to another location or if they are needed for additional shifts. Employees also know which radio stations broadcast information to employees and the community at large in the event of a natural disaster. Last but not least, someone on the team is responsible for posting information on the doors to the building if it's accessible, to give information to any employees who show up for work.

"We're always open to moving people to other locations," says Lovvorn. "We look to volunteers first-and find warehouses to convert to temporary call centers. We've thought of every detail. We've got banks of portable toilets in the parking lots, food in our on-site kitchens and generators in all the facilities."

Other Options

Although some companies, especially those that already outsource a particular segment of their calls, could turn to an outside party for help when they encounter problems, it's not so practical for everyone. Yellow Freight representatives, who handle 41,000 to 45,000 customer contacts a day, go through eight to 10 weeks of training before being considered "minimally qualified" for phone calls.

Although there are a number of simple transactions, says Marshall, "the complexity of what we do is pretty high. Many people who visit us are surprised by that. A `Where's my shipment?' call can lead to a 10-minute investigation to see if the shipment was misrouted, if something was missing or a check on reconsignments and upgrades."

Sometimes calls stop deliveries enroute and have them rerouted to other parts of the country. Or someone could call for a rate quote. Sounds simple, perhaps, but in many cases it's not. Since the cost of a shipment is based on weight, determining a shipment's weight is often the first hurdle. Then there's the matter of getting the product item number from the National Motor Freight Classification. "Not a day goes by that someone doesn't ask me something I don't know," explains Marshall. "For example, there are five pages of electrical switches. It's not enough for us to be told that the items to ship are electrical switches. Are they for cars, trains, planes or something else? It takes a lot of time to determine rates based on classification. Part of the training our reps go through is learning where to go for such answers and how to work with our knowledge management system."

However, for the right company, using an outsourcer to keep calls answered might indeed be an appropriate solution. But don't wait until the disaster strikes to form the necessary relationship with such a firm.

"As needs arise, we invent the answers. We don't have someone else's best practices to follow."

Meeting Clients' Business Recovery Needs

With a name that evolved from Computer Discount 30 years ago, Comdisco, headquartered in Rosemont, Ill., got its start in the mainframe leasing business. Since that time, Comdisco has sold its mainframe portfolio to IBM and has evolved to a technology services company with core services that include IT control and predictability, managed network services and business continuity and availability services. Comdisco Continuity Services, which has been in business for 20 years, helps organizations ensure the availability of their systems, applications and data across data centers, networks, distributed systems and work areas. With more than 50 recovery facilities worldwide, Comdisco's availability services include alternate sites, work area, network and mobile recovery, advanced work area recovery, which provide for financial services, and item processing recovery. In addition, Comdisco offers a full range of availability services for the CIC.

"We're focused on helping companies build redundancy into their business continuity plans," says Bob Felch, director of business continuity products. "Companies can look to us to recreate their CIC for a number of different reasons ranging from a system failure through a natural disaster. We can help organizations protect their mission-critical call center technology that encompasses computer telephony integration, automated call distribution, interactive voice response and voice over Internet protocol."

The first steps in creating a business continuity plan are to identify business-critical functions that an organization can't exist without, estimating the cost of a disruption, as well as determining what the recovery times are for each critical function," Felch says.

Comdisco's Professional Services group can assist a company in designing and implementing its business continuity plan. If desired, Comdisco will examine a client's internal business process and can conduct a business impact analysis to evaluate the organization's business functions to determine what is critical to the company's revenue and customer service and the potential problems if such functions are interrupted. Organizations can then establish an optimized timetable for recovery. The client's technology is evaluated, as are its external service providers, and possible disaster situations are examined. Comdisco's team can also help organizations identify the criticality of agents with the company. For example, a business may have 200 agents but only 20 to 30 might be viewed as critical, at least for a short-term problem.

Another critical part of business continuity planning is testing the plan on a regular basis. "Annually testing your plan dramatically improves the likelihood of success," Felch says. "Testing can often reveal planning details that can make the difference between a smooth recovery versus a crisis. That's the whole idea behind testing-to find out what doesn't work and fix it. For example, some call centers may find during testing that they have a document of a disaster plan, but there's only one copy of the plan, and it's stuck in the production office, which they may not be able to get into if there's structural damage to the office."

For longer-term recoveries when relocating employees is not an option, companies can use mobile recovery services for their call centers. Space can be set up with modular trailers that can accommodate everything from a small, 20-seat environment up to a 250-seat configuration complete with custom workstations, phone equipment and office furnishings. Comdisco can rapidly deliver the mobile facility and configure it to meet an organization's specific business needs, complete with the hardware, including PCs, phones, T1 line access and dishes for networks to help a company resume operations as quickly as possible.

If a company calls on Comdisco, there is an initial declaration fee to implement their assistance and a daily usage fee. Comdisco works with organizations of all sizes, working with them to design a custom solution to match that company's specific business needs. Customers sign contracts and pay a monthly subscription fee that can range from a couple hundred dollars to hundreds of thousands dollars per month depending on the size and complexity of the technology being recovered as well as the organization's recovery time requirements....."