In pursuit of consolidation

Michigan is proud of its three state-of-the-art data centers, which now handle most of the data processing for the state’s 19 agencies. One center is for data processing. Another center handles emergency services such as enhanced 911 communications. The third one is a disaster recovery center, which the other two centers could use if a disaster disrupted their operations. If you’re a state official and would like to visit, chances are the Michigan Department of Information Technology would be happy to accommodate you.But if you had asked for a tour of the state’s data processing facilities as recently as five years ago, you might not have received as hearty a welcome as you would today. A complete tour would have required a bit of traveling. “We had 19 different centers, each with varying degrees of efficiency,” said Ken Theis, chief deputy director of the IT department. Although some of the data centers were clean and well-equipped with modern ventilation, dual-power supplies and state-of-the-art servers, others were unprotected. Some had backup batteries that had been burned out and power cords that were plugged directly into wall sockets. Others relied on ceiling fans for ventilation. Many were cluttered with strands of cables that were difficult to track. One center had electric cables running along the outside of the building. Anyone with a cable cutter could have shut down the entire data center operation for hours or longer.In addition, many of the centers used unique tools, which necessitated specific training at those sites. And in many instances, servers and storage were underused, adding unnecessary costs to the state’s IT budget.If that weren’t enough to get data center managers to cry for help, budget problems in 2001 forced the state to cut one-third of its data center employees. “The old system was too resource- and labor-intensive,” said Patrick Hale, the IT department’s director of infrastructure services. Michigan’s answer to the problem was to close 17 data centers. Although merging services isn’t a panacea, it can provide efficiencies of scale; better use of resources; and improved management, customer support and uptime. But the challenges to achieving those benefits aren’t trivial. For example, most states that are considering a consolidation project will have to work hard to achieve buy-in from clients, executives and IT employees. Even though many of Michigan’s data center managers were in distress, they weren’t overjoyed at the prospect of transferring their data center activities to a central megacenter. “There are always fiefdoms,” Hale said. Hale and Theis said IT departments must earn some street credibility if they hope to garner the support they need to consolidate data centers. “You have to prove yourself, provide evidence that the new center will be better than the previous ones and that there won’t be any problems with the move,” Theis said. Michigan’s IT department leaders reasoned that if a picture is worth a thousand words, an actual visit combined with pictures could be many times more valuable.So the department bused clients and state officials to the new data center facilities and gave them VIP tours. Then they showed them photos of their current centers — with the poor ventilation, power hazards and disorganization. Theis and Hale handed out cards with their home telephone numbers and invited people to call them with questions at any time. “It was a watershed event,” Hale said. “Most people had no idea how vulnerable their data centers and their data were.”And realizing that no one can resist a bargain, the IT department made a commitment to charge agencies no more for their data processing than they were paying and promising an eventual, if not immediate, reduction in data center costs. Nevertheless, data center managers still had concerns about the consolidation plan. Theis said he knew that a misstep could thwart the whole project. He also knew that if one data center went through a successful conversion, word of that success would spread.  To say that Michigan’s IT department took the necessary time to do the conversions right is a bit of an understatement. Officials began the consolidation of IT resources during the administration of former Gov. John Engler, a Republican, in the early 1990s. The data center consolidation project began under the administration of Democratic Gov. Jennifer Granholm. If someone had to make a trade-off between doing it right and meeting a deadline, doing it right won, Theis said. “We were not afraid to extend schedules, even going 60 days beyond expected switchover, if we had any questions about getting it right.” To move data to the consolidated center, the IT department extended the central data center’s storage-area network fabric to the data centers it was replacing. Then it replicated the data onto the SAN while the old center continued running. Only after the data was successfully transferred and tested did officials shift data processing operations to the new center. Theis said the state completed 17 conversions without a single hour of downtime. The IT department announced in February the closing of three more data centers.Since consolidating the work of 17 data centers, the IT department has standardized service levels for all state users. “We can view every node on the network and get alerts when anything needs attention,” Theis said.Michigan’s IT officials have not fully calculated the return on their investment in consolidation. The only hard numbers they have so far are based on the results of consolidating messaging services. Officials said the project saved the state more than $600,000 this year. They estimate savings of more than $11 million in four years. The messaging consolidation brought increased service levels and 50 percent improvement in performance.But Michigan officials said many benefits of the consolidation, such as greater efficiency, are difficult to measure. For example, many of the state’s data centers followed the conventional practice of having servers running a single application. In one instance, the IT department replaced 40 servers with one high-end Sun Microsystems server. Another benefit is more efficient storage. Michigan installed a SAN that spans all three new data centers. The IT department charges agencies a per-gigabyte fee for storage. Agencies’ storage costs have been dropping because of greater storage efficiency, although IT department officials said they do not yet know at what level those fees will stabilize.  In Michigan, savings from consolidation are already manifest in messaging services, and benefits have shown themselves in virtually all systems. “Our consolidation [is] a win for everyone,” Theis said. “Costs are down, service is up, and we have a much safer center.”