7th Global Conference on Business & EconomicsISBN : 978-0-9742114-9-7
Lessons Learned From Hurricane Katrina for US Higher Education
Ruben Xing
Montclair State University, USA
Tel: (973)655-4252
Lessons Learned From Hurricane Katrina for US Higher Education
Ruben Xing
Montclair State University, USA
ABSTRACT
Most US universities planned and prepared their disaster recovery (DR) and business continuity strategies after September 11 attack. The recent devastating hurricanes caused unprecedented damages for many campuses in the gulf coast.Some of their plans worked and some of them failed. For the lessons learned, US higher educationsnationwide shouldreexamine, re-plan, and redesign their DR strategies and all the procedures.
1. LESSONS LEARNED FROM 9.11 CROSS THE NATION
Property Loss
The whole world was shocked by the tragic events of September 11thin 2001. Just few moths ago, Hurricane Katrina and Rita devastated entire New Orleans city, ravaged the Southern United States,most GulfCoastand a large area of Texas. In addition to several thousands of casualties suffered from both disasters, hundreds of business organizations were wiped out in just few hours. Business giants and tech industry found themselves so vulnerable and paralyzed easily with no offices, telephones, email, or computers. While 250 of a total of 450 World Trade Center (WTC) tenants declared business disaster and 150 went out of business. Price-Waterhouse- Coopers estimated later that the overall WTC losses were approximately $40.2B. And the losses from business interrupted economic activity caused by Hurricane Katrina exceed $100M per day. This natural disaster is considered the costliest one ever - with the estimated total loss of 125B and insurance industry slice of the clean-up bill reaching as much as US$60B.
Information Systems Disasters
Destroyed property losses are comparably recoverable. However, business interruption losses, especially losing critical data, are immeasurable and usually non-recoverable Above 30% of WTC losses in 2001 represent business interruption costs. Lighthouse Technology reported that annual data loss to PCs cost US businesses $11.8B in 1998. In 2001, the business downtime caused loss was $1,010,536 per hour for an average of all industries. Meta Group reported this statistic in major financial industries reached $16.6M per hour in 2003. A study shows, as much as 60% of corporate data resides unprotected on PC desktops and laptops, and more than 109,000 TB of unique enterprise PC data are not being regularly backed up.
(Please Refer to Figure 1 in Appendix)
One asset many tenants avoided losing in the September 11th disaster was electronic data. As Morgan Stanley’s technology team characterized, the WTC as “probably one of the best prepared office facilities from a systems and data recovery perspective” (Wired News, 2001). The road to the disaster recoveries of the WTC tragedy was paved in the wake of 1993 WTC bombing. It led most businesses firms to develop detailed plans that were crisply executed when disaster struck.
After the WTC attack, most people recognize the truth - no business is really secure until its data are recorded on two different sets of media. Nowadays, disaster recovery and crisis management are becoming more challenging, reflecting the rapidly growing complexity of information systems technology and pressures to keep e-commerce systems highly available 24 hours a day. Plus, an effective DRP must be able to protect a business from losing partial or entire physical locations within the shortest time as possible. Figure-1 shows the average losses from system downtime.
Besides disaster losses, information system vulnerability and site outage caused more losses. The Internet Worm of 1988 is a classic lesson of a security threat that used the Internet as a vehicle to travel around the world within minutes. It infected 6200 computers (10 percent of the computers on the Internet at the time).
Industry experts put the total cost of the worm closer to $100M. Due to web service outage, auction sites lost $2.8M and brokerage sites lost $5.2M within 8 business hours. Meanwhile, slow performance costs e-commerce site $362M per month, according to The Industry Standard. EBAY, a major online auction firm, was crippled for nearly 24 hours due to site outage in June 1999. This led to a 26% stock drop in the company’s stock price. In August 1999, a software glitch caused 3,000 MCI WorldCom customers to lose Internet access. The outage lasted for about 10 days. A denial-of-service worm attacked Yahoo in February 2000. It took the site offline for at least 3 hours and cost millions of dollars in loss.
(Please Refer to Figure 2 in Appendix)
2. GESTURES FROM U.S HIGHER EDUCATIONS FACING DISASTERS
History shows, campus disasters occurred frequently throughout the country. The following table only summarized some typical ones within past 15 years:
DRP Imminence Recognitions
Higher education, however, disaster recovery planning is not well prepared at many universities, and enterprise-wide defensive strategies are generally lacking if they even exist. According to Info-Tech’s DRP in the Education Sector 2005 Benchmarking Report, a surprising 47% of universities and colleges currently have no disaster recovery plan in place. These institutions, do, however, acknowledge the importance of having such a plan: 68% of them say they are currently in the process of planning. Unfortunately, it may be some time before many college and university DRPs see implementation: 32% of schools with no current plan concede it may be up to three years before they have one in place, according to the report, and that may be because security and end user support are higher IT priorities than disaster recovery—just a notch above the categories of network/LAN/WAN, Web site and IT governance. But the good news is that, according to Info-Tech, among the 53% of schools currently with a plan in force, a whopping 86% are improving that plan. Figure-3 shows a little progress on IT disaster recovery planning for US higher educations in recent years.
(Please Refer to Figure 3 in the Appendix)
Reluctancefor taking DRP actions at campuses is mainly because of traditional cultural and political factors. Also schools create difficult and complex settings to enact change of any kind. Plus, the mission (an emphasis on teaching and learning) at academic institutions is fundamentally different. DRP and “information security deficiencies at educational institutions are more pervasive than that at profit driven corporations. “New efforts to move to an improved level (of DRP) are not easy and don’t happen quickly.” (Stephen Reeder, Improve Higher Education Information Security, May 23, 2002; ) Although “most universities are doing at least the minimum necessary to protect computer data, … few have well-thought-out systems for recovery once a disaster happened.” (Dan Carnevale, “Preparing for Computer Disasters”, The Chronicle of Higher Education, 2/28, 2003) John W. Toigo, CEO of Toigo Partners International, says, “data recovery often isn’t a big priority for institutions unless people there have already witnessed some major catastrophe.”
3. DRP STRATEGY RECONSIDERATIONS
In the aftermath of Hurricane Katrina, universities in the gulf coast were devastated. State higher-education officials have estimated that the hurricane cost Mississippi's public and private colleges at least $674-million. That includes $495-million to repair and rebuild campus facilities. Table-2 listed rough losses from most of universities.
(Please Refer to Table 2 in the Appendix)
Non-Recoverable Academic Resources Loss From Hurricanes
Beside huge number of financial losses from these universities, the hurricanes forced full institutional closures longer than any on record, and it ravaged a whole region of colleges and universities.The wrath of Hurricane Katrina wreaked billions of dollars in damage and claimed hundreds, maybe thousands of lives. In many universities, decades of research, some scientists find, and took the lives of thousands of lab animals were destroyed. For many researchers at universities affected by the storm, it also destroyed or menaced their lives' work.
Many researchers in New Orleans did not know exactly how much they lost. Most of those at TulaneUniversityand at LouisianaStateUniversity's HealthSciencesCenter have not been able to return to their labs to survey the damage or recover specimens. Administrators at both universities have been collecting lists of the most important research materials and working to ferry them out. As many of professors stated "it's tens of millions of dollars of research. It's over 20 years of work. And it's all going to be gone." Cecil D. Burge, vice president for research and economic development, estimates that 25 faculty members lost specimens and collections, some worth millions of dollars. Most of the scientists who lost materials were at the university's Gulf Coast Research Laboratory, in Ocean Springs, close to the beach.
The hurricane's intensity vanquished researchers in other tragic ways. Those who stayed at the LouisianaStateUniversity medical school during the storm and subsequent flood ultimately made the difficult decision to euthanize the surviving animals before evacuating. All of the HealthSciencesCenter's 8,000 experimental animals died or were euthanized -- including the rhesus monkeys belonging to Joseph M. Moerschbaecher, vice chancellor for academic affairs, who was one of the last to evacuate the medical school. "There was nothing we could do" for the animals, he says. He estimates that 150 to 175 faculty members lost research materials. That work often involved animals that perished. At Tulane University, the Louisiana State University Health Sciences Center, and the University of Southern Mississippi, Katrina destroyed thousands of animals -- including fruit flies, mice, rabbits, dogs, and primates -- and materials ranging from tissue samples to cell lines to microorganisms like yeast and bacteria. In some cases, the damage encompassed years, even decades of work. At Tulane, that included invaluable blood samples from the Bogalusa Heart Study, which has been tracing heart disease in thousands of people since 1972, according to Paul K. Whelton, senior vice president for health sciences (The Chronicle of Higher Education, Sept. 21 2005)
In addition to academic losses, information systems and IT resources were damaged badly too.Hurricane Katrina has destroyed tens of thousands of PCs and servers. In New Orleans alone many thousands of computer installations will have been flooded out or damaged. The local IT networking infrastructure will have been severely degraded. It includes the New Orleans mayor’s office IT facility. This large scale disaster will bring data protection and IT disaster recovery concerns to the fore. Every business that has an IT data centre in the region, now non-operational because of the hurricane floods, but which has a DR facility will now be thanking their lucky stars. They were wise to invest in a DR facility. Those that didn't will either be wishing they had or not be concerned at all - because their entire business has gone.
Worst-CaseScenario
As a typical case, we looked closely at the largest campus in New Orleans - TulaneUniversity which had a disaster plan. Table-3 summarized what of it worked, what of it didn't.
(Please Refer to Table 3 in the Appendix)
Since many plans could not meet the emergency needs, several prepared measures, and procedures failed to be implemented in expected ways. Compare withthe assessment for what Tulane planned and what they prepared in Table-4.
(Please Refer to Table 4 in the Appendix)
4. MISSIONS FOR DISASTER RECOVERY RE-PLANNING
Hurricane Katrina was so unlike any disaster no one could have ever imagined. There were so many things that both business and schools could not have predicted or prepared for. They all had to rethink their plan in the middle of the disaster.We further studied more lessons learned from TulaneUniversity, summarized some major failures and lessons,reasons and advices for future plans.
(Please Refer to Tables 4 & 5 in the Appendix)
Critical Campus Resources Redundancy
From what we listed above, it is clear for our colleges and universities that disaster recovery strategies must include maintaining continuity for the scholarly work of faculty and researchers. The importance of the research enterprise calls for paying significant attention to both academic and IT personnel, also preservation of the institution’s digital assets, particularly those that are unique to the campus. Campus systems that store these unique digital assets can be found in central IT units and in almost every academic and administrative department. Systems that hold universities’ assets including not just those set up specifically to store individual files but also those for campus-wide e-mail, courseware, e-portfolio, and other learning systems.
(Please Refer to Figure 4 in the Appendix)
Since September 11, most companies and campuses recognized the truth - no business is really secure until its data are recorded on two different sets of media. When a primary site gets seriously affected or destroyed, in order to recover essential business data, or service availability, either a “cold-site”, “warm-site” or “hot-site” is needed. The critical data are backed up at a redundant storage site, and can be restored within different time period. Zero-tolerance requires a “hot-site” (standby-failover-site) mirrored with the primary site. So it is also called “site-mirroring”, or “real-time-backup.” Figure-4 above shows this strategy designed by Cisco. Each site is configured with Storage Area Network (SAN), Direct Attached Storage (DAS), or Network Attached Storage (NAS) based storage networks and connected through fast link media. Global load balancing, and global replication systems are also needed. When the primary site failed, the hot-site could quickly failover with little interruption break. This feature allows companies to tap into a live system with little or no interruption when disaster strikes.
However, the hurricane’s aftermath revealed some weaknesses for this most updated model. Some firms had stored backup tapes at their offsite locations, but these sites were also in New Orleans and couldn’t be accessed. So many campuses plan to look into doing online backup. For those who had started online backup in advance of the storm, most of their data are safe. This is a lesson learned, and brought a major challenge facing the up-to-date,multi-site-data-recovery-strategy. To deal with worse case in future, how far between each site should be safe enough remains to further study.
Reliability of Telecommunications
Telecommunications, e-mails and Internet access issues have risen to the top of the priority list for most IT managers. New technologies and improved emergency procedures could help avoid prolonged communications outages. Universities should keep a full-time connection with their state government emergency communication systems. However, the system became overloaded as the state moved large numbers of its responders into the affected area. This overload, combined with the loss of public network connectivity failures caused system degradation and impaired communications. Although there is a common operating system between campuses and state emergency operation center, the statewide system must be expanded to assure collaborative information sharing in a common situational awareness environment among local, state, and federal agencies. Degraded communications among the emergency responder community severely interfered with their ability to deliver necessary services.
Land-line based telecom systems are being used in most gulf coast area. People were expecting cell phone to help their communications during the disaster. However, only a small portion of cellular call is carried over wireless links. Cellular call is actually carried over cell sites connecting to switching station via T1 and fiber-optic cables. When flooding brought most dramatic effect on land lines, “about 1million land-line phones were knocked out of service in gulf area” according to BellSouth. So the wireless communications are not fully wireless. This challenge will be one of major concerns facing telecom companies and all the campuses’ future recovery plan.
From developer and vendor sites, since most telecom companies are all relaying on backup generators or portable ones carried on panel trucks along with cellular transceivers, the telecommunication services, in particular the cellular and other kinds of wireless services still have difficulties to gain access to most parts of New Orleans. Theprocess to repair, troubleshoot or replace of vendor’s networks became rather slow. It made very difficult for the universities in this area to resume their regular communication systems. Universities are working with telecom companies to mount more rooftop cell sites and increase the power.
DRP Drills
No one can tell how well their DR plans will work until the plans get thoroughly tested. The schools fully or partially rely on outsourcing for DRP support, or never drilled their own DR plans, are taking a big risk. They may never know whether their data and IT systems can be correctly backed up/restored on time until an incident strikes. An IBM storage management director says, “During those drills, officials will be able to tell whether the stored data are easily accessible and compatible with other equipment. Drills also make people realize that they need spare copies of the software to run the data they’ve been storing” (Chronicle H.E, 2/28, 2003). A survey shows that drills are performed very differently and informally at most schools. Some schools run limited drills when they first set up a new system, and some play drills at backup sites occasionally, testing loaded data from tape, and running production jobs. Several universities never took the drills because of unaffordable costs and disruptive activities.