Project Closeout Report
Project Name / Enterprise Operations Business Continuity – Disaster Recovery Assessment and Feasibility Study / Date / 7/20/2010Executive Sponsor / DoIT Secretary Marlin Mackey
DoIT Deputy Secretary Bob Mayer / Lead Agency / Department of Information Technology (DoIT)
Project Manager / DoIT Mary Wanda Anaya / Agency Code / 361
Project Description (Provide a brief description and purpose for this project)
The Department of Information Technology received funding for a Disaster Recovery (DR) Assessment and Feasibility Study to determine the best approach for redundancy for its most critical Information Technology (IT) based services and applications. The purpose of this project was to determine the most cost effective means of providing this service.
The Initiation Phase of this project includes three visits to the state of Colorado, Arizona, and Oregon to understand and see their operating solutions to providing redundancy within their data centers; to include business continuity and consolidation efforts. In conjunction with the out of state site visits, selected vendors that operate commercial data centers which provide cold, warm and hot recovery services were toured. In-state commercial data centers were also toured. The sites visits included an assessment of their offering for Business Continuity and DR management services.
The Planning Phase of the project included meeting with business partners to define the needs for the DR Assessment and Feasibility Study. This portion of the project was developed in house. A high level BIA was conducted. The development of the scope of work for the study and the established requirements for research DR services was completed.
The Implementation Phase of this project includes a DR Assessment and Feasibility Study. The DR Assessment included a Threat and Risk Assessment. The Feasibility Study determined the top critical applications for the State of New Mexico and provided the DR recommended model per critical application. The top twelve (12) critical applications spanned across ten (10) agencies. Each agency was interviewed by the project team to assess their Business Continuity and Disaster Recovery preparedness. Within the study are also recommendations for possible DR efforts for consolidating applications and platforms for greater cost savings and operating efficiency.
In order for Business Continuity to be successful within the state it is essential that agencies have at lease one staff member that is knowledgeable in Business Continuity in order for the agency to evaluate the recommendations brought forth by the Feasibility Study. Two (2) Business Continuity Awareness classes and one Business Continuity Planning class were provided to the agencies. Some agencies took advantage of the training and sent staff to all classes. The Office of Business Continuity staff also attended formal training and received the Certified Business Resilience Manger (CBRM).
Schedule and Budget
Planned Start Date / December 1, 2008 / Actual Start Date / December 1, 2008
Planned End Date / June 30, 2009 / Actual End Date / June 30, 2010
Planned Cost: (Budget) / $ 250,000.00 / Actual Cost: (Total) / $ 249,793.31
§ Professional Services / $ 200,000.00 / § Professional Services / $ 199,974.80
§ Hardware / $ 0.00 / § Hardware / $ 0.00
§ Software / $ 0.00 / § Software / $ 0.00
§ Network / $ 0.00 / § Network / $ 0.00
§ Other / $ 50,000.00 / § Other / $ 49,818.51
Appropriation History (Include all Funding sources, e.g. Federal, State, County, Municipal laws or grants)
Fiscal Year
/ Amount / Funding Source(s)2009 / $250,000.00 / Laws 2008, Ch. 3, Section 7(13)
For an assessment and feasibility study for redundancy of the most critical information technology-bases services and applications.
Scope Verification /
Requirements Review
/ Yes / No /Explanation/Notes
/Were the project objectives (expected outcomes) accomplished? / X
Were all Deliverables submitted and accepted? / X
Did the IV&V vendor verify that all deliverables met the requirements? / X / IV&V exception for study.
DoIT Quality Assurance provided.
Have all contracts been closed? / X
Have all final payments been made (i.e., invoices paid) / X
Has adequate knowledge transfer been completed? / X
Transition to Operations: (Describe agency plan to migrate project solution to production. Include DoIT impact if different than previous report)
The information gathered within the study provided the requirements that formulated the mandatory specifications for the scope of work for the DoIT Disaster Recovery and Data Resilience Data Center Site(s) RFP.
Maintenance/Operations
/ Yes / No /
Explanation/Notes
Are there recurring maintenance/operational costs for the product/service? / X / $ per YearAre there any recommended enhancements or updates? / X / (Attach comments)
Funding source for maintenance/operational costs? N/A (Describe)
Business Performance Measures (Complete for all phases)
Comments:
Phases / Completion Date / Goals/Objectives / Amount / Results
Initiation:
/ May 11, 2009 / Business Objective 3 –Ability to resume critical business functions, i.e. business continuity
Business Objective 4 –
Identify cold, warm, hot sites and mange DR services.
Business Objective 6 – enable an individual from each agency to work directly with the OBC who will be responsible for departmental business continuity and recovery.
Technical Objective 1 – Evaluate the impact to DoIT business / operational functions resulting from a disaster
Project Goals
1. Other State Government Site Visits
2. Out-of-State Commercial Site Visits
3. In-State Commercial Site Visits
4. Education Site Visits
5. Business Continuity Agency/DoIT Staff Training
6. Business Continuity Staff Training **
** Training was moved to April 2010 (due to budgets were placed on hold) / $49,818.51 / 1. January/February 2009 – New Mexico
§ Northrop Grumman
§ BigByte
§ Oso Grande
§ Qwest
2. March 2009 - Colorado
§ State of Colorado
§ eFort DR DC
§ IBM BC Resilience Center
§ Qwest DR Center
§ Cisco Cyber Center
§ State of Michigan(ph)
3. May 2009 - Arizona
§ State of Arizona
§ I/O DC
§ SunGard DR Site
§ AZ State Univ 3DC
4. August 2009 - Oregon
§ State of Oregon
§ CIO Advisory Committee
§ Intel – Lights Out
§ Opus Interactive
§ Infinity Internet
5. April 2009 – Washington
§ Business Continuity Training
§ Business Continuity Certification
6. April/ May 2010 – New Mexico
§ Business Continuity Awareness Training
§ Business Continuity Plan Training
Planning:
/ March 12, 2009 / Project Goals1. Project Plan
2. Define Needs
3. Develop Scope of Work for DR Assessment and Feasibility Study / $ 0.00 / March 12, 2009
POD Inc. contract:
Activities:
1. Threat and Risk Assessment.
2. Critical Applications Assessment
3. Overall Disaster Recovery Recommendations
Implementation: / March 30, 2010 / Business Objective 1 – Identify the state’s mission critical systems.
Technical Objective 2 – Define the amount of sustainable time from outage to recovery of IT infrastructure
Technical Objective 3 – IT Recoverability Assessment / Strategy recommendations – Evaluate DoIT’s data center’s recovery capability using current processes and procedures for services above. Recommended improvements will be made to meet the Recovery Point and Recovery Time Objectives.
Project Goals
POD Inc. Contract
DR Assessment and Feasibility Study
1. Conduct a Threat and Risk Assessment
2. Conduct a Critical Application Assessment
3. Provide Overall Disaster Recovery Recommendations. / $ 199,974.80 / Del #1 Discovery Document
Del #2 Threat Analysis Report
Del #3 Risk Analysis Report
Del #4 Threat & Risk Recommendations Document
Del #5 Critical Applications Determination
Del #6 Evaluate Selected agencies BC & DR Plans
Del #7 Evaluate Twelve Critical Applications Architecture
Del #8 Critical Applications Recommendations
Del #9 Determine Disaster Recovery efforts for Developing Linkage of Like Applications and Platforms Report
Del #10 Overall Disaster Recovery Recommendations
Closeout: / June 30, 2010 / Business Objective 2 –
Make accessible the critical and vital computer production environments for each agency within the timeframe specified by each agency
Business Objective 5 –
Provide business systems that support and enhance the efficiency of State Agencies and sustain their ability to deliver services to the citizens of New Mexico
Technical Objective 4 – Continue to implement Redundant Network Recovery strategies and develop documentation to support the switching of systems to the backup networks that will meet Business/Operational recovery requirements. / $0.00 / RFP# 00-361-00-01416
DoIT Disaster Recovery and Data Resilience Data Center Site(s)
Category-1 Resilience Data Center Site (GOLD)
§ Production/Failover
§ Available 24x7x365
Category-2 Hot Data Center Site (SILVER)
§ Host Equipment
§ Operating System
§ Application Software
§ Copy of Data – test
§ Available within 8 hrs
Category-3 Warm Data Center Site (BRONZE)
§ Racks, Power, Data
§ Available within 24 hrs
Category-4 Cold Data Center Site (PAPER)
§ only floor space
§ Available within 72 hrs
Lessons Learned
1. When the product of the project is a study, it is critical to the project to include a requirement in the contract for a technical writer. The reports that were initially received from the contractor were not written well. On the first half of the contract the Office of Business Continuity exhausted resources and time, working with the contractor’s project manager to clean up the reports. This issue with the quality of the reports was brought to the contractor’s attention. The contractor did respond and restructured the process to route reports to a more qualified individual with technical writing skills before delivering the reports as a final product.
2. When critical information is gathered through an interview process, note gathering should extend to include voice recording to assure all important information is documented. The contractor had one of their staff members responsible for taking notes at each agency interview, however when the notes were reviewed on the contractor’s share point site the notes were minimal and did not record all the information discussed.
3. Even in small projects it is difficult for the Project Manager and the Project Team Leader to be the same individual. When the roles of Project Manager and Team Leader are the same person it becomes increasingly hard to direct the project and yet meet all the project management requirements. The resources to the project were also impacted by the reassignment of a key team member to another project.
4. Continuous Business Continuity training is required within the State of New Mexico to educate the agencies that Business Continuity is not only Disaster Recovery and an IT responsibility, but an ongoing business process to continue providing critical services.
IT System Analysis
On this document, or as an attachment, provide a summary response, including changes, to the following IT infrastructure topics relating to this project:This project was a study and did not impact the following;
· Describe or estimate this project’s impact on the State Datacenter infrastructure.
o Hardware (List type of hardware anticipated. Keep in mind the State Datacenter may have pre-built hardware stacks available):
o Network (Include Diagram):
o Software / Applications (Provide application schematic if available):
° Hosting Considerations (If not hosted at the State Datacenter describe your strategy to host at the State Datacenter):
Business Continuity Strategy
On this document, or as an attachment, provide a summary response, including changes, of your business Continuity Strategy.· Emergency and Disaster Management
Business Continuity Management
PurposeBusiness Continuity Management will ensure that the appropriate level of administrative management of responsibility is in place to sustain the operation of Information Technology critical business services following a major disaster or emergency. To ensure information technology support services to State government agencies with minimal disruption due to disasters or unforeseen events that would impact the states’ ability to service the citizens of New Mexico.
Policy Statement
The Office of Business Continuity and Disaster Recovery, under the direction of the Department, shall maintain and test a Business Continuity Plan. The plan will support the continuity of operation of the Departments information technology, to include operations that the Department supports on behalf of other departments or external entities.
Procedure
The Office of Business Continuity has the primary leadership responsibility to identify risks and to determine what impact these risks have to business operations. The Department’s Management Team shall plan for business continuity based on these risks and document recovery strategies and procedures in a defined business recovery plan that is reviewed, approved, and updated on an annual basis. The plan includes all divisions: business, technology and operational support. All divisions perform functions critical to sustaining service delivery. Responsibilities of Division Directors, IT Managers, and IT Supervisors (information owners) include but are not limited to:
· Identification and prioritization of critical business processes.
· Regular assessment of the potential impact of various types of unforeseen events /disasters.
· Definition of responsibilities and emergency arrangements.
· Documentation of all procedures and responsibilities.
· Communication of business continuity and recovery plans to all necessary individuals.
· Regular testing of business continuity and recovery plans.
· Regular review of business continuity and recovery plans to ensure they are correct, complete and up-to-date.
The specific rules and procedures guiding the responsibilities and the actions to be taken in the event of a disaster are specified in the Business Continuity Plan. The plan is assembled from the individual section plans under the direction of the Office of Business Continuity. The Business Continuity Plan shall include the following types of activities:
· Procedures and criteria for Disaster Declaration that will activate the Disaster Recovery Plan. Include the process for activating the hot site which will restore computer systems and the statewide network within forty-eight (48) hours.
· Define a notification process of all responsible individuals to include:
§ Incident Manager;
§ Office of Business Continuity;
§ Office of Security;
§ Division Directors;
§ Public Information Officer;
§ Line Managers
· Define a Damage Assessment Team with procedures for making recommendations to Executive Management regarding the extent of the damage and whether the facilities can be used safely in a reasonable amount of time or whether the hot site should be notified.