<Branch>
Disaster Recovery Plan /
<TEAM NAME> / <SERVICE NAME(S)> /
Document / SEQUENCE, Recovery Time Objective (RTO)
The “Total” fields are automatically calculated using a formula. If changes are made to the estimate figures click on the total figure and press the F9 button to calculate an updated total. To update all fields in the document press CTRL+A and then F9. Please verify all calculations in the relevant sections.
Version: 00.01
Date last updated:
Last updated by:
Date last printed:
Total pages: 165
Total words: 18223 / Operational Level Event (Section 7.1):
Total (if restore): 595min (9.92hr) (0.41days)
Total (if rebuild): 805min (13.42hr) (0.56days)
Total Loss of Service Event (Section 7.2):
Total (if restore): 45235min (753.92hr) (31.41days)
Total (if rebuild): 45445min (757.42hr) (31.56days)
Location: <Location>
For updates to this plan contact:
DRP Coordinator:______
Phone:000000-0000
Email:
1General
Asset Type: /Asset / Service: /
Owner: /
Disaster Recovery Leader: /
Disaster Recovery Alternate Leader: /
Plan Update Contact: /
Information Security Classification: / High Sensitivity
1.1Approved By
Disaster Recovery Leader, <TEAM NAME RESPONSIBLE FOR SERVICE>:Name:
Signature: ______/ Date Signed (yyyy mm dd)
Director, Technical Services, <Branch>:
Name:
Signature: ______/ Date Signed (yyyy mm dd)
Service Delivery Manager, <Branch>:
Name:
Signature: ______/ Date Signed (yyyy mm dd)
Director, Service Operations, <Branch>:
Name:
Signature: ______/ Date Signed (yyyy mm dd)
Director, Identity Architecture, <Branch>:
Name:
Signature: ______/ Date Signed (yyyy mm dd)
Director, Program Development, <Branch>:
Name:
Signature: ______/ Date Signed (yyyy mm dd)
Director & Financial Strategist, <Branch>::
Name:
Signature: ______/ Date Signed (yyyy mm dd)
Executive Director, <Branch>:
Name:
Signature: ______/ Date Signed (yyyy mm dd)
1.2Document Amendment History
DRP Workbook Serial Number______
Amendment Number(yyyy.##) / Date Amended
(yyyy/mm/dd) / Amendment Detail
(e.g. Section/Page/Tab Amended) / Issued by
1.3Table of Contents
SECTION PAGE
1General
1.1Approved By
1.2Document Amendment History
1.3Table of Contents
2Document Purpose
2.1Policy Statement
2.2Objectives
2.3Amendments
2.4Documentation Storage
3Business Impact
3.1Client, Ministry, or BPS
3.2Partners
3.3Core Shared Services Internal
3.3.1Core Shared Services which consume and depend on the <Service Name> service
3.3.2Core Shared Services which the <Service Name> service consumes and depends on
3.4Risk or Recovery Delay Exposures (High)
4Requirements
4.1Recovery Dependencies and Assumptions
4.1.1Application & Tools
4.1.2Server
4.1.3Network
4.1.4People
4.1.5Suppliers
4.1.6Building
4.2Required Applications / Tools
4.3Vital Records / Manuals
4.4Escrows
4.4.1Definition
4.4.2Escrows Required for Recovery
5Incident Notifications
5.1Notification linkages for activation – Priority 1 or 2 Incidents
5.2Notification List – Priority 1 or 2 Incidents
6Recovery Preparation
6.1Disaster Declaration Process
6.2Disaster Declaration Triggers
6.3General Recovery Preparation and Assessments
7Service Recovery Strategies
7.1Operational Level Event (Single incident impacting capacity or performance)
7.2Total Loss of Service Event
7.3Scenarios Overview Grid
7.3.1Application Scenarios List
7.3.2Server Scenarios List
7.3.3Network Scenarios List
7.3.4People Scenarios List
7.3.5Suppliers Scenarios List
7.3.6Building Scenarios List
8Recovery Reporting and Escalation
8.1Reporting During Recovery
8.2Escalation Process
8.3Incident Management Priority Levels
Priority Levels
8.4Post Incident Reporting
9Stand Down
9.1Stand Down Process
9.2Strategy to Migrate Back to Production Environment
Appendix A – Contact Information
A-1 Disaster Recovery Team
A-2 <Branch> Core Shared Service Owners
A-3 General Internal Contacts
A-4 Staff authorized to draw escrows
A-5 Client Contacts
Ministries
Broader Public Sector Organizations
A-6 Contractors, Vendors, Suppliers
A-7 Partners
A-8 Websites for emergency and disaster alerts and bulletins
A-9 Information Security Contacts
<Branch> Security Team
Information Security Branch – Security Investigations
Ministry Information Security Officers
Appendix B – DRP Distribution List
Appendix C – Risk Rating Table
Appendix D – Harm Reference Table
Appendix E – Databases
Appendix F – Server List
Service List – Condensed – By server
Server List – Detailed - By service
Appendix G – Architectural Diagrams
Appendix H – Server Re-build Steps
H-1 <Service Name>
Prerequisites
Install
Configurations
Shell or Desktop Setup
H-2 <Service Name>
Prerequisites
Install
Configurations
Shell or Desktop Setup
H-3 <Service Name>
Prerequisites
Install
Configurations
Shell or Desktop Setup
Appendix I - Client, Ministry, and BPS Reference Application Lists
Mission Critical List
Business Priority List
Appendix J: Reference material from <Branch> consolidated BCP - Essential business processes and recovery time objectives (RTO)
Appendix K: Emergency Changes and Recovery Management
CHANGE MANAGEMENT
Change Management reports
Emergency Changes and Recovery Management
Infrastructure Changes on an Emergency Basis
Communication of Emergency Change
Technical Information Bulletins
Timeliness of Change Management
Requests for Special Processing (RSPs)
Best practices about Change
Security Classification: Choose an item / Disaster Recovery PlanSection: Service Recovery Strategies / Page 1 of 79
/ <TEAM NAME RESPONSIBLE FOR SERVICE> / <Service Name>
<Branch>
2Document Purpose
This document details the Disaster Recovery Plan for the <SERVICE NAME> serviceoffered by the branchof the BC Office of the Chief Information Officer. This document is limited to topics in which <branchstaff or contractors might potentially become involved.
2.1Policy Statement
This plan is subject to compliance with the requirements outlined in Core Policy, Chapter 16, with respect to Disaster Recovery Plans. In addition, <branch management approved the following policy statement for the<branchdisaster recovery program:
- The branch shall develop a comprehensive IT Disaster Recovery Planning Workbook (DRPW) or Disaster Recovery Plan (DRP) for each <branch asset/service.
- DRPWs and DRPs should cover all essential and critical infrastructure elements, systems and networks, in accordance with key business activities.
- DRPs should align with the current Business Continuity Plan and Business Impact Analysis relevant to the branch.
- DRPswill be exercised at least once annually in a simulated environment to ensure that they can be implemented in emergency situations and that the management and staff understand how they are to be executed.
- All staff must be made aware of the Disaster Recovery Plans and their own respective roles.
- DRPs will be kept up to date to take into account changing circumstances.
2.2Objectives
The principal objective of the disaster recovery program is to develop, document and exercise a well-structured and easily understood plan which will help the program recover this service within the agreed upon time lines, from an unforeseen disaster or emergency which interrupts information systems and business operations. Additional objectives include:
- The need to ensure that all employees fully understand their roles in activating the plan.
- The need to ensure that operational policies are adhered to within all activities.
- The need to ensure that proposed contingency arrangements are cost-effective and best suited.
- The need to consider interdependencies and impacts, including business impacts on client organizations.
- Disaster recovery capabilities as applicable to key clients, partners, vendors and others.
2.3Amendments
It is important that the DRP amendment process be properly structured and version controlled. Whenever changes are proposed, they should be fully tested and appropriate revisions made to the plan and related training materials.
NOTE:
- Text in blue font (8pt) within sections indicates instructions for users of this template.
- Red text within sections indicates information that should be added or revised specific to this service/application.
2.4Documentation Storage
Password Escrow and copies of this workbook will be stored in secure locations as defined by the Ministry.
Location of <branch Password Escrow
Guard Station, <location>
Withdrawal is by authorization only.
Updated Escrows are submitted to:
<Name>
<Title>
Phone:
Email:
Location of Software License Information and Contract
Contract Location:
Withdrawal is by authorization only.
Contract Management Contact:
Phone:
Email:
Software License Key Location:
<LOCATION>
Software License Key Contact:
Disaster Recovery Lead
Location of Master copy of this DRP
Access by authorization only. A master online protected copy is stored on SharePoint at
Service Operations service DRP folder URL> established for this purpose.
The master printed hard copy, <DRP Serial # and version, is stored in a binder, along with an electronic copy on an encrypted USB stick, and is located securely with the Business Continuity Manager, Technology Solutions.
Distribution of workbook copies
Document Handling:
-Plan storage shall be in keeping with Information Security requirements, or as directed by the OCIO, especially given the sensitive nature of the information in the plan.
-No additional copies of this plan shall be made, copied, shared or retained, in whole or in part, without prior approval of the Executive Director, or as immediately required in order to support the activation of this plan.
-All document disposals shall be in keeping with expectations of secure and sensitive government documents.
A list of distribution members with signature lines will be detailed with numbered copies (versions). See Appendix B:
-Each member of the <branch management team will be issued the Encrypted Electronic copy of this plan to be securely filed at home.
-Designated members of the <branchmanagement team will be issued the hard copy of this plan to be securely filed at home.
-Each member of the <branch TEAM NAME>team will be issued the electronic copy of this plan to be securely managed at work.
-The Disaster Recovery Lead and Alternative Disaster Recovery Lead will be issued the hard copy of this plan to be securely managed at work.
-All members of the Disaster Recovery Team will be issued a hard copy of Appendix A (Contact Information) to be securely filed at home.
3Business Impact
The Harm Reference Table (Appendix D) and impact change over time are used to assess SLA or OLA allowable outage for each application listed. Likelihood, level of impact, and risk rating for each application is detailed in the Scenarios Overview Grid (Section 7.3).See Appendix A(A-5)for contact information.
Definitions:
* MC = Mission Critical / BP = Business Priority
- Mission critical: processes that, should they not be performed, could lead to loss of life (safety), personal hardship to citizens, major damage to the environment, or significant loss of revenue or assets.
- Business priority: processes that are not mission critical, but, should they not be performed, could lead to the loss of a major business function.
References:
It is recognized that clients’ applications could be impacted by a service outage.
In an effort to help guide and expedite service recovery, the reference lists provided in Appendix I were developed based on assumptions from limited information. The list has not been validated and may be incomplete or may contain errors. As this plan matures, it is intended that the list will improve in accuracy with input from the client.
When recovering from a service outage, recoveries will reference the list to assist in setting priority and assessing recovery actions.
For recoveries from a provincial disaster led by the Provincial Emergency Coordination Centre (PECC), especially where priority or resourcing conflicts occur, recovery actions and priorities may be influenced or directed by the PECC through established processes and reporting.
3.1Client, Ministry, or BPS
See Appendix A-5for contact information.
See Appendix I for the reference application lists.
3.2Partners
For each partner listed, list partner applications (one line per app) dependent on this service/application and what affect an outage would have, if any, on the partner application (dependency). State whether the partner application is classified as mission critical (MC) or Business Priority (BP) or Normal Business (blank) and state the Service Level Agreement (SLA) / Operational Level Agreement (OLA) Allowable Outage for the client application. Note: The example shown may not be appropriate for this service. Add or revise the list as needed.
In the event of an outage, the partner applications listed would likely be impacted. SLA / OLA Allowable Outageis determined by a partner representative. See Appendix A-7for contact information.
Partner Name (Acronym) / Application Full Name (Acronym) / Dependency / Rating(MC or BP or blank) / SLA / OLA Allowable Outage (hr)
*THIS TABLE HAS INTENTIONALLY BEEN LEFT BLANK
3.3Core Shared Services Internal
3.3.1Core Shared Services which consume and depend on the <Service Name> service
List each core shared service within<branch>, internal applications (one line per app) dependent on this service/application and what affect an outage would have, if any, on the internal application (dependency). State whether the internal application is deemed as mission critical (MC) or Business Priority (BP) or Normal Business (blank) and state the established SLA or OLA for the internal application. If no SLA or OLA is established, enter the word ‘none’.
In the event of an outage, the core shared services listed could be impacted. See Appendix A (A-2) for contact information. Core shared service owners are responsible for documenting their dependency on <Service Name>within their respective, BCP or DRP.
Core Shared Service Name / Application Full Name / Description of dependency on <Service Name> / * Rating(MC or BP or blank) / SLA / OLA
Allowable
Outage (hr)
<Core shared service name / <Core shared service application full name / acronym> / <Description> / MC / BP / none / 0hr (zero)
Exceptions:
Change Windows on Sundays
from 6am to 9am;
Scheduled changes;
Emergency Changes
Supporting Documents:
Change management – Emergency changes and Recovery Management:
or
Appendix K
3.3.2Core Shared Services which the <Service Name> service consumes and depends on
List each core shared service (one line per app) which the <Service Name> service is dependent on and what affect an outage of these services would have, if any, on the <Service Name> service. State whether the Core Shared Service is deemed as mission critical (MC) or Business Priority (BP) or Normal Business (blank) and state the established SLA or OLA for the internal application. If no SLA or OLA is established, enter the word ‘none’.
In the event of an outage, the core shared services listed could cause impacts to <Service Name>. See Appendix A (A-2) for contact information. Core shared service owners are responsible for documenting their dependency on <Service Name> within their respective, BCP or DRP.
Service Name / Application Full Name / Description of dependency on Core Shared Service / * Rating(MC or BP or blank) / SLA / OLA
Allowable
Outage (hr)
-<Service Name> / <Core shared service application full name / acronym> / Description> / MC / BP / none / 0hr (zero)
Exceptions:
Change Windows on Sundays
from 6am to 9am;
Scheduled changes;
Emergency Changes
Supporting Documents:
Change management – Emergency changes and Recovery Management:
or
Appendix K
Also see the Scenarios Overview Grid in this document.
3.4Risk or Recovery Delay Exposures (High)
List issues that if discovered during recovery might expose risks or delays to recovery time objectives.
In the event of an outage, the following risks will delay full recovery:
Description of Risk or Exposure / Level of Risk or Exposure(High /
Medium /
Low) / Level of Impact to Recovery
(1-5)
(Refer to Appendix C) / Anticipated delay (in hours)
Total / Optimal Details / Options in place
It is important to note that after-hours support and availability does have costs attached:
- ((IS## wage with 5 years or more of experience) + (temporary market adjustment if applicable)) * (number of hours needed for recovery and support activities (this assumes active work is needed as opposed to simply standby)) + (Applicable provisions of ARTICLE 16 covering overtime based on the current Master Collective Agreement).
- Hosting partner after hours costs as per contract.
Also see the Scenarios Overview Grid in this document.
4Requirements
This section includes assumptions and requirements for activation of this plan and service recovery.
4.1Recovery Dependencies and Assumptions
4.1.1Application & Tools
- At least one available staff has an “_a” IDIR domain administrative account with access to required <Service Name>servers.
- At least one available staff has an IDIR domain account with access to the required <Service Name>servers.
- All required software installation packages are available.
- Computer (physical or remote) access is available with working login to IDIR domain.
- Tools and relevant documentation are available.
4.1.2Server
- Related servers are online and accessible; or adequate replacement hardware is online and accessible.
- Computer (physical or remote) access is available with working login to the IDIR domain.
- Functioning backups are available containing required backup data.
4.1.3Network
- Communications means are available (e.g. phone, fax, email, IM).
- SPAN BC network is available with internet access.
- VPN is working and accessible.
- Firewall rules are correct.
4.1.4People
- Disaster recovery lead or alternate is available.
- Staff and contractors are reachable, available, and able to perform recovery tasks.
- Management is reachable, available, and able to provide recovery related decisions.
- Other supporting teams are responsive and effective.
- Recovery personnel have the training, knowledge, and skills required to perform necessary recoveries.
4.1.5Suppliers
- Vendors, contractors, suppliers, and hosting partner(s) are reachable and promptly respond to support requests.
4.1.6Building
- Required office buildings are accessible with required utilities functioning.
- Data Center facilities are operational and accessible with required utilities functioning.
4.2Required Applications / Tools
List any software or tools required to recover the service, software versions where possible, dependencies, and requirements. Thesetools and applications must be prepared, maintained and availablebefore executing the recovery of this service:
Application Full Name (Acronym) / Version Required / Dependency? / Requirement?SOFTWARE TOOL / <VERSION> / OTHER SOFTWARE APPLICATION IS DEPENDENT ON / WHY IS THIS NEEDED? / REQUIREMENT
SERVICE’S APPLICATION NAME / <VERSION> / OTHER SOFTWARE APPLICATION IS DEPENDENT ON / WHY IS THIS NEEDED? / REQUIREMENT
SOFTWARE TOOL / <VERSION> / OTHER SOFTWARE APPLICATION IS DEPENDENT ON / WHY IS THIS NEEDED? / REQUIREMENT
SERVICE’S APPLICATION NAME / <VERSION> / OTHER SOFTWARE APPLICATION IS DEPENDENT ON / WHY IS THIS NEEDED? / REQUIREMENT
4.3Vital Records / Manuals
IMPORTANT: Primary Location may be electronic, but must be physically available.
List minimum vital records and manuals required to do the recovery and their location.
These vital records and/or manuals are required to do the recovery (not just business):
Document Name / Author / Description of Contents(related requirements) / Primary Location / Secondary Location
<DOCUMENT NAME> / <VENDOR> / <Service Name>software download site (Requires valid credentials with the vendor. The Technical Recovery Lead has access. If no validcredentials are available then contact <VENDOR>to get credentials; explain it is an emergency.) / <URL> / Encrypted USB Stick
and/or
<TEAM NAME> network share
NETWORK PATH>
Emergencies.htm / SSBC / Client Resource Centre Emergency Ordering / / Encrypted USB Stick
and/or
Binder
(With Manager of Technical Services or Team Lead responsible for <TEAM NAME>)
Index.html / SSBC Service Level Agreement (Nov 2013) / / Encrypted USB Stick
and/or
Binder
(With Manager of Technical Services or Team Lead responsible for <TEAM NAME>)
<DOCUMENT NAME> / <TEAM NAME> / License and key files for <Service Name>Servers. / <NETWORK PATH>or <URL / Encrypted USB Stick
and/or
Existing <Service Name>Servers
<DOCUMENT NAME> / <TEAM NAME> / <SERVICE NAME> Architecture for Calgary and Kamloops Data Centres / <NETWORK PATH> or <URL> / Encrypted USB Stick
and/or
Binder
(With Manager of Technical Services or Team Lead responsible for <TEAM NAME>)
<DOCUMENT NAME> / <TEAM NAME> / Credentials / <NETWORK PATH> or <URL> / Password escrow. See section 4.4 for details.
service_centre.htm / Customer Service Centre / Service Level Targets / Priority Levels / / Encrypted USB Stick
and/or
Binder
(With Manager of Technical Services or Team Lead responsible for <TEAM NAME>)
<DOCUMENT NAME> / <TEAM NAME> / Architecture of <SERVICE NAME> / <NETWORK PATH> or <URL> / Encrypted USB Stick
and/or
Binder
(With Manager of Technical Services or Team Lead responsible for <TEAM NAME>)
<DOCUMENT NAME> / <TEAM NAME> / <Service Name>Build documentation. / <NETWORK PATH> or <URL> / Encrypted USB Stick
and/or
Binder
(With Manager of Technical Services or Team Lead responsible for <TEAM NAME>)
<DOCUMENT NAME> / <TEAM NAME> / Server names, IP, environment, virtual/physical, role, city, … / <NETWORK PATH> or <URL> / Encrypted USB Stick
and/or
Binder
(With Manager of Technical Services or Team Lead responsible for <TEAM NAME>)
4.4Escrows