Transcript Pre-Disaster Recovery Plans - National Association of

Pre-Disaster Recovery Plans
Presentation to:
NACRC Legislative Conference
March 3, 2007
Carol Foglesong, Orange County Comptroller’s Office,
Orlando, FL
and
Paul Ketz, Broward County Records Division,
Ft. Lauderdale, FL
Agenda









Introductions
Purpose
Unpredictability of Disasters
Business Continuity Planning Definitions
Reasons for a Pre-Need Contract
What do you need to consider?
Orange County’s Process to Contract
Other Resources
Questions
Purpose



We are not here because we are
experts in the field of Business
Continuity/Disaster Planning.
We are here because all of us are
struggling with this topic.
We would like to share some of our
experiences with you, and learn from
the experiences of others here today.
Cycle of Disaster Planning
RISK REDUCTION
PREVENTION
MITIGATION
PREPAREDNESS
RECOVERY
RESPONSE
So, what’s a disaster?

Disasters come in all sizes and shapes



Big events: Hurricanes, Tornadoes, Earthquakes,
Blizzards, Floods, Vandalism, Terrorism
Smaller events: Fires, Water Pipes, Roof Leaks
Disasters’ impact



A large geographic area
A localized area (county or city)
A particular office operation/location
Hurricane Wilma

Exposed office
suites are seen in
the damaged
Regents Bank Plaza
on Tuesday, October
25, 2005 in Fort
Lauderdale. (SunSentinel/Joe
Cavaretta)
Hurricane Wilma

The Broward County
School Board
building as seen
from the 30th floor
of the AutoNation
Building in
downtown Fort
Lauderdale. (Brian
Hekman, Coconut
Creek)
Hurricane Wilma

Virtually every window
on the west side of the
Broward School Board
building in downtown
Fort Lauderdale was
blown out by Hurricane
Wilma. (Sun-Sentinel
/Carl Seibert)
Hurricane Wilma

Repairs are
underway at the
Broward County
Courthouse offices
and parking garage
in downtown Fort
Lauderdale. (SunSentinel.com)
What’s a common theme?




Disasters are typically unexpected
Disasters happen when you have or are
making other plans
Disaster size or impact can’t be readily
predicted
Disasters aren’t all BIG events; it can be
the little ones that cause the most
concern and frustration
Disaster Readiness

Business Continuity Plan



Disaster Recovery Plan



Staffing plans
How to get up and running again
Setting priorities on what has to be tackled first
Papers versus computers
Forecast Disaster Plan


Before the event planning
Office preparations when a disaster is predicted
But who do you call for help?

You’ve made a list



You’ve identified responsibilities of staff



Businesses with addresses, names, phone
numbers, faxes, and email addresses
Services offered and needed
Who calls whom when
Who’s in charge
Do you and your staff keep a copy of your
plan/phone lists with you? It won’t help if
your list is at the office!
How to guarantee availability?



If a large area is impacted, there will be
many organizations needing help; you
may need to “get in line”
If a small area is impacted, will you get
the specific help you need quickly?
If it’s a particular building or location
impacted, what’s the best solution?
One answer





A PRE-NEED CONTRACT
A commitment to your organization by a
service provider
You have a spot on the top of their list
You’ve pre-negotiated prices and
services
You’ve already checked out who the
provider is and their references
Getting the Pre-Need Contract


You will have to convince your
purchasing/contract group that a preneed contract is not only possible, but
logical
You knowingly commit to going through
the purchasing process of your
organization even with no known
disaster on the horizon
Cost


Be clear: no services = no dollars
But money has to be available when services
are, in fact, necessary



Does your organization require a “set aside”?
Does your organization have a “reserve fund”?
Contract has to be renewed



Annually?
Every 3 years?
What are your renewal options?
Who’s in charge of the
procurement process?

You may be the designated Records
Management Officer for your organization,
but you will need:






The purchasing/contract group
Likely users must “buy in”
Agreement on who sets the priorities for service
Recognition that the “disaster” may be small or
large scale
Approval by your governing body
Who will negotiate the contract? Do they
have a solid understanding of your business
needs to secure the best vendor?
Other Considerations


Can you opt for higher priority by
negotiating a retainer? Will the retainer
be applied against services rendered?
If something happened to your
materials, how long would it take to
access them?



Do you have 24/7 access?
How do you get someone to the facility?
Is HIPAA a consideration in the
custody/restoration of your records?
Response Time


Do you have a Continuity of Operations
Plan (COOP)?
What kind of time will it take to respond
to an event?

For Emergency Services



Fire, Plumber, Repairmen
For you and your staff
For the Paper-Recovery Vendor
What events are important
based on your geography?

Where are you located and what kinds
of events do you expect every year?





Florida – Hurricanes
California – Earthquakes
Northeast – Snow
Low lying – Flooding
What is it for you???
Vendor Capabilities






How many linear feet of paper can be
recovered per hour per piece of equipment?
Is the Vendor’s equipment wholly owned by
company, or is some subcontracted?
Does the Vendor have a schedule of who will
respond after they are first notified?
Process for chain of custody of records
Where will the vendor set up “base camp”?
What kind of turn-around time
Vendor Experience



How much experience will you require?
What kinds of references do you want
to see?
Past Performance




Program Name
Owner / Architect
Contract Amount / Percent Complete
Completion Date
Be clear about the scope

What media do you want used to recreate
your records?




Film
Electronic
Back to paper?!?
Who can activate the contract



Do you have other agencies/business units who
need access to this contract?
If multiple agencies/business units can access this
contract, have you told the vendor who has
priority?
Whose budget is used to pay for services?
Writing the RFP





Plagiarize!
Don’t reinvent the wheel unnecessarily
Each organization has their own
standard language and format
Stick with the assistance particulars that
you might actually need
Have a way to compare apples and
oranges between vendors/proposers
Computers vs Papers?

Typically, there’s already a recovery plan for
computers, servers, databases, etc.




Communication is key – talk to your IT people and
be sure they have a plan too!
Computer recovery involves not only the
data, but also the equipment (machines,
hardware)
There are specialized companies who handle
this specialized service
Make sure your organization has an active
and realistic plan in place
Paper or Paper Plus?







Paper in a warehouse, awaiting its retention
time or its microfilming
Paper in all those office filing cabinets and
individual desk drawers
Microfilm?
Microfiche?
CDs?
Maps or big drawings?
Other media – besides computers?
Which documents first?




Are your documents prioritized?
Do you store records for multiple
agencies? What gets saved first? Are
your boxes segregated?
All things being equal, whose records
are most important?
Does it make sense to secure your truly
irreplaceable records elsewhere?
What type of paper recovery
process will be used?

There are multiple types of document
recovery/restoration techniques






Vacuum Freeze Drying
Molecular Sieve treatment
Desiccant dehumidification
HVAC decontamination
Gamma Radiation (Remove Mold/Bacteria)
Are there techniques that cannot/should
not be used on your records?
Paper Contamination Factors




Debris
Sewage
Silt
Haz-Mat materials



Asbestos
PCB
Other petroleum products
Extras








Re-filing, Re-indexing, Re-shelving
Daily equipment rentals
Small equipment purchases (shovels)
Will they provide their own generators, or
does the vendor require electricity (water)?
Freight / Transportation Costs
Travel / Per Diem or subsistence costs
General Debris removal at your site to gain
building access
Warranty on services performed/document
reproduction quality
Who does what?








Inventory
Packing
Sorting/Identifying
Prioritizing
Labeling
Loading/Unloading
Re-shelving
Security
Price per cubic foot






Pack out
Gamma Radiation
Cleaning
Drying
Ozone
Deodorization
Price per each item








Audio tapes
Blueprints
Maps
Compact disks
Diskettes
Manuscripts
Laser disks
Photographs







Slides
Microfiche
Microfilm
Negatives
X-rays
Pack out boxes
Pack in boxes
Price per hour


Will any of the vendor’s services be
provided per hour in consultation
fashion?
Does the vendor have per hour charges
for anything not specifically covered in
your contract?
Where will the actual restoration
and recovery take place?

Where will the vendor set up “base
camp”?





Onsite
Remote Location
Vendor’s Site
How will the materials be transported?
What kind of turn-around time does the
vendor give for recovery onsite?
Offsite?
Before you go back in…

Who declares whether the building is safe to
re-enter?





Fire Marshal
Building Inspector
Facilities Management
Some other authority?
Where are your emergency triage supplies?


What will you provide?
What will the vendor provide? Where will the
vendor set up “base camp”?
Before you go back
in…(cont’d)

Will you need assistance to secure the
structure itself?



Water removal
Temporary Power
Temporary building repairs



Securing Walls
Roof Repairs
What is your plan if your staff balk at
going back in?
One solution



Set up scenarios and make pricing
those responses a part of the evaluation
process
Pick 3 events of varying significance
Mix up the media to be recovered
Orange County FL Sample




Full RFP and evaluation grading sheets
being lifted to NACRC document library
Understand that we decided our biggest
problem was going to be water damage
We don’t get snow or many tornadoes
Substitute your own boiler plate front
end
Scope of Work




Location for services (Orange County FL; mostly mid-Orange
County)
Volume of work undetermined (who knows what’s going to
happen?)
Not for computer systems
What we asked for

There shall be no retainer paid in order to keep the contract in
effect. The resulting contract will be on an as-needed basis and
used only in case of a disaster or emergency as so deemed
by the County. The Contractor shall provide 24 hours per day, 7
days per week emergency response service. Response by phone
after the first notification shall be within 2 hours. The Contractor
shall have a representative on site after first notification within six
(6) hours with the necessary equipment to be transferred within
twelve (12) hours.
Selection Criteria and Points
CRITERIA
Qualifications of Staff
Qualifications of Firm
Technical Approach
M/WBE Utilization
Location
Fee Proposal
TOTAL
Welfare Recipient Hires
WEIGHT
10
20
30
10 (minimum allowed)
10 (minimum allowed)
20
100
5 bonus points
Qualifications of Staff


“Include a listing of all staff to be assigned to provide the
required services and resumes for each describing experience,
training and education in the required consulting services.”
Reasoning and/or recommended changes

Tell them to make the bios BRIEF


Clarify the kind of training or certification staff has



You don’t need or want 5 pages per person!
WHEN did they get that training or certification?
“Identify staff experience working with governmental entities
and list those projects.”
Reasoning and/or recommended changes


Hold the government entities and projects for the firm’s
qualifications
You just get the same information in two places
Qualifications of Firm #1
“Provide a description and history of the firm focusing on
previous governmental experience. Provide the number of
years in business showing proof of a minimum of 5 years in
document recovery, the current number of full-time
employees and the excess labor that can be provided in case
of an emergency.”
Reasoning and/or recommended changes






Does governmental experience actually matter?
Five years in business seemed a logical threshold so that we
weren’t getting a firm that sprung up in response to our 2004
hurricanes
Full-time employees seemed important to evaluate whether the
firm could respond to more than one disaster
Excess labor? Probably would just hire locally and teach/train
while on the job
Qualifications of Firm #2

“Identify your equipment inventory available to
Orange County within the Southeastern United
States.



Identify the capacity to process in linear feet per hour for
each piece of equipment at each location.
Identify equipment wholly owned by your firm, or which is
available through subcontracting.”
Reasoning and/or recommended changes



Making up the equipment list was VERY interesting
We decided on Southeastern US as a logical staging area for
us
We learned you can lease/rent anything!
Qualifications of Firm #3


“List at least five references, with a minimum of two
from governmental entity experience, for which the
firm has performed similar work including the contact
name, address, telephone number, email, and date of
the contract.”
Reasoning and/or recommended changes




5 references seems about the standard
2 government entities seemed to make purchasing people
happier
Make sure to ask for email so you can readily contact the
references
Send the reference organizations questions and then call to
interview them
Technical Approach #1



“Provide a brief description of the firm’s approach
to the project. Identify the equipment and
recovery methodologies you employ to recover
water-damaged paper documents, microfilm,
CDs, maps and plans, etc.
Provide a schedule (not later than) of arrival of
your equipment in hours after first notification.
Identify how you ensure records are properly
inventoried and identified as well as the chain of
custodianship.”
Technical Approach #2



“Identify the methods and instruments used to
assess moisture content of records.
Provide an operational plan for rapid deployment
of required supervisors to provide direction to
County employee “first responders” to assess
damage and take interim steps to minimize loss.
Include number of people, and number of
hours for arrival.
Should no power be available, contractor shall be
responsible for providing all power required to
operate equipment.”
Technical Approach #3



“Provide a plan for how your clients are
prioritized in the event of a widespread
emergency in which there is multiple-client
demand.
Identify your permanent dehumidifying
equipment locations in the Southeastern
United States and the capacity in pounds of
water extracted per hour for each piece of
equipment at each location.
Present options available to provide
dehumidifying of buildings and mold and
mildew treatment.”
Technical Approach #4



“Submit your warranty or guarantee of
your services.
Confirm the firm’s agreement to meet
the minimum requirements of this
Request for Proposal.
Proposers may offer alternative
solutions/options to achieve successful
completion of the scope of work herein.”
Technical Approach Notes




This area provided us with the greatest
flexibility in evaluating proposals and the
largest points to award/withhold.
We don’t know if we got this area “right” as
we haven’t had to use the contract.
But we did get it in writing in advance from
the firm with which we contracted.
We drew upon notes from prior NACRC and
disaster recovery sessions. We tried to
imagine many/most scenarios.
Fee Schedule #1
“Submit a rate schedule for equipment, supplies, and staff
services.
Travel, lodging, and meals shall be at rates allowed to public
employees under state law per state statute 112.061
Identify the cost per cubic foot, separately, for each of the
elements as listed in item below. Provide a cost for all
recovery services.










freeze conventionally
blast freeze
freeze dry
rapid freeze
treat for mold and mildew
packing and unpacking
treatment for soot”
Fee Schedule #2






“Identify mobilization and demobilization rates for
equipment and staff.
Identify your markup for any equipment rented,
and supplies purchased.
Identify freight cost per mile by type of equipment.
Provide the cost to implement the recovery
process.
Provide the cost of the Recovery Service.
Describe, if offered, your pre-registration service
and cost associated for such service.”
Fee Schedule #3

“In order to adequately evaluate and
compare services provided by proposers,
three (3) hypothetical scenarios have been
created for which proposers must provide a
written response using prices submitted on
Attachment A, the scope of work provided,
and the technical approach of your firm.
These scenarios are located in Attachment B.”
Scenario #1: Roof Leak




It has just been discovered that the roof of the County’s
Records Center has been leaking for some time now. It is
estimated from the buckling ceiling tiles that this has occurred
over the past six (6) months.
When the actual damage was assessed, 200 boxes (dimensions:
10x12x15 = 1.2 cubic feet) containing paper documents and
microfilm had been severely damaged by water, resulting in
mold and mildew.
Contents of the boxes are as follows: 7/8 copier and bond
paper and file folders, and 1/8 microfilm.
Based on the limited information in this scenario, the scope of
work, and the services listed in Attachment A, the proposer is
asked to provide estimated costs for complete recovery and
restoration of the 240 cubic feet of documents. It is understood
that the type or method of restoration will vary depending upon
the type of records involved.
Scenario #2: Water Pipe Burst


A water pipe has just burst in a County building,
which houses permits and application forms of the
County. Standing water, approximately 8” deep has
seeped into the records (i.e., CDs, microfilm, paper,
maps, and plans, etc.). The damaged records consist
of 1/2 copier and bond paper and file folders, 1/8
CDs, 1/8 microfilm, and 1/4 building maps and plans.
The room size for this County office is approximately
20,000 sq. ft. Housed in this office space are records
contained in 10 four-drawer vertical letter-size filing
cabinets (@ 6.0 cubic feet per unit for approximately
60.0 cubic feet total); 10 five-drawer lateral, lettersize filing cabinets (@ 13.0 cubic feet per unit) for
approximately 130.0 cubic feet of records.
Scenario #2 (continued)


This space also contains approximately 50
workstation cubicles at 100 sq. ft. each, which is
equivalent to an estimated 4 to 5 cubic feet of
records housed at each workstation. The 50
workstations contain approximately 250 cubic feet of
water-damaged records.
Based on the limited information in this scenario,
scope of work, and services listed in Attachment A,
the proposer is asked to provide estimated costs for
complete recovery and restoration of the records. It
is understood that the type or method of restoration
will vary depending upon the type of records
involved.
Scenario #3: Small Fire



A coffee pot with a potential electrical shortage problem was left
plugged into an outlet too long and sparked a small fire. Smoke
from the fire triggered the sprinkler system, which caused water
damage to some of the records.
When the actual damage was assessed, 300 boxes (dimensions:
10x12x15 = 1.2 cubic feet) containing paper documents,
microfilm, CDs, and maps had been moderately damaged by
both water from the sprinklers and soot from the fire. The
damaged records consist of ¾ copier and bond paper and file
folders, 1/8 CDs, and 1/8 microfilm.
Based on the limited information in this scenario, scope of work,
and services listed in Attachment A, the proposer is asked to
provide estimated costs for complete recovery and restoration
of the records. It is understood that the type or method of
restoration will vary depending upon the type of records
involved.
Fee Schedule Notes



Having the scenarios was very
important
We had trouble figuring out how to
evaluate all the “standard” information
the proposers submitted
The scenarios gave us a window into
comparing apples to apples side-by-side
Checking References



You really have to do this part
We designed a series of 10 questions
(lifted to the NACRC document library)
which we sent to the reference people
in advance of our call to them
We had at least 2 people and usually 3
people on each call – and then we
compared notes
Doing the Evaluation




We created spreadsheets and allocated points
among the criteria for each category (lifted to
the NACRC Document Library)
Each of the evaluators completed their own
spreadsheet separately
We then met together and compared scores;
we were very close to each other
What one person couldn’t find or figure out,
another person did
The Contract




We selected our finalist just as Hurricanes
Katrina and Rita plowed in
Getting our contract fully executed dragged
out for 3 months – and we only needed their
properly executed insurance form!
Since the contract was finally and fully signed
in January 2006, we have not had to use the
contract
But we’re sure sleeping better!
Additional Thoughts




Start now; don’t wait for your “season”
Anticipate delays along the way
From first concept to actual completion
was 2+ years for us
Writing the scenarios and considering
all the possibilities requires multiple
brains and various styles
Lessons Learned from Katrina

Having a pre-need contract crosses the “T’s”
and dots the “I’s” in advance.


Remember to look at the whole process
New Orleans Parish experience with their vendor,
“The work that they did was excellent but the
collection department was awful. They relied on a
technical line in the work authorization to keep our
books in Chicago until they were paid eliminating
the opportunity for an audit of their bill. I suggest
that any pre arrangements include reviewing the
billing and collection methods.”
Use your NACRC Contacts


Use NACRC as a resource to find
vendors
Talk to other jurisdictions about their
experiences




Conferences
Listserv
Email
Conference Calls
Questions & Answers





What experiences have you had? Good or
bad?
When will you have a pre-need contract?
How will you develop your potential proposer
list to which to send the RFP?
Can you join with other organizations in your
area?
Can you “tag along” to an existing contract
someone else has?
Additional Resources



NACRC www.nacrc.org
Florida DOS State Library & Archives
http://dlis.dos.state.fl.us/disasterrecove
ry/#HVAC
National Parks Service – Wet Collections
Recovery
http://www.nps.gov/hfc/products/cons/
wet-recovery.htm#
Thanks and for any follow-up


Carol Foglesong, Assistant Comptroller,
Orange County Comptroller’s Office,
PO Box 38, Orlando, FL 32802; 407.836.5982;
[email protected]
Paul Ketz, Broward County Records Division,
115 S. Andrews Ave., Room 120,
Ft. Lauderdale, FL 33301; 954.357.7272;
[email protected]