HOW TO CONDUCT A FIELD TEST TO SELECT PPE
It’s nearing the end of this budget year and the beginning of a new budget cycle for a lot of departments and the Grants are finally beginning to flow, which means a lot of departments are going to be purchasing Bunker Gear over the next few months. If your department is in the market for some new gear, a properly conducted Field test is the best way for you to ensure that you find the equipment that best fits your needs.
Determine what it is you are looking for
The first step in designing an effective field test is to determine what your departments’ needs are. This is typically done by conducting a risk assessment. The goal of a risk assessment is to compile a requirements document that outlines exactly what it is that your department needs in their PPE. At this point you are trying to decide what your general requirements are for protective properties, features and configurations. A risk assessment is usually conducted by the departments Safety Officer with input and recommendations from a PPE committee and takes into consideration the following criteria:
Type of duties performed – The fire fighting and rescue tactics that your department employs, and the day to day duties of the firefighters that will be wearing the gear are critical factors to consider when developing the requirements for the most effective ensembles for your department.
Frequency of use of ensemble elements – The average number of runs a department sees and the frequency of use of PPE in both training and non-training activities will play a role in developing requirements for your department’s protective ensemble. As an example, there are a number of different outer shells to choose form when developing a bunker gear spec. Departments that experience a high volume of activity may choose an outer shell with high aberration characteristics to extend the life of their gear.
Organizations experiences – This can run the gambit from first hand experience on the performance of gear worn in the past, to feedback on service levels and repair issues from your or neighboring / mutual aide departments. How long does it take you to get your gear after you order it, is what you receive what you ordered and do you get it when you need it? All of these factors should be considered when developing your requirements and deciding which products you want to include in your field test.
Incident Operations – What types of calls will you be wearing the gear to. Does your department wear their bunker gear to EMS or extrications calls or do they have other gear for those calls. The type of calls that you will be utilizing the gear for is key in designing your requirements for the gear you will be testing.
Geographic location and climate – Obviously gear that is designed for a department in North Dakota will have different requirements from gear designed for a department in Southern California. The weight of the gear, Thermal protection and breathability are all factors that need to be considered. Do the firefighters in your department experience climate related heat stress, or is it more important that they are protected from an extremely cold environment? All manufacturers offer a variety of PPE materials and these materials have differing protective properties, your risk assessment should provide you with the information to determine which combination of materials best fits your departments’ needs.
Addition information that can be used to develop your departments’ requirements will come from the PPE manufacturers who can tell you what is new in the marketplace related to features, designs and composites. A formal presentation from a manufacturer will provide you with information on how their gear is constructed, what they are doing to improve the gear and what advantages their products provide that can align with the needs of your department. All manufacturers will tell you that their gear is the best in the marketplace, it’s up to you, through your field test evaluation process to determine which is really best for your department.
You will also need to insure that the requirements that you develop take into consideration NFPA 1917 – 2007 with the understanding that these are minimum requirements for all structural PPE.
Examining all of these elements and understanding the tradeoffs and advantages of each element in the initial design phase of developing your requirements is critically important to ensuring that you know what you want and need prior to starting your test to determine which manufacturer or product will do the best job in meeting that need. We plan to do a follow up article that will provide a lot more detail on how to conduct a detailed risk assessment, but the information outlined above should get you started.
Design the field test parameters
There are a number of different ways to structure a Field test. Some departments prefer to put field test gear on a busy company and let them test the gear in their normal day to day activities. Other departments conduct field tests at training facilities where each field test participant is run through a series of training simulations to determine the gear’s performance levels and the participant’s preferences. Still other departments use a combination of both the methods described above.
Regardless of the method that you decide to use, there are a number of factors that you need to include when you develop the structure of the Field test for your department.
Make sure all participants test all gear included in the evaluation. In order to get the most objective and fair evaluations of gear being tested it is crucial that all field test participants wear and test each different product model being evaluated form each manufacturer. I also recommend that departments cycle through each model a minimum of 3 times. This allows a better chance for each participant to experience the gear in a wide variety of conditions and to validate their initial findings or opinions of each product being tested.
At the end of your field test you are going to have to compile and analyze a lot of data. Having a quantifiable scoring system in place throughout the testing process with make that job a lot easier. I recommend a numeric scale for each facet of the gear that you are testing as opposed to a subjective narrative. It is important that when you develop your field test evaluation forms that you define the activities that you are scoring and what is being measured for each activity. Be as specific as possible in what you are asking the participants to measure – and then provide a scale from which the participant can select a score that best matches their experience with the gear being tested. At the end of each section or component being tested you may want to provide an opportunity for additional comments that can provide more detail on exactly what each participant experienced. Some Manufacturers will provide field test evaluation forms with their samples. Most of these forms can be provided electronically and will provide you with either a ready to use tool or a starting point for developing a form that fits all of your needs. I have included as a separate attachment a list of sample questions that will give you an idea of what I am describing above.
If you want your field test to run smoothly you will need to define all of the timeframes and develop a schedule that includes all of the activities and key milestones associated with your field test. You should determine in advance how long the test will last, when the participants will cycle through each product and when they will fill out evaluation forms, when the gear will be collected and when the Safety officer or committee with analyze the results and present their recommendations. All of this information should then be communicated to everyone included in the test.
The primary reason for conducting a field test is to generate a recommendation for a product that best fits your department’s needs. This recommendation will be delivered in the form of a final report which should be used to develop purchase specifications. When putting together your field test parameters you should determine in advance who will be responsible for developing the final report and any interim reports associated with the testing and what will be included in the reporting at each stage. The more transparency that you have in association with your field test the better the chance that you will secure buy-in and support for both your process and your resulting recommendations. An objective, transparent and well documented field test that results in a recommendation based on quantifiable results can be a Powerful tool in providing justification for the purchase of the gear that will be the best solution for your department.
Select participants
According to NFPA 1851 Standard on selection, care and Maintenance of Structural Fire Fighting Protective Ensembles, 2001 edition, “Test participants should be selected based on a cross section of personnel, willingness to participate, objectivity and level of operational activity.”
To me this means that when selecting participants for your field test you want to be sure that you are including members who are experienced and know what to look for in proper functioning bunker gear. These participants should also have an open mind about all of the products being tested, you obviously don’t want to choose a participant for an objective field test who has already decided that one product is really the only clear choice before beginning the evaluation. You also want to make sure that the participants that you select are involved in all area’s of your departments operational activity and that they are willing and able to commit time to the process of testing the gear and completing the field test evaluation forms in a timely manner.
Conducting the test
The initial step in the implementation of your filed test should be a kick off meeting. In this meeting the Safety Officer will provide the participants with all of the details of the field test to insure they know what the objective is and how the test will be conducted. The risk assessment, field test schedule, participant requirements and an explanation of products being tested are all subjects that should be covered in this meeting.
Prior to this meeting you should have the manufacturers of the products that will be included in the test measure the participants and then come in to fit the gear instruct participants on how to properly wear the gear.
It is recommended that the participants only wear and retain one set of the test gear at a time to minimize confusion and the chances that all gear is equally tested. You can distribute the gear for the initial round of evaluations at the kick off meeting.
In accordance with your field test schedule, at preset intervals, the Safety officer should provide participants with evaluation forms and have them completed and returned where they can be filed for future analysis. This can be done in person or electronically. The next gear in the rotation should not be provided by the Safety officer until the evaluation forms are completed for the current gear being tested. Evaluation forms should be filled out throughout the test – it is recommended by NFPA 1851 that 3 evaluation forms per ensemble or product be completed by each participant.
The Safety officer should also establish a process for the reporting of events that fall outside the normal reporting procedures. Examples of these types of events my include the need to repair or replace field test gear, any service related issues associated with the field test gear or any critical information that comes to light as a result of the field test that may not be included on the field test evaluation forms. These events should be documented by the Safety officer and included in the data used to compile the final report.
The participants for your field test should remain consistent throughout the test. If for some reason one participant needs to drop out of the test, it is better not to replace that participant and to discard any evaluation forms submitted by that participant unless you have a complete cycle of evaluations from that participant for all gear being tested. If, based on a limited number of participants (less than 4 in total) you need to substitute Field test participants, do so only at the beginning of a new cycle.
The Safety officer should collect and retain all field test gear once the FT is completed so that the selection committee can examine all samples prior to developing a formal recommendation.
Tabulate the results
Once the official testing in completed, it’s time to compile the results and develop a formal recommendation. Prior to writing your formal recommendation, be sure that you take the time to look at all of the information and results that your field test has generated. If your field test evaluation forms provide quantifiable scoring data, then is will be easier for you calculate the results for each category that you tested, but you should not limit your report to just the information generated by the evaluation forms.
The Safety Officer and/or gear committee should also perform a thorough Inspection of all sample gear that was used in the field test. Any finding associated with wear, component failure or workmanship should be included as part of the final recommendation.
In addition, the Safety officer should provide any information pertaining to the field test that relates to service, delivery, repair or accuracy to specification for inclusion in the final recommendation.
The most effective recommendations that I have seen begin with an overview of the requirements that were developed based on the initial risk assessment. This sets the stage by letting readers know exactly what the gear committee was looking for and why. This section is then followed by results of the field test by category tested. In most cases departments tabulate all scores and then provide an average product score for each question on the evaluation form. Any participant comments are then listed by manufacturer at the end of each section of the report. The overall organization of the report will be based how the field test was conducted and how the evaluation forms were designed.
The final section of the report should be a formal recommendation of a specific product that includes the rational of why that specific product was selected. This recommendation should form the basis for the development of a formal bid for the product that you tested. If your bid specification does not align with the results of your field test then you are probably not going to get the product that best meets your departments’ needs. A future article will describe the details of designing a formal bid specification around the results of your field test.