The MITRE Corporation 1/19/2007
Benchmarking Survey
Effectiveness of Performance-Based Service Acquisition (PBSA)
This benchmarking survey is administered by the MITRE Corporation, a Federally Funded Research and Development Center (FFRDC), in support of a current research initiative to determine how Government agencies can maximize benefit from Performance-Based Service Acquisition (PBSA). Information collected from this survey will help us:
· Capture Federal Government agency PBSA experiences and results-to-date;
· Identify challenges faced by Government agencies:
In determining when it is appropriate to acquire services using PBSA methodologies;
In preparing for award of PBSA contracts;
In implementing and managing PBSA contracts after award;
· Obtain information to begin evaluating the cost, benefit, and risk tradeoffs associated with PBSA implementation in the Federal Government.
This research initiative is funded by MITRE in support of our Government sponsor community. MITRE will not release the specifics associated with your name, your organization’s designation, or contact information. Please contact Kevin Buck (), (410) 923-2611, or Audrey Taub (), (781) 271-8363, if you have any questions, comments, or concerns.
We greatly appreciate your time in completing this survey and will provide you overall results. Please provide your completed surveys as attachments in an email to and . We recognize that this survey is lengthy and that your time is valuable. Feel free to concentrate on those particular questions for which you have readily accessible information.
Background Information
Your Name: ______
Your Job Title: ______
Name of Your Organization: ______
Organization Mission: ______
Your Phone Number: ______
Your Email Address: ______
Your Work Address: ______
1
The MITRE Corporation 1/19/2007
(1) What is your role as it relates to PBSAs? (check more than one as needed)
_____ Make decisions/recommendations as to whether contracts should be structured as PBSAs
_____ Develop and/or ensure compliance with acquisition/performance management processes
_____ COTR/COR for contracts structured as PBSAs
_____ Responsible for ensuring that specific services are provided to my organization, some of which may be contracted as
PBSAs
_____ Contracting Officer for PBSAs
_____ Responsible for measuring and/or managing performance of contracts that are structured as PBSAs
_____ Other (describe): ______
(2) Please describe the prevalence of contracts structured as PBSAs within your organization by providing the following information:
Approximate % of total contracted service dollars that are provided through PBSAs: ______
Number of contracts that are currently in place that are structured as PBSAs: ______
Number of years that your organization has been contracting for services using PBSA guidelines: ______
1
The MITRE Corporation 1/19/2007
(3) For your most important/significant contracts structured as PBSAs, please complete the following table:
PBSA #1 / PBSA #2 / PBSA #3(a) Please describe the contracted requirement/capability/service (e.g., operations and maintenance of the call center). The more detail you provide, the better we can understand your PBSA experiences/challenges.
(b) How complex (for example, low, medium, high) is the service being acquired through this PBSA (please include your reasoning, which could be based upon aspects such as the number of different functions being provided, the number of different skills levels required, the degree of geographic dispersion of support locations, etc.)?
(c) Is the service being provided through this PBSA widely available in the commercial marketplace?
(d) What is the approx. $ value of the contract/task order?
(e) What is the duration of the contract/task order?
(f) How long has the PBSA been in place?
(g) What is the type of contract (e.g., Cost-Plus-Award-Fee, Time and Materials)?
(h) What contracting documentation was/is applied to describe and/or manage PBSA requirements (please check more than one as needed)?
- Statement of Objectives (SOO)
- Performance Work Statement (PWS)
- Quality Assurance Surveillance Plan
(QASP)
1
The MITRE Corporation 1/19/2007
PBSA #1 / PBSA #2 / PBSA #3- Performance Evaluation Plan (PEP)
- Service Level Agreements (SLAs)
- Statement of Work (SOW)
- Other (please describe here)
(i) Did the upfront PBSA contract documentation (prior to award) explicitly and comprehensively identify exactly how the successful bidder must provide the services contracted? Or, were the bidders allowed to propose how they would provide the services? Please explain.
(j) Does the PBSA include penalties if the contractor(s) does not meet Minimum Acceptable Service Levels (MASLs)?
(k) Contractually, what types of penalties are allowed to be imposed for this PBSA?
(l) Does the PBSA include provision of incentives if contractors meet minimum required performance?
(m) Does the PBSA include provision of incentives if contractors achieve a specific level of performance above/beyond minimum acceptable performance? If so, how is this level typically defined?
(n) Who determined what should be incentivized (Government or contractor)?
(o) Who developed the Service Level Agreements (Government or contractor)?
(p) Describe the benefits you have realized from structuring these contracts as PBSAs (an example description would be “Resulted in an innovative solution” or “Quality of services provided is better”).
PBSA #1 / PBSA #2 / PBSA #3
(q) Number of months it took from determination that PBSA should be applied for the contract requirement until the contract was actually awarded.
(r) Was the service provided by this PBSA previously provided by the Government?
(s) Are the Government’s required outcomes from the service (s) provided by this PBSA readily measurable/quantifiable?
(t) Does sufficient flexibility exist in your PBSA contractual documentation to make changes as requirements/conditions change after award?
(u) Has your organization made numerous modifications to fee structures, performance measurement/monitoring requirements, and performance levels after award?
(v) Please provide examples of how the contractor(s) and Government partner as a team in achieving contract objectives for this PBSA.
(w) Overall, do you believe that this PBSA has been successful (explain)?
(x) If this PBSA is failing to achieve what it needs to achieve, please describe symptoms and root causes.
1
The MITRE Corporation 1/19/2007
(4) How does your organization determine if PBSA might be an appropriate acquisition approach/strategy (some examples might include “contract/task order exceeds $xyz”, “service requirements are very complex”, “there is a real need to ensure that sufficient competition exists for contract bidding”)?
(5) For the following organizations/personnel, indicate your opinions as to whether sufficient PBSA training has been obtained (“yes” means that sufficient training has been obtained):
Yes or No Organization/Personnel If “no”, type of additional training needed
______PBSA vendors/service providers ______
______Contracting Officers ______
______COTRs/CORs ______
______Program Managers ______
______Staff responsible for specific functionality/ ______
capability that is/may be contracted using
PBSA techniques
(6) Based on your organization’s experience with PBSA, please answer the following questions related to cost/benefit/risk tradeoffs associated with PBSA implementation. Feel free to include explanations of the answers you provide:
(a) Has your organization applied sufficient resources (including staff) to ensure success of PBSA (please explain)?
(b) In general, have the benefits realized from PBSA outweighed the costs incurred (please explain)?
(c) Have PBSAs reduced your organization’s risk of cost, schedule, and technical performance problems/failure (please explain)?
(d) Is your organization, or your parent organization, placing an insufficient, appropriate, or excessive amount of emphasis on the need to apply PBSA techniques (please explain)?
(e) Do you believe that PBSA has been applied to the appropriate types of contracts (please explain)?
(7) What particular reference materials do you rely upon to ensure that you are able to effectively carry out your responsibilities as they relate to PBSAs? Feel free to provide templates/guidance, cite sources, and provide links/URLs:
(8) Would you be willing/able to participate in round-table discussions with other Government staff and acquisition experts from commercial industry to discuss incentivization approaches, lessons learned, and best practices?
(9) What do you feel is required in order for PBSAs to be successful?
1
The MITRE Corporation 1/19/2007
(10) What do you feel is required in order for services to be effectively contracted by the Government?
(11) What are your most significant criticisms and praises of PBSA based on your experiences within your organization?
(12) For what phase of the acquisition lifecycle is PBSA most effectively applied/adopted by your organization (check more than one choice as needed)? For what phase of the lifecycle is your organization most challenged to ensure PBSA is most effective (check more than one choice as needed)?
My organization is most effective in applying/adopting PBSA during these acquisition phases:
_____ My organization is not effective at implementing PBSA during any phase
_____ Decision-making phase regarding whether or not PBSA should be adopted
_____ Prior to PBSA contract negotiations
_____ During contract negotiations
_____ During set-up/initial implementation of the PBSA contract
_____ On-going PBSA management/monitoring
_____ PBSA contract termination
My organization is most challenged to apply/adopt PBSA during these acquisition phases:
_____ My organization is not challenged with implementing PBSA during any phase
_____ Decision-making phase regarding whether or not PBSA should be adopted
_____ Prior to PBSA contract negotiations
_____ During contract negotiations
_____ During set-up/initial implementation of the PBSA contract
_____ On-going PBSA management/monitoring
_____ PBSA contract termination
(13) Does your organization have sufficient Government resources available for maximizing benefit from PBSA (please apply descriptive words such as “extremely” to make it clear how closely resource availability aligns with PBSA requirements)?
(14) Are there other questions that you feel we should have asked or other concepts related to PBSA that we should have explored to obtain the right types of information for determining your PBSA experiences/opinions?
1