Independent Peer Review of Legal Advice and Legal Work

A consultation paper from the Legal Services Commission

“the most significant step that the LSC has taken on the path to ensuring that quality legal service is available to all recipients of legal aid” – Clare Dodgson, Chief Executive of the Legal Services Commission (LSC)

Consultation paper available at (pdf copy so you need Adobe Acrobat to download)

Responses by 25th June 2005 to:

Jennifer Will

Supplier Development Group

Legal Services Commission

85 Gray’s Inn Road

London

WC1X 8TX

e-mail -

The purpose of this consultation is to comment on the model of the peer review process, as constructed by the Institute of Advanced Legal Studies (IALS) on behalf of the LSC. This is with a view to independent peer review being used on a national basis as a direct measure of the quality of advice and legal work by Legal Help suppliers, both in the criminal and civil fields. This briefing note is only concerned with civil law suppliers, and concentrated on welfare benefits.

Contents of this briefing

p.1 – What is peer review?

p.2 – Who are the peer reviewers?

p.3 – How does peer review work?

p.8 – Identifying and enabling best practise.

p.8 – Equal Opportunities

p.9 – Peer review criteria, including welfare benefits specific criteria

p.10 – Questions.

What is Peer Review?

In this context, peer review is a form of contract audit based on a review of a sample of a supplier’s files and undertaken by an experienced practitioner who is trained in the peer review framework. This involves the assessment of files using a standard criteria and ratings system to determine the quality of advice and legal work provided to individual clients, with an overall judgement produced on the quality of the supplier. Peer reviews are category specific and are carried out by a practitioner who is experienced and skilled in that area of law.

The assessment of files for peer review is carried out using a standard criteria (the Peer Review Criteria) and a ratings system developed by IALS researchers with the Not for Profit sector (see Appendix 1A, p.9). The criteria are designed to highlight:

  • the information gained from the client and other sources;
  • the advice given based on that information;
  • the steps taken following that advice.

As well as generic criteria for all civil law categories, there are also category-specific criteria under Debt, Employment, Housing, Immigration, Mental Health and Welfare Benefits.

For each civil file reviewed (usually a sample of 15 files), an overall assessment of quality is made with a 1 (highest) to 5 (lowest) rating. Following the sample files being reviewed, the peer reviewer prepares a quality assessment report with particular emphasis on trends and patterns identified across the sample, in order to determine an overall supplier rating from 1 to 5 being given for the quality of legal advice and work, as well as a value-for-money assessment. An overall rating of 3 or above constitutes a “pass” at peer review.

The peer reviewer should not automatically arrive at the final rating simply by averaging the scores on the individual files, but rather should use their skill, experience and training to inform the overall rating of the supplier from the trends and patterns seen on individual files. The peer reviewer does not employ any specific formula to arrive at the overall rating as the fundamental nature of peer review is that it is the judgement of an experienced practitioner.

Who are the Peer Reviewers?

Peer reviewers are carefully selected, trained and monitored by IALS. A Peer Review Panel has been established in nine categories of law, with ongoing recruitment being required to maintain these panels. Peer reviewers are recruited on a category specific basis. Appointment to the panel is through an open selection process. All peer reviewers are contracted with the LSC to carry out peer reviews under the guidance and monitoring of IALS.

Standard application criteria are used during the shortlisting phase of the recruitment process for peer reviewers. Alongside extensive experience and specialisation, applicants must demonstrate as a minimum:

  • five years post qualification experience or equivalent in category of law;
  • three years supervisory experience in category of law;
  • three years experience of LSC Specialist Quality Mark/contract work.

Applicants must be currently working under an LSC contract in a caseworker and supervisory capacity. Shortlisted applicants are interviewed and must undertake a written exercise, assessing a category-specific case file. This is to investigate applicants’ experience, their motivation to become a peer reviewer, their understanding or consideration of concepts of competence, and their open-mindedness and independence.

Candidates that successfully pass the interview and written exercise then undergo a category-specific peer review of their own work, conducted by an existing member of the peer review panel. Candidates are only accepted onto the panel if their work is rated at a level of 2 or above. References will also be sought.

Training is then provided on the peer review process, including using the Peer Review Criteria, the ratings and feedback reports, as well as undertaking example assessments and working with existing peer reviewers from the panel. Applicants are selected and trained to be open minded towards different practice approaches, provided that these conform to appropriate standards of competence. The training also aims to bring about consistency in assessments.

Following the training, trainees are required to conduct a mini peer review in order to assess whether they will be effective peer reviewers and successfully apply the training that they have received. If, following all of the above, the applicant is judged by IALS to be suitable and competent, they will be accepted onto the panel. In order to verify competency, an assessment is carried out on the first two peer reviews conducted by a new reviewer. Further, there is regular performance monitoring, both at regular and random intervals, along with further training exercises, guidance and assessment of model files. A peer review of the peer reviewers will be conducted every three years.

How does peer review work?

Peer review is used to monitor supplier performance for a variety of reasons, including concerns over supplier performance, for national benchmarking of performance, etc. A key part of the methodology is that peer reviewers are not informed of the reason for the review.

A peer review result of Threshold Competence (3) or above is usually valid for three years from the date of the review, provided there are no significant changes to personnel (especially the category supervisor), or the supplier’s quality profile does not indicate a possible reduction in performance.

File Sample

A random sample of 20 files are selected per category of law, with 15 of these files actually being reviewed. The files will normally have been closed within the last 12 months. Checks are carried out with suppliers and reviewers to ensure there are no conflicts of interest.

Conflicts, for the purpose of peer review, concern situations where there has been the opportunity for the peer reviewer to form a view about a supplier’s level of competence or to have their view of a supplier affected or biased in some way. For example circumstances where the peer reviewer:

  • has a partner, or a member of their immediate family, or close friend, that works in a firm/organisation that is scheduled for review.
  • has previously worked for the supplier.
  • has recently acted against the supplier.
  • has taken over a matter from the supplier.

Conflict checks also consider whether the organisations being reviewed are in direct competition with the peer reviewers themselves. A peer reviewer will not review any firm within their geographic region unless both the supplier and the reviewer consent.

All reviews take place on LSC premises – files are not sent to reviewers’ homes or places of work.

Peer review process

There are two distinct stages of peer review possible:

  • initial peer review;
  • second peer review (only scheduled where initial peer review returns rating of 4 or 5).

Suppliers have the right to make representations where peer review rating is 3 or below. (see p.6 for further information on representation process)

Initial peer review

Each review takes approximately one day to assess file sample, and half day to complete the report. Peer reviewer works from the top of the list of sample files to ensure integrity of randomisation of file selection. Each file is assessed against standard criteria and ratings scale. The criteria provide a framework against which the peer reviewer evaluates:

  • the quality of information gained from client and other sources,
  • the advice based on that information, and
  • the steps taken following that advice.

Following consideration of the files, the peer reviewer prepares a peer review report, which details the findings of the assessment. This is the main document that is sent to the supplier following the peer review, and is detailed with appropriate examples from the files to support and substantiate the rating given.

The report has 8 main sub-headings:

  • The review;
  • Positive areas of good practise or good performance;
  • Areas for development/major areas of concern;
  • Other areas of concern;
  • Further comments;
  • Suggested areas for corrective action;
  • Overall value for money score;
  • Overall quality of work score;

A peer review rating of (1) or (2) will not have any major areas of concern, and therefore any issues on the file sample are more likely to be noted under “areas for development”.

Otherwise, “major areas of concern” represents concerns that most influence the overall rating, for example, where the quality of advice has produced an adverse impact on the client. This could include a lack of knowledge or proper understanding of the law, incorrect or unclear advice, lack of advice, failure to address mental health issues, conflicts of interest and possibly costs issues (cost is not an integral part of the peer review system but it is taken into account).

“Other areas of concern” are those that do not have an immediate impact on the client, such as files not in good order, failing to follow standard procedures, poor administration, or inappropriate language.

The “corrective action” section is not intended to direct the supplier how to put such concerns right, though the report can indicate what approaches might be considered. Suppliers should decide themselves how to best implement corrective action. The peer reviewer will write a specific comment if they feel the supplier could benefit from attention to any specific area, such as supervision, file review, training, etc. It is expected that all peer reviews at level 3 or below will require some form of corrective action.

The “value for money” rating assesses whether there has been effective use of public resources, taking into account the value to the client and the use of resources to achieve the outcome. No consequences are incurred as a result of this assessment and it has no impact on a suppliers contract.

An “overall quality of work” rating is then given, ranging as follows (see p.6 below for more information on ratings):

  1. Excellence
  2. Competence plus
  3. Competence
  4. Below Competence
  5. Failure in Performance

A supplier should be able to understand why a particular matter is considered by a peer reviewer to be an area of concern, so that they can implement remedial measures. Suppliers should also be able to clearly understand why they have achieved their rating and not a higher or a lower one, as well as enabling them to understand how they could improve that rating.

Representation process

Where a supplier receives a rating of (3) or less, they have the right to make representations on the following grounds:

  • the supplier disputes the overall peer review rating;
  • the peer review file sample does not appear to the supplier to be sufficiently representative;
  • any other reasonable grounds.

Representations must be made within 21 days of receipt of the peer review report. The peer reviewer considers the representations and decides whether, in light of supplier comments, any amendments should be made to the report and if the rating should be changed. The peer reviewer draws up a report recommending one of three actions:

  • the original rating is upheld;
  • the original rating is overturned and a new rating is decided;
  • a new review is requested.

This report is submitted to IALS for monitoring, and IALS ensure that the correct process has been followed and that the peer reviewer has considered and addressed the representations made. A decision with written reasons is provided to the supplier, normally within 28 days.

Second peer review

A second peer review is scheduled in the following circumstances:

  • where the supplier performance is rated (5), a second review will be carried out immediately;
  • where the supplier performance is rated (4), a second review will be carried out within six months.

The second peer review is carried out on a new random sample of case files, and conducted by a different peer reviewer, with the process being the same as under the initial peer review. It is stated that suppliers could be expected to meet the charge of the second review (in the region of £1,300), with the cost being reimbursed if the second review overturns the findings of the first review. However, there are no immediate plans to introduce a charge.

Peer review outcomes and reporting

Suppliers receive written feedback on the outcome of a peer review in the form of a Peer Review Report. No report is fed back, or action taken on the basis of a rating result, until IALS has reviewed the rating and associated comments. Where IALS considers there to be a concern regarding the outcome of a review, (examples given include where comments do not support rating, or reviewer appears to be consistently awarding a particular rating) IALS will investigate and resolve the matter before the report is fed back to the supplier. The result remains provisional until the full peer review process has been completed, as described above.

Ratings

Excellence (1), Competence Plus (2) – indicates that the supplier is conducting work to a high degree of competence, and is considered necessary for a supplier to be awarded “Preferred” status. The result will normally be valid for 3 years.

Threshold Competence (3) – indicates that, although the quality of work is proficient, there are areas of concern regarding the quality of advice and legal work which need to be addressed, as well as indicating that Specialist Quality Mark requirements might not be being met. The Peer Review Report identifies potential corrective action and areas for development, and should be used to form a development plan for the supplier. Proposals for corrective action should be submitted to the supplier’s Account Manager within 28 days of receipt of the final confirmation of the peer review outcome, with progress against these proposals being considered at future audits.

Below Competence (4) – indicates that supplier’s work is being conducted at a standard below that which a client is reasonably entitled to expect from a competent solicitor. Proposals for corrective action should be submitted to the supplier’s Account Manager within 28 days of receipt of confirmed peer review outcome, and a second peer review scheduled within 6 months to check that quality has improved. If the second review also results in a rating of (4), it will normally be appropriate to terminate the supplier’s right to conduct publicly funded legal services in the relevant category of law, and may be appropriate to terminate the entire contract.

Failure in Performance (5) – indicates that, following two separate peer reviews, the supplier is conducting work at a standard that is significantly below that which a client is reasonably entitled to expect from a competent solicitor (as defined by Clause 3.2 of Contract Standard Terms). It will normally be appropriate to terminate the supplier’s right to conduct publicly funded legal services in the relevant category of law, and may be appropriate to terminate the entire contract.

In the event of a termination, the supplier has a right to review as set out in Clause 23 of the Contract Standard Terms.

LSC staff will not be able to explain, justify or alter any comments that appear on a report, as the report and the ratings decision is an independent assessment written by the peer reviewer.

Identifying and enabling best practise

From peer review results, the LSC will summarise and prioritise information relating to quality of advice and legal work provided to clients, identifying areas of good practise and areas most in need of development or improvement. This exercise will be conducted on a quarterly basis and published on the LSC website and in relevant publications.

Against this information, suppliers are expected to undertake self assessment against key areas identified and where necessary, implement changes to make improvements.