Building the Evidence Base

JBuilding the evidence base

The rationale for establishing an evidence base in the civil justice system is presented in chapter25. This appendix details some of the data required to help build the evidence base.

Collecting data is not costless and some participants in this inquiry have raised concerns that data are sometimes collected with little apparent purpose. To ensure the relevance of data collected, the Commission has identified a number of policy questions that can be used to frame an ongoing assessment of the civil justice system, and identified the data that would help answer those questions.

These questions form the basis of table J.1. The table also takes stock of available data and suggests possible responses to identified data gaps. Greater detail on data requirements is contained in the relevant chapters.

Building the evidence base / 1
Table J.1Policyrelevant data
Policy question / Data required / Available data and gaps / Data response
Understanding and measuring legal need

How many people have legal need?
What are the factors that contribute to legal need?
How are people resolving their legal needs? What works and what does not work?
How many people have unmet legal need?
What are the consequences of unmet legal need?
What are the characteristics of people who experience multiple legal problems?

Information about the legal problems that people face, as well as the demography of those with legal need.
Information about the steps taken (or not taken) to resolve legal problems — ideally through time.

There are no regular surveys of legal need in Australia. Demographic data are usually limited to those that have a dispute in the formal system.
No consistent definition of legal problems — infrequent surveys use inconsistent definitions, which make comparisons difficult, especially across time.
There is a lack of longitudinal information to track individuals through time.

The Legal AustraliaWide (LAW) Survey,undertaken in 2008, examined legal need and responses to legal need. A more contained survey should be repeated on a regular basis.
There should be better collaboration between researchers in this field toensurethat methodologies and definitions become more consistent.
A longitudinal component to legal needs surveys should be added where possible.

How many businesses have legal needs? How do they resolve their legal problems? How many have unmet need?

Information about the characteristics of businesses, their legal problems and the steps taken to resolve them.

There is no regular survey to address the legal needs of businesses. Only one survey of small businesses has been undertaken recently in Australia.

A survey of businesses should be undertaken to provide data in this area. The Australian Bureau of Statistics (ABS) should consider adding questions around legal disputes to the Business Characteristics Survey.

Table J.1Policyrelevant data
Policy question / Data required / Available data and gaps / Data response
How accessible is the civil justice system?

Do legal costs, delays and complexity reduce access to the civil justice system?
How is accessibility changing over time?
Does accessibility vary according to the nature of users?

Costs to consumers.
Timeliness by case type.
Survey data on the ease of using the system.

Timeliness data reported by ombudsmen and some tribunals, and Report on Government Services (RoGS) measures the timeliness of courts. However, timeliness measures are not consistent across these institutions.
Private costs (including legal costs) to users of courts are unclear and comparable data across different time periods are lacking.
LAW Survey provided information on characteristics of users, and asked whether action was not taken because it was too stressful or the respondent did not know what to do.
Australia Institute survey asked respondents how long it took to resolve their legal problems, and whether they thought the system was too complicated to understand properly.
Some courts measure user satisfaction by asking court users how clear they found the processes and forms.
Demographic data are collected but may not assist in answering accessibility questions because they are not linked to barriers.

There is a need for consistent terminology across institutions.
Information from claims lists can form the basis of a measure of average costs/time by case type. For example some jurisdictions, such as South Australia, prepare claims lists (such as small claims and motor vehicle claims) that can be used to form such a basis.
Comparable sources of legal cost data are needed to allow for study of legal costs over time.
Surveys should be repeated periodically to assist in understanding longitudinal effects and changes to accessibility.

Table J.1Policyrelevant data
Policy question / Data required / Available data and gaps / Data response
Understanding and navigating the system

How many people lack legal capability including knowledge of their rights and the capacity to take action?
Do disadvantaged people have lower levels of legal knowledge and does this affect their access to the civil justice system?

Unprompted and prompted recall of legal services and rights.
Demographic and income data on survey respondents.

LAW Survey provided information on the educational levels of respondents and whether respondents took no action because they did not know what to do.
Australia Institute survey asked respondents whether they know their rights and whether they know how to get help if they had a legal problem.
‘Disadvantage’ is not defined consistently across survey instruments, providers and institutions.
Better understanding of why some groups are more likely to have problems is needed.

Surveys should seek to incorporate measures of legal knowledge and capacity.
‘Disadvantage’ needs to be consistently defined so that it is easier to measure the legal knowledge of disadvantaged people.

How effective and efficient are services that aim to improve legal capability including community legal education (CLE), and legal information?
How effective and efficient are legal health checks, outreach and holistic services?
Are referrals appropriate?

Cost and activity count of each type of service.
Client satisfaction and followup data.
Count of best practice legal information and CLE that is shared amongst legal service providers.
Count of referrals from legal and nonlegal service providers.

Activitybased performance targets are a requirement of the National Partnership Agreement on Legal Assistance Services (NPA).
NPA reports track the number of services delivered including website traffic, number of education sessions, publications printed and referrals.
Aggregate expenditure is recorded but not disaggregated by types of services.
Outcomes are not measured.
Reported data are inconsistent and incomplete across jurisdictions.
Measures of effectiveness are generally not based on empirical evidence or evaluations.

Clear, consistent definitions of each type of service are needed.
Client satisfaction and whether services led to satisfactory outcomes could be revealed through surveys of users.
Disaggregated expenditure data on each type of service should be recorded.
Benchmark average costs across jurisdictions over time for each type of service.
Review CLE and information to ensure best practice.
Wellrecognised entry points to record whether callers were referred to the helpline and if so, the type of organisation that made the referral. Where callers are referred to should also be recorded.

Table J.1Policyrelevant data
Policy question / Data required / Available data and gaps / Data response
Protecting consumers of legal services

How can consumers be better informed about the costs of taking legal action?

Publicly available, aggregate billings data and information on billings models for consumers.
Average, median or range of fees by legal matter.

Information is not public.

Aggregated information on average and typical legal costs, by type of matter, could be published in each jurisdiction.

How can more consistency be introduced to how cost assessors determine ‘fair and reasonable’ costs?

Breakdown of costs by legal matter.

Cost assessors do not publish their determinations of ‘fair and reasonable’ costs.

The results of cost assessors’ decisions should be published by type of legal matter. Where necessary, these should be deidentified.

Are complaints bodies achieving effective redress for consumers of legal services?

Data on use and nature of sanctions, timeliness and user satisfaction.

Sanctions and timeliness data not consistently collected.
While complaints bodies seek feedback, more rigorous and systematic followup is not undertaken. This means that there is not a complete picture of what works.

Complaints bodies should report publicly on outcomes achieved for consumers.
Surveys should be conducted periodically.

A responsive legal profession

How responsive is the legal profession?

Number of lawyers in total and practicing by area of law.

ABS definition of lawyers is not sufficiently descriptive.
NSW Law Society publishes information on the number of lawyers.

ABS definition of lawyers needs to be redefined to get better survey results.

Table J.1Policyrelevant data
Policy question / Data required / Available data and gaps / Data response
Alternative dispute resolution (ADR)

How effective is alternative dispute resolution(ADR) and does effectiveness vary depending on: the nature of the dispute; the parties; or the ADR techniques employed?
How often is ADR used?

Cost of ADR.
Count of individual ADR processes.
Demography of users.
Settlement rates and determinants of settlement, for example referral stage.

Unclear whether demographic data are collected.
Costs of provision unknown.
Terminology is inconsistent and data are reported in an ad hoc way and cannot be easily collated and compared.
ADR carried out in the informal sector is not recorded, so extent of use and settlement rates are unknown.

Terminology needs to be agreed on and standardised.
Settlement rates and stage of settlement need to be collected.
At a minimum, how settlement was achieved should be recorded.
Legal assistance providers should be required to report on use of ADR services.
Courts and tribunals should report on how disputes have been settled and whether ADR was used.
Surveys or studies should ascertain ADR use among general population, including for what kind of legal problem, how it was initiated, and whether it was successful.

Ombudsmen and other complaint mechanisms

How efficient are ombudsmen? Where could improvements be made?
Are generalist or specialised ombudsmen more efficient and effective?

Full list of ombudsmenand data on type, cost, caseload, ‘other’ responsibilities and timeliness.
Breakdown of resources devoted to complaints.

Timeliness, costs and caseload are reported by most ombudsmen, however definitions are sometimes inconsistent.
Complaints functions of most ombudsmen are not separately costed.
Data are not coordinated across ombudsmen, making comparisons difficult.

Measures relating to types of disputes and timeliness should be standardised and used consistently for data collection and reporting in order to assist benchmarking.
Benchmarking of similar entities.
Complaints functions need to be separately costed to aid benchmarking of ombudsmen so that average costs are not overestimated.

Table J.1Policyrelevant data
Policy question / Data required / Available data and gaps / Data response
Tribunals

How efficient are tribunals?
Where could improvements in tribunal services be made?

Data on the number of tribunals and their caseload, costs, fees and timeliness, by case type.

Consistent and complete data not reported in Annual Reports.
Not all amalgamated tribunals collect cost data separated by case type.

Consistent terminology needed for benchmarking.
Caseload, fees, cost and timeliness data required; separated by case type for amalgamated tribunals.

Are tribunals easy to use, including for selfrepresented litigants?

Rate of legal representation.
Referral rate of ADR.
Settlement rates through ADR.

Data not reported for all tribunals; reporting measures are inconsistent.

Data on the rate of legal and other types of representation, by type of case.
Data on how disputes were settled and whether ADR was used.

Court processes

What is the relative effectiveness and efficiency of different case management approaches?
What is the scope of disproportionate discovery?
Have reforms to discovery rules been effective?
Have reforms to expert evidence rules been effective?

Cost and timeliness data by case type by case management method.
Costs of discovery relative to total costs of litigation including the value of what is at stake for the parties involved and number of discovered documents.
Surveysof judges’ and practitioners’ views on the extent to which discovered documents assist in the resolution of the dispute.
Cost of expert evidence to litigants, court time utilised for expert evidence, survey data on the quality/utility of different forms of expert evidence.

Limited data overall.
Limited data on the costeffectiveness of different case management approaches for resolving different case types.
It is unclear the extent to which different courts measure the impacts of different case management approaches on timeliness, court resources and litigant costs. Little information is publicly available.
Very limited data on prevalence of disproportionate discovery and the impact of discovery reforms.
Cost of expert evidence not publicly available.

Courts’ case management systems should collect statistics which allow courts to evaluate the impact of case management and procedural reforms on timeliness and court resources.
An appropriate body for coordinating analysis and evaluation of different case management approaches should be determined. Evaluations should include the impact of procedural and case management reforms on litigant costs.
Data could be collected at infrequent intervals on: total litigation costs and amount of costs associated with discovery; the value of what is at stake for the parties in the litigation; the number of discovered documents that are relied upon at trial; whether settlement was achieved after discovery; and lawyers’ impressions of the extent to which discovered documents were crucial in resolving the dispute or narrowing the issues in dispute. A closed case survey instrument could be used for this purpose. Surveys developed by the Federal Judicial Centre in the United States could be drawn upon for this purpose.

Table J.1Policyrelevant data
Policy question / Data required / Available data and gaps / Data response
Court processes (continued)

How often is ADR used in the resolution of disputes lodged in courts?
How efficient and effective is ADR in assisting in the resolution of each type of dispute?

Referral rate of ADR by legal matter.
Type of ADR process used.
Timing of ADR referral.
Cost of ADR.
Settlement rates.
User satisfaction with the process.

Some data for ADR referral rates and settlement rates are reported in court annual reports but these data are not consistently reported and not broken down by legal matter.
Limited evaluations assessing user satisfaction with different ADR processes.

Courts to report on how different legal case types have been resolved, if ADR was used and settlement rates.
Terminology for legal case types and ADR processes would need to be agreed.

Costs awards

How can costs awards better encourage parties to only incur reasonable costs?

Periodic calculations of representative costs.
Typical costs awards by case type, amount in dispute and length, relative to legal expense.

Lack of transparency around how costs awards are determined. Costs awards are based on scales of costs, but method of setting scales is unclear.
Most recent studies of legal costs for state courts released in 1993 and 1994. Legal costs of Federal Court and Family Court are from 1999.

ABS should collect data on legal costs and costs awards at regular intervals.

Table J.1Policyrelevant data
Policy question / Data required / Available data and gaps / Data response
Selfrepresented litigants

What proportion of court and tribunal users are selfrepresented?
What are their characteristics?
Why do people selfrepresent?
How does selfrepresentation affect courts and tribunals?
Are measures aimed at assisting selfrepresented litigants effective?

Number of selfrepresented litigants relative to total users, by legal matter.
Demographic data on selfrepresented litigants.
Data on reasons why people selfrepresent.
Outcomes of selfrepresented litigants compared with other users, and how assistance measures affect outcomes.
Court/tribunal time and resources used to support selfrepresented litigants compared with other users, and how assistance measures affect this.

Most federal courts publish tallies of selfrepresented litigants. The Family Court and the Federal Court publish the most extensive information.
Few state and territory courts and tribunals publish data on selfrepresentation and it is unclear whether they collect this information.
Published data are too highlevel. Courts and tribunals do not hold demographic data on users.
Queensland Public Interest Law Clearinghouse (QPILCH) surveys selfrepresented litigants who have used its services.
It is unclear whether courts and tribunals collect data to assess the effectiveness of initiatives aimed at assisting selfrepresented litigants.

Greater and consistent reporting of proportion of selfrepresented litigants in courts and tribunals should be undertaken in each jurisdiction.
The Family Court or Federal Court are possible models for other courts in this area.
At a minimum, the number and type of legal matter should be collected to inform policy.
Ongoing collection of demographic data on court users may be too onerous. Instead, annual surveys of court users could be undertaken to study differences between selfrepresented litigants and represented users.
Smarter use of case management technology including software could capture information on case outcomes and use of court/tribunal resources so that effectiveness of measures could be assessed.