The Growth of Psychometric Testing

The growth of psychometric testing

Alison Wolf

Institute of Education & Centre for the Economics of Education

Paper presented to the British Educational Research Association
Annual Conference, University of Exeter, 12-14 September 2002

(This research was carried out with Dr Andrew Jenkins under a grant from the DFES to the CEE. DfES support is gratefully acknowledged: but the usual disclaimers apply and the views expressed are the author’s own.)

The last quarter century has been marked by an explosion in the number of formal credentials awarded to individuals, in this country and indeed worldwide. This is partly a response to changes in the labour market; partly a self-fuelling ‘diploma disease’ in which the more people obtain diplomas, the more others feel the need to do so too; and partly a result of deliberate government policy: notably in the UK, where formal qualifications are seen as an effective way of maintaining and improving the quality of provision as well as valuable to the individual holder (Dore 1997; Wolf 2001, 2002). So, for example, in summer 2000, in this country, five and a half million GCSEs were taken, and 2 million students were enrolled in award-bearing courses in further education.

The primary motivation of most students when obtaining qualifications is to strengthen their labour market position – either directly, or via entry into desired and desirable higher-tier courses (notably universities). Governments also develop their qualifications policy (both in terms of design/content and funding) with labour market concerns in mind – hence, for example, the current UK policy of linking the funding of vocational training courses and apprenticeships, as well as more general further education, to the successful attainment of qualifications. And both longitudinal analyses (e.g. those based on the cohort studies) and cross-sectional rate of return analyses generally indicate that higher levels of qualification are associated with higher earnings and lower unemployment rates.

It might therefore seem quite logical to conceive of employers’ selection and hiring practices as part of a more or less seamless web, from learning to qualifications to application to job acquisition, with employers using qualifications as a clear signal of applicants’ skills and/or general ability. As more and more qualifications are obtained, presumably qualifications become more and more used and useful in the process of entering and changing employment. Or do they?

In the same period as formal qualification numbers have exploded, there has also been an apparent and large increase in the use of testing for selection, and especially in the use of psychometric tests which provide measures of general ability and personality (generally expressed in terms of population norms). Rather little was known, however, either about detailed patterns of use, reasons for uptake, or reasons, above all, for the apparent increase in testing. Did it signal an increasing concern among employers that education standards were falling/or changes in the type of skill needed, for which educational qualifications did not cater satisfactorily?

As part of the work programme of the Centre for the Economics of Education, we therefore undertook a study the growth of test use for selection, in both the public and the private sectors. The major sources of data were (a) the 1998 Workplace Employee Relations Survey (WERS 98), a large government-sponsored survey of public and private sector workplaces in Great Britain; (b) data from a 2001 survey of recruitment practices carried out by the Chartered Institute of `Personnel and Development, with whom we collaborated on question design; and (c) a series of case studies of selection and hiring procedures in 53 UK enterprises from a wide range of sectors. The detailed results are presented in three Discussion Papers in the CEE series, available from the CEE website (Jenkins 2001; Jenkins and Wolf 2002; Wolf and Jenkins 2002). This paper presents key findings and their implications for an education audience.

Levels of test use

The two best estimates of current levels of test use come from WERS and from the CIPD survey mentioned above. Both used national samples, but WERS included smaller enterprises (and was stratified by size) and had a much larger sample size (n=2191). The CIPD (n=253) was also drawn to be broadly representative by size, but only included workplaces with more than 50 employees. The WERS survey asked about use of personality and competency tests, but did not collect information on which groups of employees were tested with which. The CIPD survey asked about use of personality, general ability, specific skill and literacy/numeracy tests by employee type.

Tables 1 through 4 summarise levels of test use for selection. As they make clear, while far from universal, a very large number of employers (in both the public and the private sector) are using tests for selection purposes: extrapolating to the labour market over time, test use is likely to have an impact on millions of contemporary workers.

Table 1: Use of Personality Tests by Establishment Size (WERS)

Personality Test User

Establishment Size

/ Yes / % / No / % / Total
10-24 / 183 / 16.7 / 913 / 83.3 / 1096
25-49 / 97 / 16.9 / 477 / 83.1 / 574
50-99 / 58 / 21.2 / 216 / 78.8 / 274
100-199 / 36 / 26.9 / 98 / 73.1 / 134
200-499 / 35 / 41.7 / 49 / 58.3 / 84
500 or more / 16 / 55.2 / 13 / 44.8 / 29
TOTAL / 425 / 19.4 / 1766 / 80.6 / 2191
Weighted Data.
Establishment Size measured as number of employees

Table 2: Use of Competency Tests by Establishment Size (WERS)

Competency Test User

Establishment Size

/ Yes / % / No / % / Total
10-24 / 471 / 43.1 / 623 / 56.9 / 1096
25-49 / 267 / 46.5 / 307 / 53.5 / 574
50-99 / 154 / 56.2 / 120 / 43.8 / 274
100-199 / 82 / 61.2 / 52 / 38.8 / 134
200-499 / 53 / 63.1 / 31 / 36.9 / 84
500 or more / 21 / 72.4 / 8 / 27.6 / 29
TOTAL / 1048 / 47.9 / 1141 / 52.1 / 2191
Weighted Data.
Establishment Size measured as number of employees

Table 3: Proportion using selection tests: % of all respondents(CIPD)

Selection tests
Tests of specific skills / 60.1
General ability tests / 54.5
Literacy/numeracy tests / 44.6
Personality questionnaires / 40.7

Table 4 Test use by type of employee (CIPD)

Professionals / Managers / Skilled manual
Specific skills / 45.8 / 29.6 / 33.6
General ability / 37.9 / 30.4 / 33.6
Literacy/numeracy / 30.0 / 25.3 / 23.3
Personality / 26.1 / 38.7 / 9.1

In order to build up a picture of changes in the extent of test use over time, we reviewed 17 surveys of test use, published between the early 1970s and 2000, Most of the studies were cross-sectional, and there was enormous variation in methodology, sampling frame and sample size (such that we do not think that any of the previous surveys can provide accurate estimates for comparison with the WERS or CIPD surveys). Nonetheless, it is clear that test use has grown substantially since the 1980s, even though the costs of test use are substantial. Many can be used only by a trained and licensed user: so a company must either train in-house staff or buy in the services of an external consultant. The test themselves also tend to require either an annual licence fee or to be available in computer-based forms which allow only for so many candidates/analyses. Test use is therefore not something undertaken quickly or lightly: high and growing levels of use must reflect employers’ conviction that they add something substantial and valuable to the selection process.

What makes an employer likely to use tests for selection?

In reviewing previous work on test use, we found that the vast majority of analyses focused on a very limited set of explanatory variables, and often provided only bivariate results. Thus, test use was generally related to either firm size, or to the presence of personnel/human resources specialists or (sometimes) to union presence. The size of, in particular, the WERS dataset allowed us to carry out a much fuller multivariate analysis, one which was also based on a number of theoretical hypotheses about test use. The results indicate that the size of enterprises is much less important than appears at first sight, and is largely operating as a proxy for other underlying factors. They also suggest that concerns related to the regulatory environment in which employers operate, and to general personnel policies, may be at least as important as the issues of skill need and skill supply which most interest educationists.

An economic perspective on recruitment and selection suggests that employers will typically make an assessment of the costs and benefits of alternative selection processes, although not necessarily in a rigorous, quantitative manner. Formal selection procedures help the employer to screen more precisely, improving the quality of the signals provided about the abilities of applicants and so reducing the probability of hiring unsuitable applicants (Barron et al, 1989). Variation in selection methods can thus be explained mainly by variation in these costs and benefits according to the different kinds of job vacancies that employers are aiming to fill.

One implication is that employers will spend more on selection techniques in order to minimise the risk of hiring unsuitable employees, given that monitoring once employees have been recruited may therefore be imperfect. The difficulties of monitoring vary by type of occupation, and this can also help to explain variation in the use of selection tests for different types of employee. The economics literature on incentives focuses on the problems of monitoring and setting appropriate reward schemes for those who perform multiple tasks in their jobs (Prendergast, 1999). In both the public and private sectors, employees may be multi-tasking and their output or productivity difficult to observe, and they could also have rather different objectives and interests from those of their superiors. As a general rule, managers’ behaviour will be less easy to monitor and measure than that of skilled manual workers, clerical workers, or professional and technical workers and we can predict that this will lead to greater expenditure in the recruitment and selection process for managers, including high levels of test use for managerial occupations. Testing will be used to try to ensure that unsuitable managers will not be hired since it will be difficult to monitor their activities once in post.

Straightforward application of the cost/benefit logic also suggests that the use of selection tests is more likely for jobs which require substantial amounts of training because training is costly, and this makes it more important to hire employees who are suitable for the post and can benefit from training. Careful selection through the use of tests can help to ensure that applicants who are suitable for training are hired (Pfeffer, 1998). Similarly, the use of formal selection methods, including tests, is worth investing in for jobs with long tenure or where organisations are operating an internal labour market (Marsden, 1994). If employees are likely to be with the company for a long time, or are likely to be promoted to more senior positions, then this increases the benefits of using relatively expensive selection procedures such as tests. Conversely, companies with high levels of labour turnover and/or many employees on short-term contracts are less likely to make use of tests. Employers will also search more intensively in order to fill positions when the variation in productivity between good and bad employees is high. In such cases, additional search activity, including the use of tests, will help to reduce the chances of missing a highly productive applicant. Conversely, there is not much point in drawing on costly selection tests if the variation among job applicants is very small (Barron et al, 1997). Again, it may be conjectured that there will be much more variation in performance among managerial job applicants than among, say, unskilled manual workers.

With all these factors,. we can also expect that tests will be less valued the more employers are able to get the information they require from educational qualifications - and vice versa. So if economic factors turn out to explain a great deal of the variation in test use among workplaces, it is also plausible to hypothesise that increasing test use indicates either a loss of faith in the accuracy and usefulness of formal qualifications as a signal of underlying skills, or a disjunction between qualification content and the skills in which employers are interested (or both) The former argument is made regularly by employer organisations such as the IoD: while the latter is one of the major arguments advanced by proponents of ‘key skill’ awards. Although the current awards encompass only commuication/numeracy/IT, the key skills concept, as developed originally by NCVQ and promoted by the QAA, also addresses ‘soft’ skills such as problem-solving or teamwork which are seen as critical to employers.

However, there is another set of possible explanatory variables underlying test use by enterprises, which can be described as ‘organisational’ rather than economic. Theories about the effect of organisational characteristics on selection test use start from the supposition that test use is one aspect of a broader formalisation of the personnel function. In the HR research literature, selection tests are often used as a measure (sometimes the only measure) of sophistication in an organisation’s approach to recruitment and selection (Guest, 2000, 2001; Ramsey et al, 2000).

As already noted, for many tests, although by no means all, correct and legitimate use requires that the test administrators have obtained certification in test use, and are therefore aware of how the tests should and should not be used, the right conditions in which to use the tests, and how the test results should be analysed and interpreted. Such a body of knowledge will be more common in organisations which have specialist staff operating within a personnel or human resources department. This implies that test use will more frequently be found among those organisations with a personnel/HR department. It is likely that HR professionals will be keen to make use of tests and will promote their use: partly from genuine faith in their efficacy but also in line with the tendency of professional groups consistently to use their (genuine) professional knowledge to strengthen their overall influence within organisations and so to increase their financial and other rewards (Cohen and Pfeffer, 1986; Dobbin et al, 1988; Baron et al, 1986, 1988).

Test use may also be associated with the presence of trade unions. The argument here is that the absence of formal recruitment and selection methods will make it easier for unscrupulous firms to exclude those with pro-union sympathies. Hence trade union officers will be eager to ensure that formal selection procedures, such as tests, are put in place (Cohen and Pfeffer, 1986). An organisational perspective also suggests that, other things being equal, public sector organisations will be more likely to adopt selection testing than those in the private sector. Because of their high visibility, public sector organisations, it is thought, will be under particular pressure to adopt selection methods which appear to be objective and fair (Dobbin et al, 1988). Case studies have revealed substantial differences between private and public sector approaches to personnel selection ‘in local government and health, for example, the requirements of employment legislation were given a much higher priority than in other sectors. Demonstrating impartiality and consistency and making decisions which could be subject to public scrutiny occupied a more prominent role than elsewhere’ (Scholarios and Lockyer, 1999). The presence of formal grievance procedures is another aspect of formalisation which might well be associated with test use, both reflecting a general commitment to transparent procedures, and giving managers a strong incentive to formalise selection decisions.

Testing may be part of an effort by organisations to make the selection process more transparent and fair, either because some organisations have a strong substantive commitment to fairness at work, or because of a desire to demonstrate that formal equal opportunities procedures are in place. There is considerable variation in the extent to which organisations are committed to equal opportunities – many have a formal equal opportunities policy, but a sizeable minority do not; some organisations also put in place monitoring of applications and promotions by gender and ethnic origin . It is likely that the extent of the organisation’s commitment to equal opportunities will be related to the use of selection tests so long as tests are perceived to be valid and non-discriminatory predictors of performance at work.

Results Tables 5 & 6 summarise the main findings of our analyses of the WERS data-base, which provided a large enough sample for full multivariate analyses to be practicable, and from which we were able to examine most of the key economic and organisational variables either directly or through proxy variables. They indicate first, that the factors determining personality test use are quite distinct from those affecting competency test uptake: and second, that organisational variables are at least as important as economic ones, and probably more so.

Those establishments which provided high levels of formal off-the-job training for employees were significantly more likely to use personality tests as part of the hiring process than those which provided only little or no training. This was not true for competency test use, however: firms which train a lot were no more likely to use these than were firms which train only a little. This is perhaps because competency tests tend to be used for jobs with well defined skills which employers expect to be already available on the labour market rather than in the context of new and changing skill needs. Workplaces which reported managerial and professional vacancies were also more likely to use personality tests for selection compared to other workplaces (though the percentage of professionals in the workforce was negatively related to personality test use). Workplaces with craft and technical vacancies were markedly more likely to report that competency tests formed part of the selection process. This suggests the same underlying pattern. Personality tests are, it seems, being used to evaluate potential of a general kind: educational qualifications may be seen as either failing to assess such potential or simply, for most new hires, as simply too far in the past.