The Measurement Mystique: A Review of the Contribution of Sylvia T. Johnson to the Measurement Community

Michael B. Wallace

The following is a modified presentation of the Career Award Address presented by Dr. Michael B. Wallace at the Annual Meeting of the National Council on Measurement in Education in New Orleans, LA, April 4, 2002. This segment of the address focuses on Dr. Johnson’s work with regard to “Major Issues in Educational Measurement and Their Implications for Educational Policy.”Other presenters in this address discussed other aspects of her work.

MICHAEL B. WALLACE is a Senior Research Associate at The Center for Research on the Education of Students Placed at Risk (CRESPAR), Howard University; . His research interests include equity in educational assessment, early childhood education, parent involvement in education, and the psychology of art. Dr. Wallace served as Dr. Johnson’s Graduate Research and Teaching Assistant and was mentored by her for a number of years throughout his graduate training. They coauthored a number of studies. Dr. Johnson served as Dr. Wallace’s dissertation committee chair and advisor in the Department of Human Development and Psychoeducational Studies. They were also colleagues at CRESPAR/HU in the Broadening the Scope of Assessment Project, for which Dr. Johnson was the Principal Investigator.

Good morning! It is indeed a pleasure and honor to be before you today to recognize the work of our esteemed teacher, mentor, colleague, and friend, Dr. Sylvia Taylor Johnson.

Many of you may recall that Dr. Johnson would often begin a talk with a carefully chosen analogy. In so doing, she would often explain very clearly, complex concepts in a short time frame. I think we recognize her as having been a master at making these things clear to us, some of us as students, and as a community. In her spirit, I will use an analogy to introduce my remarks on Dr. Johnson’s focus on some of the major issues in educational measurement and implications for educational policy.

Some of you may be familiar with the fact that we have in the eastern region of our country, winding through what is known as the Appalachian Mountain Range and the Atlantic Coastal Plain, is the Appalachian Trail. This trail has been carved out of some of the most picturesque wilderness and is maintained by the U.S. Park Service from north to south. Some of you may have actually hiked on this trail. The trail runs through a point not far from our home in Washington, D.C., in a park called Rock Creek Park. If one were hiking from a point farther south, such as from North Carolina to upstate New York on the trail, he/she would come to a fork in the trail in Rock Creek Park that is not particularly well marked, a point at which a human guide or even a sign would be most helpful. For if the wrong path is taken at that point, the hiker can very well end up at a completely different destination than was projected at the outset of the trip. So, instead of arriving in upstate New York, you may end up in upstate Pennsylvania.

Dr. Johnson’s career work relates to this analogy in that she has over the years envisioned, studied, and helped tremendously in mapping out the trails and paths to be taken in order to arrive at where we need to be in the area of educational measurement. In a recent address a little more than a year ago, specifically the 21st Annual Charles H. Thompson Lecture presentation at Howard University (Johnson, 2000), she reminded us of the early seminal work and issues that were encountered in the beginning segments of the trail, as far back as the 1940s, with the work of Allison Davis and Robert Havighurst at the University of Chicago. We appear right now to be, in some respects, at that point in Rock Creek Park, where vital decisions have to be made in order for us to stay on the correct path that should lead to favorable outcomes for all. Dr. Johnson’s work will continue to help us to do that and will contribute to the adoption of sound policy, policy that takes on even greater salience as stakes are raised in educational measurement.

At this time, as many of us in the educational assessment and research community wrestle with the challenges associated with the implementation of new legislation that holds accountability as the centerpiece of the national education agenda, specifically the No Child Left Behind Act of 2002, a number of major issues and concerns are present. Most of these are not new, and others are, of course, emerging.

Dr. Johnson’s long track record in working over the years on the evolution of the National Assessment of Educational Progress (NAEP) must be mentioned here. She served on many committees, panels and other groups charged with developing, evaluating, planning, and formulating policy around NAEP. Her zeal in these efforts was perhaps proportionate to the realization of the importance of these assessments over the long-haul, in “leveling the playing field.” Dr. Johnson’s work over time has focused on fundamental considerations related to equity for all groups of test-takers (Johnson, 1979, 1980, 1988, 1991, 1995, 1998, 2000). I think that we can safely say that she was committed to what Dr. George Madaus (Madaus & Clarke, 1998) has mentioned as the “just” (as in justice, implying equity) measure (i.e., one that is as close to being as truly fair, as possible) and realizing clearly that in high-stakes, policy-driven assessment systems policy must be crafted that creates a level playing field for students and schools. Dr. Johnson, accordingly, had a particular concern for addressing the plight of racial/ethnic minorities in the measurement process, exemplified by a rather recent study that was presented in 1999 before the U.S. Commission on Civil Rights, in which she spoke to the relative achievement test scores of African American male students (Johnson & Wallace, 1999).

She, along with a number of other researchers (Haney & Madaus, 1986; Johnson, 1991; Linn & Baker, 1996), have looked at NAEP data from the perspective aimed at examining factors most related to score differences between subgroups. Briefly, a number of these studies have found that when differences between the average scale scores of states in the NAEP Trial State Assessment are examined (Fine, 1991), these differences are found to be strongly related to differences in resource allocation and availability. Minority students and students whose parents had lower levels of education were in many cases found to have less access to courses, favorable school climates, highly educated/certified teachers, and cognitively stimulating classrooms. In other words, these factors influence opportunity to learn, which can influence academic performance. Dr. Johnson pointed out that the strength of this relationship, varied across states, could provide an alternate index to the current NAEP Report Card for reporting the relative educational progress of students and subgroups within the states. She held that such an approach could be very useful in structuring policy to advance educational progress for all groups. Realizing the tallness of the order, Dr. Johnson maintained that efforts to achieve equity in assessment must be simultaneously directed toward assuring that all students have equally enhancing educational experiences. Further, she held that the validity of a program of national assessments depends on the broad distribution of opportunity to learn. She also saw, with its essential complexity, the potential use of NAEP in validation work with other measures, measures like those proposed (and to be proposed) by states and school systems at this time.

Dr. Johnson warned too that we would not benefit overall from policies that result in teachers merely teaching to the test, or creating the conditions that foster that practice. Much of her more recent work at CRESPAR (Johnson, Thompson, Wallace, Hughes, & Manswell Butty, 1998; Johnson, Wallace, & Thompson, 1999) has to do with research and development with classroom teachers and the kinds of assessments they construct day-to-day. In working with urban and suburban school systems, she recommended policies that emphasized professional development and support of classroom teachers and school administrators around assessment. She saw that a lot more emphasis needs to be placed on “embedded assessments,” those that are most closely tied to the instruction that has taken place. In most instances, teacher training programs are not sufficiently covering this area.

Another area of great interest and concern for Dr. Johnson was centered on such standardized measures as the College Board’s Scholastic Aptitude Test (SAT) and the American College Test (ACT) (Johnson, 1984; Johnson & Wallace, 1989), those that are of high stakes, and traditionally so, in affecting decisions about college admissions. In this policy area, Dr. Johnson has been one of the clarion voices strongly encouraging policies that call for the use of an array of measures and other indicators for the making of these decisions, and not merely one test score. This, I might add, hits rather close to home for me in that recently as I attended a college fair with my teenaged daughter I was glad to hear admissions officials mention more than test scores in the formulas they purported to use for making their determinations. So, in this regard, perhaps some progress is being made.

Dr. Johnson also focused on the problems related to test construction (Johnson, 1984, 1988). She pointed out that across the whole cultural range of the U.S. population, scientific effort focused on comparing the mental status of individuals must satisfy two conditions. First, test-makers must identify those systems of acts that are most representative of all mental behaviors. Second, in order to compare the mental functioning of different individuals, test-makers must select problems in each mental system area that are commonly found in the culture and practice of all socioeconomic group levels in the tested population. Problems that equally motivate all groups to be tested must be found. Additionally, Dr. Johnson also stated (Johnson, 1988) that test constructors must find ways of expressing those problems in symbols common to all the socioeconomic groups to be tested. One policy recommendation to address this area that Dr. Johnson advanced is the inclusion or involvement of a more racially and ethnically diverse workforce, particularly in the test construction function of the testing industry.

In conclusion, Dr. Johnson has stood her ground at that place in the park, where at times it has been sunny and bright, while at other times it has been cloudy and even stormy. She has in many aspects shown us the correct paths to take in order to move along toward the goal of greater achievement for all students, achievement served by optimum measurement and assessment. We are so grateful that she has been there for all of us, and her work carries on. Thank you.

REFERENCES

Fine, M. (1991). Framing Dropouts. Albany, NY: State University of New York Press.

Haney, W. & Madaus, G. (1986). Effects of standardized testing and the future of the National Assessment of Educational Progress (Working paper). NAEP Study Group. (ERIC Document Reproduction Service No. ED 279 680)

Johnson, S. T. (2000). The live creature and its expectations for the future. The Journal of Negro Education, 69(1/2), 150-163.

Johnson, S. T. (1991). America 2000 and the national testing program: Implications for African American children and youth (Editorial). Journal of Negro Education, 60(4), 497-501.

Johnson, S. T., & Wallace, M. B. (1999, April 15-16). Assessment and the educational progress of African American males. A paper prepared for the U.S. Commission on Civil Rights, Washington, D.C., Civil Rights, Washington, D.C.

Johnson, S.T., Wallace, M.B., & Thompson, S.D. (1999). Broadening the scope of assessment in the schools: Building teacher efficacy in student assessment. Journal of Negro Education, 68(3), 397-408.

Johnson, S.T. (1998). The importance of culture for improving assessment and pedagogy. Journal of Negro Education, 67(3), 181-183.

Johnson, S.T., Thompson, S.D., Wallace, M.B., Hughes, G.B., & Manswell Butty, J. (1998). How teachers and university faculty perceive the need for and importance of professional development in performance-based assessment. Journal of Negro Education, 67(3), 197-210.

Johnson, S.T. (1995). Visions of equity in national assessments. In M. Nettles (Ed.), Equity in Performance Assessments (pp. 343-366). Boston: Kluer Academic Publishing Company.

Johnson, S.T. (1988). Test fairness and bias: Measuring academic achievement among Black youth. The Urban Review, 11(1,2), 76-92.

Johnson, S.T. (1984). The test, the tested, and the test-taking: A model to better understand test performance. (ERIC reproduction Service No. ED 249 229)

Johnson, S.T. (1984). Preparing Black students for the SAT—Does it make a difference? An evaluation report of the NAACP Test Preparation Project. (ERIC Document Reproduction Service No. ED 247 350)

Johnson, S.T. (1980). Major issues in measurement—their implications for Black Americans. Journal of Negro Education, 49(3), 253-262.

Johnson, S.T. (1979). The measurement mystique. Washington, DC: Institute for the Study of Educational Policy, Howard University. (ERIC Document Reproduction Service No. ED 191 920)

Johnson, S.T., & Wallace, M.B. (1989). Characteristics of Scholastic Aptitude Test (SAT) quantitative items showing improvement after coaching among Black students from low-income families: An exploratory study. Journal of Educational Measurement, 26(2), 133-145.

Linn, R.L., & Baker, E.L. (1996). Assessing the validity of the National Assessment of Education Progress: NAEP Technical Review Panel White Paper (ERIC Reproduction Service No. ED 404 375)

Madaus, G.F., & Clarke, M. (1998, December 4). The adverse impact of high stakes testing on minority students: Evidence for 100 years of test data. A paper presented at the Harvard Civil Rights Program, Conference on the Civil Rights Implications of High Stakes K-12 Testing, Teachers College, Columbia University, NY. Educational Research Journal.