QUESTION POSED: Thinking about your own research projects and findings, are you satisfied that the results of your work are affecting STEM learning and educational policies appropriately? What lessons from your experiences would be helpful to your REESE colleagues? To prospective applicants to the REESE program? To NSF?
Jere Confrey
Joseph D. Moore Distinguished Professor
North Carolina State University
REESE Principal Investigators Meeting
February 18-19 2009
Introduction
An answer to the question of how research affects policy requires, in part, clarification about what it means to be “policy.” When I first think of policy, I think of No Child Left Behind (NCLB), the Center for Policy Research in Education (CPRE), or of positions taken by national groups such as the National Council of Teachers of Mathematics (NCTM) But this is actually a very small aspect of policy. Policy is far broader than these.
A policy is typically described as a deliberate plan of action to guide decisions and achieve rational outcome(s). Policies can be understood as political, management, financial, and administrative mechanisms arranged to reach explicit goals. (Wikipedia)
In addition, policy usually anticipates or guides a set of actions; therefore it is more properly linked to a coherent set of decisions intended to achieve a set of objectives. Plus because policies are implemented or enacted, policy should be deliberate and specific in regulating actions. In terms of research, one can conceive of a dimension of structuring, from broader to narrower, spanning policy, plans, programs and projects. Considering these aspects leads to a more specific definition in the context of STEM education:
A set of basic principles and associated guidelines for coherent decision-making, formulated and enacted by a governing body of an organization, (federal, state, district, professional), to direct, regulate, sway and limit actions in pursuit of long-term goals affecting STEM education.
Examples of policy documents and initiatives include standards, assessment, graduation requirements, licensure, laws and associated regulations, fiscal incentives, rewards, restrictions, textbook adoption, requirements, organizational structures (teacher leaders, elementary specialists, mathematicians or districts as lead PIs on grants), statements of priorities or preferences in funding, methodological requirements (RFT)
I provide this brief introduction to policy because, as the organizers probably anticipated by selecting this question, we researchers are often rather naive in thinking about the relations between our research and policy. We prefer to believe that good research will simply influence policy due to its presumed stature as evident truth. Further there is a widespread belief that influencing policy based on research is reserved for more experienced researchers, later in their careers. Neither of these claims has proven true. To explore this, consider the following multiple-choice question in relation to each statement that represents your point of view:
My research’s connections to policy are limited because:
a)I don’t pay attention to this element of policy in my work.
b)I don’t understand how to influence policy well enough to devote time to it.
c)I have tried to engage with policymakers but they didn’t seem interested in what I had to say
d)Our field is not organized properly to exert adequate influence on policy.
e)My experience with policy is primarily in relation to how policies make my work more difficult (access, research approvals).
f)None of the above.
g)All of the above.
These are among the reasons why researchers often do not engage in policy efforts. If we wish to exert the kind of leadership required in STEM education, we need to rethink our beliefs about influencing policy. Based on these introductory comments, I will discuss the elements of three of my projects that have had significant relationships to policy. I chose diverse examples: a program design, research on diagnostic assessments, and an NRC report. I identify and illustrate four elements of the design and conduct of research that I believe from my experience increase the likelihood of policy effects (Table 1).
Factors which increase likelihood of policy effectsSelecting high-leverage targets for work
Forming partnerships/relationships across traditional borders
Positioning for rapid responsiveness, credibility, and timeliness
Designing for adaptability while avoiding lethal mutations
Table 1: Factors influencing policy effects
While many of the terms in this table are self-explanatory, I define "high leverage targets" as situations in which a change can ramify widely through a system, where a topic is connected to a variety of other topics, or where a pernicious problem has resided. Such situations often require a novel perspectives on a problem, because simply intensifying typical factors has not mitigated the problem. A second characteristic of a “high leverage target” can be that a state or district has already initiated discussion around it, and by connecting one’s research to it, one demonstrates a sense of shared responsibility for the problem while one’s research can help to shape the initiative.
Example One: the UTeach Program at the University of Texas
UTeach is the secondary preparation program at the University of Texas at Austin (UT) which I co-founded with Michael Marder, Professor of Physics, along with MaryAnn Rankin, Dean of the College of Natural Science (CNS), Janice Lariviere, and Mary Long, master teachers. It is currently being replicated at 13 universities across the country and has influenced law-makers and governors on how to design successful teacher education programs. Key program elements that account for UTeach’s success included: early recruitment of students, involvement with expert teacher-leaders, redefining teacher preparation in the College of Education (COE), partnership between the colleges, ways to give high status to the recruits at the University, linking science, mathematics and technology, more recently, teacher induction work. The program has gained considerable national attention because of its promotion by Exxon Mobil Foundation and because it has spurred the interest of Colleges of Natural Sciences towards serious involvement in teacher preparation. To examine why this program became so successful in relation to national policy, I will briefly discuss it in relation to the design elements of partnerships, selecting high leverage targets for intervention, and designing for adaptability.
Partnerships across traditional boundaries. The UT College of Natural Sciences Dean committed herself to increasing the numbers and quality of the science and mathematics teachers produced by the university, after seeing distressingly low numbers graduating from such a large state university. She embarked on designing a teacher education program in CNS that would be attractive and intellectually challenging to students, accepting that it would be competitive to the one in the College of Education. When I joined the University of Texas faculty, I argued vigorously for a joint program. Both colleges needed each other. CNS was naively proposing a first course on “Learning and the Brain.” The COE was drowning its students in generic education courses of low intellectual merit. By forming a partnership between the two colleges, we designed a new approach which offered higher quality courses, focused specifically on teaching mathematics and science. Ultimately the partnership was critical for the quality of the program. Our focused target, math and science education, was initially a small enough effort, relative to the COE’s large elementary preparation program, that it did not raise the flag of challenge to business as usual in the COE. I will admit that over time, it was difficult to get all parties, Master Teachers, CNS and COE faculty acknowledged for their contributions, especially given the CNS talents for raising money based on funders’ perceptions of CNS as the sole pioneer of the project, but the benefits of that partnership far outweighed its challenges.
Selecting high leverage targets for intervention: Clearly increasing numbers of high quality teachers was and still is a high leverage target and we were ahead of the curve in addressing it aggressively. As a faculty member in COE and co-Director of UTeach, my particular target for intervention was to eliminate the generic, and often low-quality education courses, like a general secondary methods course, introductory psychology courses, and even math and science methods courses which are often repositories for a potpourri of overly specific methods, rather than broader practices of teaching. The faculty in COE took a fresh perspective conceptualizing stages of preparation for a teacher from increasing their awareness of their own learning and student learning (Knowing and Learning in Math and Science Education), examining approaches to teaching, assessment and equity (Classroom Interactions in Mathematics and Science), and focusing larger chunks of coordinated instruction (Project based instruction) before student teaching. We abandoned all generic education classes; this had a cost (for instance broader issues of multiculturalism, or larger views of curricular theory), but our focus on mathematics and science gave us sufficient time to carefully prepare teachers gradually for success in teaching students the various content strands and incorporating new technologies. UTeach addressed the high leverage problem of not successfully attracting talent to teaching and to too many students dropping out to the lack of intellectual merit. It is not a problem that is easy for a COE to address alone.
Designing for adaptability while avoiding lethal mutations: Much in education changes over the lifetime of a program: adaptation for change is critical. As teacher shortages deepened, it became essential to revise the design to support multiple points of entry. An NSF grant “Collaborative Excellence for Teacher Preparation” provided support for growth. The chart in Figure 1 demonstrates how we designed and redesigned the program to support early, middle, late, and post-baccalaureate entry. It shows how the core elements could be met, often more intensively, but the clients could clearly see what later entry meant for them.
In terms of avoiding lethal mutations, I would point to an ongoing weakness is UTeach relative to summer courses, a problem of quality that, according to secondhand reports, remains to be solved. When planning expansion for multiple entry points, one needs to ensure levels of fidelity of the program approach and the quality of instruction. The overall program quality requires direct attention to well-qualified and coordinated staff for summer programs, because the strongest program faculty commonly and quite properly devote summers to research. Growth requires consideration of policy changes, such as considering how to permit faculty to count summer coursework towards annual teaching requirements. This is particularly problematic in state universities that typically employ faculty on 9-month contracts.
Figure 1. UTeach Design of Course Sequence and Entry Points
UTeach’s replication at other universities is not its only policy effects. It has spurred the involvement of Colleges of Natural Science and other colleges to become more actively involved in the problem of addressing both the quantity and quality of well-prepared teachers. To my knowledge, research on the program and standardization and publication of related materials has been relatively scarce. Such documentation would be helpful in avoiding lethal mutations, programs that only mimic UTeach in form without the philosophy and approach.
Next I discuss a research project with direct implications for State educational policy.
Example Two: Current Work on Learning Trajectories and State Standards
We are currently conducting research that is having major policy impact as the project proceeds. These aspects of the research were not anticipated in writing the proposal, but they have substantially increased the research’s likelihood of success (as well as the work load required to be responsive to the opportunities). The research involves two NSF projects, one a research synthesis and the other a DRK-12 award. For the first, we have created a database on rational number reasoning (RNR) with over 600 articles in it, reviewing the articles and writing syntheses of concept strands into learning trajectories (LT).
The second project, the DELTA project (Diagnostic E-Learning Trajectories) involves the design of diagnostic measures for a set of learning trajectories in RNR. We are partnering with Mark Wilson and colleagues at the Berkeley Evaluation and Research (BEAR) Center. Our definition of learning trajectories is:
A researcher-conjectured, empirically-supported description of the ordered network of experiences a student encounters through instruction (i.e. activities, tasks, tools, forms of interaction and methods of evaluation), in order to move from informal ideas, through successive refinements of representation, articulation, and reflection, towards increasingly complex concepts over time.
We are using a measurement approach known broadly as evidence-centered design. With assistance from NSF and our program officer, we developed an evaluation model that involves Bill Penuel (SRI), Shelley Goldman (Stanford), and Roger Howe (Yale), helping us define a process of rigorous design.
The policy dimension here includes how we are positioning this work within North Carolina to influence state educational policy. North Carolina has committed itself to graduating “future-ready” students proficient in 21st century skills from early ages. The State Board of Education has solicited advice from the Center for 21st Century skills, the LIFE Center, and from numerous foundations, non-profit organizations, and technology-intensive firms in the Research Triangle. By the time our research group has relocated to North Carolina, the NC Dept of Public Instruction (Department of Education) had been revising the state’s K-12 standards. At the same time, a Blue Ribbon Panel on Accountability was preparing a report that eventually demanded a moratorium on new standards until the State rethought excessive assessment, focused on Essential Standards, and achieved genuine alignment of assessment, standards, and curriculum.
In discussing this project and its relationship to policy, I illustrate the importance of all four features of Table 1--high leverage targets, partnerships, positioning for responsiveness, credibility and timeliness, and designing for adaptability.
Selecting high leverage targets for intervention. In terms of the policy issues, it was clear that we had selected a high leverage area, which was confirmed even by the recent National Mathematics Advisory Panel (NMAP) Report, that fractions are the key to success in algebra. Our characterization of what this means, and what should happen, overlaps and profoundly differs from the NMAP Report, but the topic of rational number in education is of major importance to many people and institutions. Additionally, the topic of diagnostics in math has particularly high potential for impact in North Carolina, where the Leandro Court Case under Judge H. Manning has resulted in statewide attention to the state’s failure to serve rural youth fairly and adequately. In general, researchers need to pay attention to educationally-important Court actions, which can have major effects on both the feasibility and the impact of research endeavors. DELTA addresses a high leverage topic because with all the emphasis on accountability and high stakes testing, the ability to glean instructional guidance from these tests is very low. In addition there is very limited means of recovery in mathematics learning for students who fall behind or, due to mobility are behind, in their mathematics proficiency, because, in part, of a dearth of diagnostic mechanisms for students and their teachers and related interventions.
Partnerships across traditional boundaries. The mathematics educational leadership in North Carolina comprises several traditional groups—leadership in the Department of Public Instruction (DPI), the state chapter of the National Council of Teachers of Mathematics (NCCTM), various teacher leaders, university-based faculty in mathematics education. In North Carolina, these various groups routinely and increasingly collaborate and communicate with each other. Partnership in the state standards revisions has gone well beyond communication among the state leadership. While our research group was too late to have an impact on the design of the first version of K-5 standards, we did endorse passage by the State Board of Education (SBOE), through an ad hoc statewide committee of math educators which my colleagues and I at NCSU helped to organize. The SBOE subsequently adopted the K-5 standards. Then, on closer inspection, the DELTA group identified and communicated our concerns about weaknesses in rational number topics in those NC K-5 standards. The Department of Public Instruction invited us, along with other experts, address these deficiencies by contributing to a set of the descriptors of the standards, which would influence teaching practice, a large MSP to disseminate new standards and their implementation and would influence the new test. We partners did not all agree, and represented very different constituencies. We have devoted a lot of time to finding compromises we can all accept. William Penuel, our lead evaluator termed this work which took approximately two months time “ preparing the ground” and identifies it as a legitimate part of a process of “rigorous design of educational innovations”.
Through our work on descriptors, we were able to address such issues as preparing students for multiplication and division of rational number along with LCM and GCF prior to middle school, anticipating the measurement of angle (a concept important to similarity) starting in third grade, and introducing ratio or rate in elementary school with conversions and qualitative graphing. Most importantly, a concept that we had been discovering was fundamental to early reasoning on RNR, equipartitioning, was at least mentioned in early grades. While the solution was not perfect,it permitted us to intensify the foundation of RNR in multiplication, division, partitioning and ratio while the prior writers had chosen, in response to the Focal Points and concerns for an overpacked curriculum to delay many of these concepts. Both positions have merit and their marriage is often the necessary resolution.
Positioning for rapid responsiveness, credibility, and timeliness
To our surprise, we found that in the fall of 2008, due to the DPI response to the Blue Ribbon Panel, the state had chosen to design a coherent policy to address standards, multiple forms of assessment and accountability simultaneously. Because we had established our credibility to be of service Based on the work in the spring, we were positioned and credible to influence the revision to the standards. It was good news and bad, as it meant our previous work on descriptors would have to be revised again, and we had accepted the burden of additional work on in our planned work timeline.