The University of Pittsburgh
Administrative and Policy Studies
ADMPS 3001 Disciplined Inquiry (3 Credits)
Spring 2016
5700 Posvar Hall
1:00-3:40 PM Wednesday
Sean Kelly, PhD
4308 Posvar Hall
Office: (412) 648-7165
Office hours: by appointment
Disciplined Inquiry
Course Overview
Description
Welcome to ADMPS 3001 Disciplined Inquiry, a course in research methods! This course surveys the major research designs and techniques used in social and comparative analysis in education, and educational research more broadly. We will cover both qualitative and quantitative approaches, with an emphasis on the fundamental principles and logic underlying research methodology. While there will be many practical examples of methods in action, this is not primarily a hands-on course. For example, the statistical mechanics of various quasi-experimental approaches are covered in 3100 (Intro to Quantitative Methods I) and 3103 (Intermediate Quantitative Methods). Instead, in this course we will treat quasi-experimental designs as a set of methods sharing similar strengths and weaknesses compared with other approaches, and explore the theoretical rationale for this set of methods. In addition, an important aspect of the course is that many of the methodological topics are relevant to both qualitative and quantitative analyses, including: conceptual frameworks, measurement error, data reduction, sampling, historical/case study methods, and others. A core principle of this course is that regardless of your research goal—description, causal inference, or something else—a robust body of research must eventually draw on multiple methods of inquiry.
Goals and Learning Objectives
This course has a number of specific learning objectives, some of which overlap with related research courses. Note that the basic design and objectives of this course, as well as the assignments, build on the Disciplined Inquiry course originally developed and taught by Professors Gunzenhauser and Bickel in ADMPS for many years.
1. Overview of multiple methodological approaches used in education research, including a rich set of examples. I have tried to select examples that span the range of educational settings, and that do not duplicate readings/topics in other courses. As an introductory course, particular methodological approaches (e.g. instrumental variables, focus groups) are reserved for detailed treatment in other courses.
2. Exposure to epistemological and theoretical diversity in educational research.
3. Understanding of the importance of articulation between a research question, prior research (we have a session devoted specifically to the review of literature), and choice of methodological design within a research effort.
4. Important concepts related to the quality (and ethics) of educational research, such as validity, reliability, generalizability, etc.
5. Exposure to the role that research plays in educational practice and policy, although note that this topic is treated in other courses in greater depth.
Note that two topics are not taught in detail in this course: (1) The Institutional Review Board approval process, including the underlying topics of informed consent, minimizing risk, etc., are covered in 3003 (APS Doctoral Core I), not this course. (2) Publication style (e.g. American Psychological Association publication style guidelines, etc.)
Course Readings and Required Texts
We have two required texts for this course, which we will do multiple readings from, and that will serve as resources for you on additional methodological topics:
Jaccard, J., & Jacoby, J. (2010). Theory construction and model-building skills: A practical guide for social scientists. New York: The Guilford Press.
Berg, B. L., & Lune, H. (2013). Qualitative Research Methods for the Social Sciences: Pearson New International Edition (8th Edition). Boston, MA: Pearson.
Note: You may use any edition of Berg, 6th edition or more recent.
Each week, we will generally have two types of readings, (1) empirical examples of a methodological approach in action, and (2) theoretical/overview/teaching pieces on that methodological approach. The empirical examples will be listed first each week (see Calendar of readings and activities at end of syllabus), followed by the overview readings. For example, in Week 3, we have two examples (Grossman, Littwin) followed by a chapter from Berg. We will generally have three articles/chapters to read each week, or the equivalent amount of reading. Please read the session notes on Courseweb for each session, as I will sometimes post important contextual information about the readings there, or note which parts of the readings you may skim in order to lessen the reading load.
As your studies proceed, you will certainly find that you need greater information about a particular method, and often you will find that within a specific literature (e.g. if you are doing an event history analysis, there are specific resources for that). Generally speaking, overviews of methods that are easy to read and contain clear examples are generally found in methods textbooks and edited volumes, while journal articles contain much more specific, sometimes very narrow methodological issues. Additional difficulties with methodological journal articles is that they are (A) not always in education journals, and (B) focus on formal derivations or theory rather than practical examples. Journals will also occasionally publish methods that are “not ready for prime time;” that perhaps, have not been incorporated into most software programs, or in some cases have not yet withstood enough scrutiny to generate confidence for a new user. That being said, there are a number of journals devoted to research methods that you may find helpful as references, including: the Journal of Educational and Behavioral Statistics, the Journal of Educational measurement, Journal of Research on Educational Effectiveness, Journal of Experimental Education (they mean “experimental” in the broad sense), Sociological Methodology, and others. Note also though, that certain journals publish methodological articles in addition to substantive ones (e.g. Educational Researcher, Educational evaluation and Policy Analysis, Social Science Research, Review of Research in Education, Review of Educational Research, etc.). The discipline of Psychology also has journals that publish a lot of methodological articles (e.g. Psychological Bulletin, Journal of Educational Psychology, Psychometrika, etc.)
One effort by the American Educational Research Association to put together an edited volume with chapters giving overviews of different methods in education is the Handbook of Complementary Methods in Education Research (2006), edited by Green, Camilli, and Elmore.
Course Expectations
Readings and Class Discussion
Because this class meets only once a week, the reading load for each class is relatively heavy. It is also a bit uneven (as real-world work tends to be), so scan the readings for each week to anticipate challenging weeks. The expectation is that your understanding will develop through your independent reading, class discussion, and writing. In order to benefit from class then, you must complete the readings, and bring copies with you to class. Our class discussions will serve not only to develop a coherent understanding of the literature, but you will also be practicing a fundamental skill of both practitioners and academics: the capacity for substantive, analytic conversation.
Writing Assignments and Tests
The writing assignments and tests are an essential element of the learning experience in ADMPS 3001, and this course is an excellent opportunity for you to simultaneously hone your analytic thinking and writing skills.
You will have two writing assignments in this class. The first assignment is the Research Journal and Article Review, which is a two-part assignment where you do some analyses/interpretation of an entire journal, and then a particular article from that journal. The second assignment is a Proposal for an Empirical Study. In the earlier half of the course, you should be doing some work on both of these papers concurrently. The proposal assignment is worth 50% of your grade, so it is on the level of a short term paper. You will be turning in three preliminary materials for comment from me as a formal part of that assignment, including a rough draft of that proposal paper. You will also have the opportunity to make a short presentation on the final class day on your proposal for an empirical study. Both of these writing assignments will allow you to engage with the methodological literature. Writing assignments are due the Tuesday before class at 7:00 PM.
In working with you to craft excellent course papers I will emphasize several generic elements of the writing process:
1) Start with a close reading of the text(s).
2) Be explicit in (a) stating your ideas and (b) in how your paper is organized.
3) Analysis, Analysis, Analysis. All good papers contain an analysis of the text; generalizations, comparisons, causal statements, etc., not mere summaries or descriptions.
4) Writing is an iterative process. With each revision a paper improves. Your classmates and instructor are here to help you improve your papers.
The tests are non-cumulative assessments of your understanding of material for the first and second half of the course. Tests are not very common in the PhD program in our department, One reason tests are not common, is that in your professional work you will be asked to write, teach, discourse with your colleagues, etc., you mostly won’t be asked to take tests! Yet, the material in this course lends itself well to tests. It is possible for me to create tests that are representative of the course content in each half of the course, and the tests allow students an additional opportunity to demonstrate mastery. However, given the length of the writing assignments, these tests will be short and not count as much toward your final grade as the writing assignments.
Research Journal and Article Review
The Research Journal and Article Review is a two part-assignment. But both parts are due on the same date (Tuesday 3/15 at 7:00 PM).
Journal review (825-1000 words, 3-4 pages): First, you will identify a research journal relevant to your program, specialization, and/or research interest and review that journal for the year 2013. A typical journal with 4-5 articles per number (issue), published 4 times per year, will have 18-20 articles. Your goal for the first part of the paper is to analyze the methodological approach taken by works in this journal. Approach that task by posing and answering methodological questions about the journal you selected in 2013. For example: What kind of theoretical perspectives are evident in this journal? What methodologies and methods are published here? Assuming “high quality methods” are important to this journal, what dimensions of quality seem to be evident here? Are there any other issues that reading the journal raises for you? Pose and answer the particular questions that are most relevant to your own analysis.
A. If you select a journal published substantially more often, or with substantially more articles per issue, you may elect to trim the criteria to get down to a manageable number of articles (e.g. in AERJ, focusing only on the “social and institutional analysis” section).
B. Try to select a commonly used/highly valued journal in your field (program/specialization). Do not select a review article journal (e.g. Review of Educational Research), or a methodological journal.
C. Do not try to read each article start to finish, focus on the parts/aspects relevant to the questions you pose.
D. It might be useful to begin by finding information about the quality of this journal, its mission, its sponsorship, its readership, its frequency of publication, and (if possible) its acceptance rate.
E. Please include at least one quantitative analysis in this part of the paper. This could be as simple as counting the number/proportion of occurrences of a given method.
F. Use course readings and specific examples to frame and support your analysis/assertions.
Article review (1000-1275 words, 4-5 pages): Next, analyze one article from the journal by identifying and analyzing the positive value of that article. In your view, what is the meaning and value of this study? How does the methodological approach taken by the author(s)’ contribute to the value of this study? Consider the theoretical framework, the methodological approach, attributes of the data, and the possible audience/implications of this work. Strive for a well-organized and coherent progression of ideas in this paper with a thesis statement, supporting points, and a conclusion. Cite the literature and course readings to support your thesis. Use APA format for all citations and references. Cite reading and use your notes from class discussions in clarifying how to identify and evaluate these elements:
Proposal for an Empirical Study
This paper is a 3,300-4,125 word (or 12-15 page) proposal for an empirical study to address a research question within a line of inquiry that interests you. For PhD students, this assignment overlaps with Milestone 1, the preliminary exam (although the evaluation of the preliminary exam is distinct from your course grade, and may require some modifications from your term paper). You have broad latitude to select an area of inquiry/question that interests you. You are not expected to collect and analyze data for this paper, but rather, to propose a research design, and describe the methodological attributes that will help you to answer your research question. Your paper should begin with a literature review of existing studies on this topic. Published literature reviews are typically 10 pages in length; yours should be more concise than that, perhaps 4-6 pages drawing on approximately 10 references. The balance of the proposal should describe a proposed study, and the logic behind the design and methods you have proposed. In laying out your methodological plan, provide a rationale for its various elements, citing course texts and other methodological readings for support. You have three intermediary assignments that are an opportunity for me to give you feedback. First, on Tuesday 2/02 at 7:00 PM a one paragraph summary of your research question is due. On Tuesday 3/22 at 7:00 PM, a preliminary list of the references you will be using is due. Then, a rough draft is due for this paper on Tuesday 4/12 at 7:00 PM, which is a very important opportunity for me to give you feedback. In the final class session, you will present your proposed study to the class.
Additional Guidelines:
A. Well-written lit reviews can look very different depending on the author’s approach, but two common problems that reduce the impact of a literature review are (1) focusing to “sequentially on authors/studies,” rather than on questions/themes/arguments, etc., (2) providing too little elaboration on study results themselves (i.e. what was found, but also how).