Criteria for Good Research with Emphasis on Qualitative Research

1

Criteria for Good Research with Emphasis on Qualitative Research

This document is based on an exploration of the question, “What makes research good?” by students in a similar research methods course than you are taking. I have added some criteria that were missing from your notes and filled in details. You can use the document for:

·  Evaluating an example of research

·  Designing your own research (i.e., research proposal)

1) Problem statement

This is extremely important. The first question about any piece of research is: What is the problem? The problem logically comes before the research questions. A good paper/proposal has a problem statement that does the following:

·  Explain what the problem is

·  Explain why the problem is important to research. This includes the novelty of the problem (has it been investigated before? Is there previous research, but there still are important aspects that need research?)

·  Explain how the study contributes to knowledge about the problem

All this should be in very simple language. Try explaining your problem statement to someone in the canteen, or a family member who is not an expert in education. If you can’t get them interested in your research, the problem statement is not good enough.

The problem statement should help to develop a focus, so that the research does not try to cover a general topic.

2) Research Questions

The research question is also extremely important and must be researchable. A question is researchable when it is possible to design a study that can answer the research question; some questions cannot be answered empirically, or do not require empirical research. You also mentioned that studies have hypotheses, but this is less common for qualitative research. Here are some other important issues with research questions:

·  Have a small number of research questions, or else the paper/proposal becomes too unfocused and it becomes difficult to see what the new contribution of the research is

·  The research questions should be related to the research problem, and suggest specific ways to study the problem

·  The research methods must be capable of answering the research questions. Can you really come to a conclusion about the conceptual change of a student based on a 200-word transcript of an interview?

·  The research questions must be answered during the study, at least to some extent.

3) Objectives

Any one study will not be able to fully solve a difficult problem, so it important that researchers spell out what they hope to accomplish with the study via objectives. For, example, “The objective of this study is to gain a better understanding of how successful teachers encourage student to student talk, with a view to designing professional development.” Objectives can clarify:

·  What it will be possible to do with the study results?

·  The kind of knowledge expected (e.g., practical knowledge in action research versus more generalizable knowledge)

·  The boundaries of the study. What must be studied to achieve the objectives and what is better left to a future study? This is especially important for case studies.

·  Feasibility: Are the resources and time available sufficient for achieving the objectives? A project can be too ambitious for the resources available, but it also is important to be aware that a study can be so small-scale that nothing worthwhile can be learned from the study.

4) Literature review

The literature review is important for framing the study and demonstrating how the study adds to existing knowledge. The literature review develops the argument for doing the study—why it would be useful. Note that just the fact that research has not been done before is not necessarily a reason for doing it now. The literature review should:

·  Discuss what is known about the problem already. If research already exists it helps to demonstrate that the field is interested in the field

·  Discuss what is not known about the problem. There may be gaps in the literature—aspects that have not been studied at all or have not been studied well—that can help you to identify a “niche,” some research questions that would be worth studying. For example, “Almost all studies in this literature examine teacher learning, but no study was found that also examines the impact on student learning. Some authors (references) argue that professional development should have impact on classroom learning.”

·  Introduce the major concepts that will be used to analyze the data—the conceptual framework. For example, if you are interviewing teachers about classroom management, what kinds of theories and concepts will be used to analyze the teacher’s responses? Here it is impossible to define terms well.

There are different ways to think about a problem, so it is important that researchers anticipate alternate ideas than the ones that underlie a conceptual framework (i.e. criticisms). It is particularly to note that researchers may be biased or make unstated assumptions that can be challenged.

Literature reviews vary greatly. Some journals require short articles, so the literature review can be very focused; other journals require a broader exploration of the problem. Without an adequate literature review it is very difficult to evaluate if a study makes a contribution to knowledge.

It is important that literature reviews are based on good literature (e.g., not conference papers when journal articles or books on the subject are available), and are reasonably comprehensive (e.g., also works from before 2000, so that major areas of work are not missed, or also social psychology of learning in groups and not just cognitive psychology). And a good literature does not just summarize studies (“Chan said xxx, Postman said yy”) but provides analysis (your commentary), so that the case for doing the new piece of research gradually becomes clear. There is a tendency for readers to lose their way in the details of a literature review, missing the main points.

5) Description of data collection methods

The methods section of a paper/proposal is very important. Readers need to be able to understand what the researchers did, and need to be able to evaluate whether the methods used are of sufficient quality to make answers to the research questions possible.

The first job of the methods section is to justify the overall approach chosen. Why action research and not a case study? Why not an experiment? Researchers need to demonstrate that they are aware that the research questions can be studied in different ways, and that there are debates in the field about how certain kinds of research questions are best investigated.

There also are many issues with sampling and recruitment:

·  Is the “case” chosen the best one to study the problem or research questions? E.g., if you wan to study the difficulties of implementing assessment for learning, why would Queens College make a good school to study?

·  Are the phenomena sampled enough? Is solving problems in one area of the curriculum (e.g., trigonometry) enough for meeting the objectives of the study?

·  Do you have enough participants? What kinds of participants are necessary (e.g., teacher and his/her students, but not friends or the principal)? Sample size also is important for inferential statistics (t-test, ANOVA, etc., in which you are trying to generalize to a population.

·  Besides sample size, representativeness of the sample is also important, especially in quantitative research. For example, does the proportion of participants in your sample who are women match that in the population or society?

·  How are the participants recruited? Are there any criteria for excluding participants from the study? How were the participants informed about what would be involved in participating (ethics)

Description of procedures:

·  What was the nature of the intervention, if there was one? E.g., in action research, what was the action undertaken, why, and how?

·  How long did the intervention last?

·  How well was the intervention carried out? For example, if it was important to have mixed-ability groups, were these used, or did students choose their friends to be in their group?

·  What special materials were used, and how? If students did online discussions, did they do them in a computer lab or after school? What instructions were students given? Were these discussions assessed?

·  Ethics: Did the procedures follow or fail to follow ethical principles (e.g., a long experiment that gives a treatment to one class but not another class, the use of deception, payment of participants).

The basic idea here is to provide as full a description of what happened, what the students and teacher did, as possible.

Description of data collection:

·  Was a pilot study completed to clarify the research questions or various data collection methods? E.g., did the researchers try out their interviewing procedures before doing the main study?

·  Do the interview questions lead the interviewee in a certain direction or are they more neutral? Is the interview carried out in a way that produces the best information possible? Were interviews carried out in English or Cantonese?

·  How often did interviews take place? What kinds? How long did they last? Were they recorded?

·  How often did observations take place? What was their focus? How were they recorded?

·  What kinds of materials were collected? Why?

·  What tests were used? What do we know about their validity? What process was used to develop the test (if a new one)?

·  Was more than one data source collected? How does each data source contribute to answering the research questions?

·  Is the data collection plan logically adequate for answering the research questions? For example, if a study lacks a pre-test can you still evaluate learning? Is a comparison group needed? The researcher needs to make decisions regarding these issues before the data collection. Was the study long enough to answer the research questions?

Description of data analysis:

·  Are procedures for processing the data described? E.g., levels of transcription, checking of transcription by participant.

·  Are questionnaires and materials in that were developed in English translated into Chinese and back translated (to check accuracy of the translation)?

·  Are the procedures by which a coding scheme is developed explained in detail (e.g., different kinds of coding). What was the source of ideas for coding?

·  Are any judgments of a subjective kind (e.g., ratings, coding) repeated by an independent rater? Or would it make sense not to expect close agreement?

·  Are the theories or perspectives that underlie the analysis described (e.g., grounded theory, conversation analysis, hermeneutics)?

·  Are inferences made from some of the data tested on the remaining data? This can mean different things. E.g., testing inferences from interviews in the observation data (triangulation), or testing inferences from 6 cases on the remaining 2 cases (Grounded theory should be tested)

·  Are questionnaire data cleaned? E.g., removal of incomplete questionnaires or obviously bogus responses.

·  Is the coding audited, so that biases could be detected?

·  Is both data that confirm and disconfirm reported, if both exist?

·  Is an effect that happened just once over-emphasized?

·  Is it explained how the quotes shown in a paper were chosen? Why do the researchers focus on this quote and not many other possible ones that would lead to a different conclusion?

·  Have the researchers done anything special to investigate questions along the way? E.g., if 4-5 students were absent from a pre-test and were excluded from the study, did the researchers investigate if the sample was still representative?

·  Is the data analysis superficial or in-depth? Are the analytical methods appropriate for the data—i.e., are the assumptions about the data that underlie the analysis valid?

·  Ethics: are the results “too good to be true”. Are results likely to be fabricated or altered?

·  Are learning assessments sufficiently distant from the intervention?

6) Reporting of findings, discussion, and conclusion

You won’t have to deal with this in the proposal, but it is important in any research report. Some issues:

The Results section

·  Are the research questions answered? Do the researchers explain what the data are revealing about the research questions? Or is the presentation data-driven, in which the researchers summarize the results in an unfocused way, without returning to the research questions?

·  Are assertions based on the data analysis, or do they seem like opinions on the topic that are not related to the evidence presented?

·  Was a participant check carried out to see how the participants reacted to the interpretations and descriptions in the paper.

·  Are alternate explanations considered? For example, rather than the intervention maturation of the participants could explain the results

·  Do the researchers clearly delineate the boundaries around their claims? “The results show that xx but not that yyy.”

·  Does the Results section stay focused on the results, or does it wonder away and go into long discussions.

·  Does the study include clear comparisons that facilitate interpretation? Does the study have clear findings that are likely to have impact? Or are the findings so commonplace that they do not seem to require research?

The Discussion/conclusions section:

·  This should not report additional findings. That’s what the results section is for!

·  Do the researchers review whether their objectives were met?

·  Do the researchers discuss the implications of the study by returning to the problem statement literature?

·  Do the researchers point out the limitations of this study to qualify their conclusions?

·  Do they suggest next steps? Are these steps based on the findings of the study? E.g., if the researchers recommend more attention to professional development, is that a recommendation based on the study, or is it their favorite recommendation, which they make after every study of theirs?

·  Does the paper have a conclusion? Are the conclusions based on analysis of results/data?

·  Is the focus of the study preserved in the discussion or does the framing wonder away from earlier sections?