Evaluating (Critically Assessing) Secondary Research Literature:

A New Look at a Past Experience

A Past Experience: More Important Now?

In reflecting on the process of evaluating secondary research, I realized that I have personal experience “on the other side” of secondary research, as co-author of an article “Informing the Policy Agenda: The Community Voices Experience on Dental Health for Children in North Carolina’s Rural Communities,” published in the February 2006 supplement issue of Journal of Health Care for the Poor and Underserved. In light of new knowledge gained from class work, both group and individual, this experience provided an ideal opportunity for reflection, with the primary goals of assessing the article’s credibility through the eyes of readers/users seeking information on the subject matter and determining how to build upon this experience to strengthen future work based on this assessment. Perhaps surprisingly, I had never considered the strength and credibility of the article, or the journal that published the article, prior to class work. Up until this point, I had viewed this article as little more than a work assignment for me and an opportunity for “resume building” for all others involved in the process.

To gain a better understanding of how “important” I should consider this experience both professionally and personally, and strengthen any efforts to publish secondary literature I may undertake in the future, I first realized the importance of gaining a perspective on how credible readers might view “Informing the Policy Agenda,” using the criteria defined through class activities. To do so, I replicated the evaluation process learned through group assignments, and conducted:

§  An examination/evaluation of the "publishing" source of the document;

§  Verification of the content and an assessment and evaluation of the research methodology used; and

§  An assessment of the credibility of the authors of the document, including myself as a student or researcher might, and the experts cited in it.

An Evaluation of the Publishing Source

As previously mentioned, the co-authored article was published in the Journal of Health Care for the Poor and Underserved (Special Issue, February 2006). Johns Hopkins University Press, a reputable publisher of scholarly journals, publishes the journal, which is the official publication of the Association of Clinicians for the Underserved (ACU). According to the ACU website (www.clinicians.org), the organization has a current membership of over 8,000 clinicians and 900 organizations; while a sizable membership, it is roughly one-third of the size of the American Public Health Association, which is the most respected association in the field of public health. It is important to note, however, that those public health/medical professionals that focus on the provision of care and services for the poor are likely a somewhat small subset of public health clinicians in general, making assessment of ACU membership size a difficult gauge of credibility.

As with well-known scholarly journals related to health care, the Journal of Health Care for the Poor and Underserved uses a peer-review process in determining manuscripts to publish and maintains an editorial board of medical and public health professionals. The journal covers a broad range of topics related to the provision of health care for the poor.

After conducting an evaluation of the journal in regard to the credibility of its publishing source through the lens of a seeker of secondary research, I would consider the journal to be an appropriate source of credible secondary research, although it lacks the name recognition and the respect of other major journals dedicated to similar topic areas.

Peer-Review: A Personal Experience

Unlike the process many researchers (or in my case, “writer” may be a more appropriate term) encounter to publish an article in peer-reviewed journals, the article my co-authors and I submitted for peer-review was solicited as a special issue of the publication. Given this fact, which may prove true for all special and/or supplemental issues of journals, the article itself did not “compete” for publication, thus possibly somewhat diminishing the content’s credibility. Because the journal is the official publication of a respected professional organization, I can assume that a weak or sub-standard article would not be published.

The content of the article was in keeping with the special issue’s intent of providing insight into education, outreach, intervention and policy work at the community level to address health care for the underserved through case studies of work conducted by communities funded through the W.K. Kellogg Foundation’s Community Voices Initiative. Considered within the purpose of the special issue, the article’s credibility as a “case study” is bolstered based on its potential for building upon existing public dental health knowledge and the documented positive impact the Community Voices Initiative has had as a whole in regard to informing policy and changing health care delivery systems.

Despite the article’s credibility in regard to its “legitimate inclusion” in the journal issue, and its applicability to the field of public dental health, the majority of the content in the article cannot be independently confirmed, as it largely represents the authors’ experiences and subsequent recommendations. Statistics regarding dental health status that are used throughout the document are verifiable using sources cited, but websites and links are not “active,” thus making actual verification cumbersome. However, because the goal of the case study was to provide a “road map” for other communities wishing to increase access to oral health care among indigent children, the lessons learned (recommendations) provide evidence of the article’s usability and value to the reader.

Author Credibility and Verification of Content

Assessment of the credibility of the article authors, including myself, indicate that the content presented in the article presents an interesting case study on efforts of a community to address dental health as a public health practice and policy issue, but none of the authors is broadly published, as all articles attributed to them are associated with a single project, the Community Voices Initiative. In addition, while two of the authors have backgrounds in public health, none have notable (or at least detectable) dental health expertise.

Understanding the criteria, and subsequent processes, that should be used to evaluate (critically assess) secondary literature provides interesting insights into previous experiences. First, the view I have held in regard to the “value” of this article has been modified. I now see the contributions such types of articles (meaning those that are solicited and offer “real world” experiences as opposed to research) as substantial, in that these articles have a place that aligns and augments those that provide results of rigorous research.

While evaluating secondary literature served as a much-needed reminder to “question everything” and not accept all at face-value, upon reflection, the process also “awakened” me to the fact that a good researcher (writer) must know and understand the publishing entity, and strive to develop an article that contributes to “knowledge building” in the field of focus and that would be useful and credible to users of secondary research. Based on this reflection, any future efforts I may undertake to publish work in peer-reviewed journals would be modified based on these insights, as would the value I place on such efforts.