Notes on Bad Science (or even Pseudoscience)
Based on Lecture by Jack S. Damico
A quick review of the lecture (and power point) provided:
Good Science: a body of knowledge, belief, methodology, or practice based on evidence (observational, theoretical, qualitative, or experimental) organized by logic to reach a conclusion (or hypothesis or theoretical formulation) that is then tested by trying to prove it wrong.
Above all, Conclusions are tentative – open for revision
Hypotheses are framed so they are capable of being disproved
Alternative ideas are confronted by systematic observation and analysis (experimental or
qualitative)
Skeptical scrutiny is not opposed
Conspiracies are not the reason for lack of advancement
Beware:
A tendency toward paranoia
Purveyor considers him/herself a genius and others not capable
Purveyor sees her/himself unjustly persecuted and discriminated against
There is a strong compulsion to focus attacks on the greatest thinkers or
scientists
Science adopts an attitude of appreciation of human imperfections and fallibility. This is
why results and ideas are tentative.
The purveyor works in concert with his/her colleagues (not in almost complete isolation)
Bad science: a body of knowledge, belief, methodology, or practice that claims to be scientific but does not adhere to the tenets of science
· Does not adhere to the scientific method
· Lacks supportive evidence
· Focuses on resemblances rather than cause-effect relations
· Lacks scientific plausibility
· Lacks scientific rigor or status.
As a result it often employs vague, exaggerated or untestable claims, over-reliance on confirmation rather than refutation, lack of openness to testing by other experts, and a lack of progress in theory development
Bad science tends to coalesce around the following characteristics:
· There is a tendency to note that with creators or advocates of the method/idea that “Wishing makes it so”
· Bad science is embraced in exact proportion as good science is misunderstood
· Absence of critical thinking
· There is a tendency toward dogmatism
Note that this demarcation is not a clean duality; we are dealing with a continuum when dealing with knowledge not a binary choice. Also, this continuum of good versus bad science is typically accompanied by another in the form of strong versus weak scientific competence. Consequently, one must strike a balance between the two poles of the continua
Some have employed a TOOLKIT to assist in ferreting out the differences:
When possible there must be independent confirmation of the “facts”
Encourage substantive debate on the evidence by knowledgeable proponents of all points
of view
Arguments from authority carry little weight
Spin more hypotheses, explanations…multiple working hypotheses
If there is a chain or argument, every link in the chain must work (even the premise)
Apply Occam’s Razor: when faced with two alternatives, the simpler explanation is
usually the best (e.g., law of large numbers)
Can this idea be falsified?
______
A CHECKLIST TO DEMARCATE
A “yes” answer to the question moves toward bad science:
· Does the claim lack the necessary theoretical characteristics?
Remember: Theories have the following characteristics
Arise from and supported by systematic observation
Must be specific enough for falsification
Must make specific testable conditions
Must be tentative
· Does the claim lack acceptable support?
Remember: Defense by authority is not an acceptable support mechanism
· Does it sound far-fetched or too good to be true?
Remember: a claim should make sense given your knowledge of conventional science…extraordinary claims need extraordinary evidence
· Does the claim only (or primarily) come from a source dedicated to supporting it
Remember: Science starts with a null hypothesis and searches for evidence; it does not start with a positive hypothesis and then supports this hypothesis with questionable evidence and anecdotal reasoning
· Are claimants guarded about evidence or procedures employed?
Remember: Science seeks to be open with methods and data so refutation can
occur.
· Are the available supportive data of questionable quality?
· Is the claim typically announced by (or in) the mass media rather than supportive
professional mechanisms?
· Is there concern by purveyors about suppression by “authority”?
· Is the claim oriented to, or intended to show there is something wrong with the norm?
______
ASSIGNMENT
For next class please identify and investigate a practice, intervention, issue, or concept that seems suspect in the field of speech-language pathology and determine (using the checklist and “tool kit” discussed in class and on this handout) whether it is a based on good science or bad science (and why).
Prepare a one page summary of your work and be prepared to discuss and defend your answer next week.