Reliability and Validity Extra Credit EPSY 6310-- Chapter 4 and 9
Up to 5 points
Please define the following terms. You may use your textbook. Two of these terms have to do with experiments or research studies, and three have to do with tests and measurements. Which have to do with research studies, and which have to do with tests and measurements?
- Internal Validity
- External Validity
- Reliability
- Construct Validity
- Criterion Validity
2. Classical reliability theory is based on the idea that a person’s observed score (their grade) on a test is equal to their true score plus a random error component. The error can be either positive or negative. For two tests you have taken this academic year (any subject), compare your observed score (what you got) to what you think your true score (ability- knowledge) was. What are two sources of error that affected your scores? Which test do you think was more reliable and why?
3. Indicate which of the following concepts (related to characteristics of measurement instruments-methods) are illustrated in the six vignettes below (a-f) and explain why. Terms:
Reliability, inter-rater reliability, test-retest reliability, face validity, criterion (predictive) validity, construct validity, content validity.
In each case, explain whether the example describes a positive or negative instance of the concept (for example, if you choose reliability, is it high or low reliability and why). In all the vignettes except the first one, you should be able to illustrate two concepts from the list.
a. Researchers are studying conditions under which people are likely to lie. To measure their dependent variable, they use a polygraph test to identify lying. After many administrations, researchers administering a polygraph test begin to worry that the machine is actually measuring anxiety and not dishonest responses.
b. You take a personality test three times and, in order, you find out that you are:
(1) an introvert (the first time you take the test);
(2) an extrovert (second time)
(3) very easy to get along with (the third time).
c. You are researcher studying schizophrenia. You give children a test that is designed to detect the onset of schizophrenia. It consists entirely of items such as “What is your favorite food?” and “Is Spring better than Summer?” When participants take the test multiple times over 3 years, their scores are consistent and predict whether the participants will develop schizophrenia during adulthood.
d. You are the CEO of a company. You have two human resource executives administer a job skills test to 100 applicants. The executives get similar results, which you then average for each candidate. Based on the scores, you then hire the 50 most qualified candidates. You find that even among these 50 new employees, those who scored higher on the test perform better on the job.
e. Research consistently shows that scores on Dr. Smith’s newly developed “Selfishness Test” are highly correlated with score on other selfishness test, and, as hypothesized, highly correlated with score on egocentrism. In addition, scores on this test predict willingness to share in an experimental setting.
f. Students in Professor Smith’s History class are assigned to read chapters 1, 2, 3, and 5 for the exam. All the chapters are similar in length and cover about the same amount of material. Dr. Smith spends exactly one week lecturing on each of the chapters. Students are told to study the chapters and lecture notes for all chapters for the exam. On the exam, however, 90% of the questions are based on material from Chapter 2, and 10% of the questions are based on material from chapters 1, 2, and 5.
5. Students who take Dr. Z’s Research Methods test on Tuesday tend to get similar scores when they retake the test the following Monday. Based on this information, Dr. Z argues that the test is both reliable and valid. Is he justified? Why or why not?