CMST 4N03 – PRODUCING AND VIEWING THE NEWS

DR. ALEXANDRE SEVIGNY

LECTURE 6

Measurement Techniques

  1. Defining Measurement
  2. Stevens (1951): “Measurmeent is the assignment of numerals to objects or revents according to rules.”
  3. In CA: our objects and events are messages units.
  4. Measurement theory (classic test theory) assumes that there is a “true” value for each variable on each unit.
  5. We attempt, but are often unable to discover the exact true value due to a variety of sources of error:

m = t + e

A measured score is the result of a true score and an error score. We attempt to minimize the e .

There is a true number of acts of violence in a hockey game, as defined by our codebook. However, rarely will we be able to capture them all in our study because of various reason: coder misinterpretation, coer error, coder fatigue, and recording errors.

These errors can be:

  1. Random Errors: unsystematics
  2. Nonrandom Errors: also called bias, it is a threat to accuracy. It involves systematic bias to measuring procedure.
  1. Variability, Reliability, Accuracy and Precision
  2. Reliability: the extent to which a measuring procedure yields the same results on repeated trials (checking each other’s coding)
  3. Validity: the extent to which a measuring procedure represents the intended, and only the intended, concept. “Are we masuring what we want to measure?”
  4. Accuracy: the extent to which a measuring procedure is free of bias.
  5. Precision: the fineness of distinction made between categories or levels of a measure.
  6. Eg.: measuring a character’s age in years is more precise than measuring in decades.
  7. Extreme precision can be counter-productive: measuring her age in days is likely more precise, but too tedious and error prone to be useful.
  1. How the standards interrelate:
  2. See diagram on Pg 114.
  3. Target A – what we always hope to achieve – a valid measure, one that is reliable, accurate and precise.
  4. Target B – good reliability and precision but poor accuracy and suspect validity
  5. Target C – good precision but low reliability and mixed accuracy, threatening overall validity
  6. Target D – both reliability an accuracy are low, an INVALID attempt at measurement
  7. Target E – one shot attempt (no reliability assessed) in which the measure is very imprecise. Large caliber bullet hits the target but takes out most of the rest of the target. The validity here is poor.
  1. Types of Validity Assessment
  2. External validity: can the results be extrapolated to other times, settings, etc.
  3. we can assess the representativeness of the sample
  4. whether the measurement process is true to life (also called ecological validity). This is also useful for replicability.
  5. Internal validity: the match up of a concpeual definition and an operational definition (measurement).
  6. Face validity – this assesses whether the measure taps the desired concept. You take a step back and examine the measures freshly and as objectively as possible.
  7. Internal Validity: the match between conceptual definition and operationalization.
  8. Criterion Validity: this is the extent to whicha measure taps an established standard or important behaviour that is external to the measure (Carmines & Zeller, 1979).
  9. Predictive : the standard or behaviour occurs after the measure
  10. Concurrent: the standard or behaviour exists at the same time as the measure
  11. Eg.: Pershad & Verma, 1995 used as their standard the clinical diagnoses of schizophrenia when individuals’s open-ended responses to inkblots werwe content analysed.
  12. Gottschalk & Gleser 1969 : used four different types of criterion measuer to validate their measure of psychological constructs through the CA of verabl samples:
  13. Psychological
  14. Physiological
  15. Pharmacological
  16. Biochemical

To validate their anxiety scale, they looked at the relationship between individuals’ scores n theat scale and:

(a)their diagnoses by linical psychologists

(b)blood pressure and skin temperatures at the times their speech samples were collected

The researchers conducted experiments

(c)administering tranquilizers, validating lower anxiety scores for the treatment groups.

They also found their content analysis scale to be validated by:

(d) the presence of greater lasma-free fatty acids (blood tests) for those with higher anxiety scores.

  1. Content Validity
  2. Reflects the extent to which the measure reflects the full domain of the concept being measured.
  3. Smith (1999) treid to tap a wide variety of aspects of female sex-role stereotyping in film characters. She measured 27 characteristics, traits and behaviours that had been identified in past research as associated primarily with women in the hope of being exhaustive.
  4. Construct Validity: the extent to which a measure is related to other measures (constructs) in a way consistent with hpotheses derived from theory.
  5. It is hard to get this right.
  6. Some interesting attempts at validation:
  7. Bales’s (1950) interactin process analysis system
  8. Stiles’s (1980) taxonomy of verbal respones modes
  9. Fisher’s (1970) decision proposal coding sstem
  10. Poole & Folger (1981) : tested the validity assumpton that a coding scheme should relatee to the meanings of the utterances as judged by the interactants …. They called this representational validity.
  1. Operationalization
  2. This is process of developing measures… the construction of actual, concrete measurement techniques (Babbie 1995).
  3. For content analsis, this means the construction of a Coding Scheme
  4. This means either a set of dictionaries or a set of measure in a codebook.
  5. Categories or Levels that are exhaustive
  6. This means an appropriate code for each and every unit coded. The category “other” is necessary.
  7. Look at table on page 118
  1. Categories or Levels that are Mutually Exclusive
  2. There should only be ONE appropriate code for each and every unit coded.
  3. If there is the possibility for mulitple codes, then these should be broken down into separate measurs. Look at table on page 119
  4. The categories are not mutually exclusive here. It is better to take the checklist approach of the second figure on 119.
  1. An Appropriate Level of Measurement
  2. Each variable should be measured with categories that are at te highest level of measurement possible, given the goals of the measure.
  3. Stevens has his Four Levels of Measurement:
  4. Nominal – the lowest and least sophisticated level.
  5. A nominal scale consistes of a st of categories that are distinct from one another.
  6. The order of categories is arbitrary, reording the categories make not difference in the meaning of the scale.
  7. In Box 6.1, there are a number of examples of nominal measures (gender, marital status, etc.)
  8. Ordinal – a set of categories that are rank ordered on some continuum.
  9. The use of numbers is for maintaining the propoer ordering, bu the numbers do not signify equal intervals between the groups. We can’t add the numbers together, they are just there to indicate placement in a list.
  10. An interval scale: consists of categories or levels represented by numbers that are quantitative or numeric in the ordinary sense. It differs from ordinary numbers in that the zero point is arbitrary. It is unusual to use this measure in content analysis – there is little advantage to this interval measure compared to ratio measure.
  11. A ratio – the most sophisticated or highest level of measurement.
  12. A ratio scale consists of categories or levels represented by numbers that are quantitative or numeric in the ordinary sense, including a true or meaningful zero point.
  13. Chronological age is constructed to be a ratio measurement
  14. Variables vs. Measures of Variables: A common mistake is to think that level of measurement is attached to the variable rather than to a particular measure of the a variable.
  15. A variable can be measured at different levels. Notice that in the smaple codebook, character age is measured in two ways – one at the ratio level (estimate the charcter’s chronological age) and one at the ordinal level (1 = child, 2 = adolescent, etc.)
  1. Computer Coding
  2. We will not be doing this sort of coding in this class.
  1. Human Coding
  2. All measures of human content analysis have to be fully explicated in a codebook.
  3. The codebook corresponds to a coding form.
  4. You have to make the codebook as COMPLETE and UNAMBIGUOUS as possible.
  5. Even the smallest, most mundane details must be spelled out.
  6. You can choose where to put the most detail – I suggest putting more detail into the codebook, not the form.
  7. Coder Training –
  8. It is IMPERATIVE to train your coders well!
  9. Pilot coding – this is practice coding that will enable you to assess overall viability and reliatbility of protocol.
  10. Sometimes you will change you codebook based on the pilot coding exercise.
  11. Final coding – this is done by each member INDEPENDENTLY (no chatting or comparing notes). This way you can test the reliability of your code book.
  12. Blind Coding – here the coders do not know the purpose of the study.
  13. Demand characteristic: the tendency of participants to give the experimenter the anwers he/she wants.
  14. Judge Indepedence – the freedom of coders to make judgements without input from the researcher.
  15. The Proceses
  16. See diagram on page 134
  17. Medium Modality and coding
  18. Coding in the same modality makes sense (if you are listening to music, code for just music).
  19. Researchers need to know the limitations of their medium.
  20. Tips for Coding
  21. Text
  22. – people work better with HARDCOPY, this way they can easily mark up the pages.
  23. Using computers can bring in more confusion than benefit.
  24. Static Images
  25. Audio Messages
  26. Moving Images
  27. Multimedia