SEC/ELL Protocol for Coding of Standards and Assessments
Using Language Demands—Academic English Language Functions
Materials included in this packet
· Protocol
· Language Demands—Academic English Language Functions reference sheet
· Language Complexity Operational Definitions reference sheet
· Coding sheet
· Decision rules sheet
· Linguistic Analysis Training Sample Standards
Introduction
Thank you for your participation in this linguistic analysis workshop. Your assistance will aid us in collecting descriptive information about the Language Demands contained in the standards and assessments documents to be analyzed. Our goal is to analyze linguistically several state standards and assessments using a two-dimensional taxonomy for describing Language Demands.
This study examines the degree of alignment between the states’ content standards, assessments, and instructional practices for English Language Arts (ELA) and mathematics and includes a focus on the language that students, and particularly those who are English Language Learners (ELLs), need to meaningfully engage with and achieve academic content.
Language Demands (described in detail later in this document) is based on research and theory and help make explicit the language necessary for achievement in the content areas, as presented in content standards, assessments, curriculum, and instruction (Bailey, Butler, & Sato, 2007).
The data collected will be summarized into content maps and graphs that can be used to highlight the relative emphasis on Language Demands in these standards and assessments. The resulting content maps and graphs permit graphic comparisons of teacher reports of instructional content with relevant assessment instruments and/or standards. Linguistic analysis will also serve to support alignment analyses of the relationships among instruction, assessment and standards, particularly for ELLs.
The purpose of this protocol is two-fold: (1) to articulate the criteria and processes used to code standards and items and (2) to help ensure the accuracy and consistency of the application of the criteria throughout the study.
This protocol describes the ELL component of the SEC-ELL alignment model and is intended to be used in conjunction with the SEC Coding Procedures for Curriculum Content Analysis.
Coding Dimensions
Each standard and item is to be coded for its mode (i.e., Receptive [Listening, Reading] or Productive [Speaking, Writing]). Then, for each mode found, the standard or item is rated on two intersecting dimensions—Language Demands and Language Complexity.
Language Demands—Academic English Language Functions
The first dimension, Language Demands, is a research- and theory-based framework that operationalizes the language needed for academic achievement and helps clarify where language and content converge or diverge (language skills and/or content skills) as well as distinguish construct-relevant from construct-irrelevant language. Language Demands facilitate the systematic analysis of the specific linguistic skills and language functions in standards, curriculum and instructional materials, and assessments (Sato, Lagunoff, Worth, Bailey, & Butler, 2005; Bailey, Butler, & Sato, 2007; Sato, 2007; Sato & Rabinowitz, forthcoming). A Language Demand is categorized as a linguistic skill if it is fundamental to the development and use of language. A language demand is categorized as an academic language function if it is a contextual application of language. The focus of this model is on academic language because the intent is to understand the academic language functions and complexity required to engage with academic content and achieve in content area standards, assessments, curriculum, and instruction. Language Demands is similar to Content in the SEC model. As a framework, it can be applied across content areas and grade levels just as, for example, Cognitive Complexity (Bloom, 1956) or Depth of Knowledge (Webb, 1997) as frameworks can be applied across content areas. The Language Demands—Academic English Language Functions reference sheet is included in this packet.
Language Complexity
The second dimension related to language, Language Complexity, interacts with Language Demands. Language Complexity is similar to the Cognitive Demands (Expectations) in the SEC model.
Factors affecting Language Complexity (i.e., the complexity of language related to academic content) are (Sato & Rabinowitz, forthcoming):
· Density (length; amount of variation/repetition; amount/level of detail; level of abstraction; amount of graphics/relevant non-text features);
· Construction (common/familiar vs. uncommon; simple vs. complex; level of sophistication; text structure; discourse style; genre characteristics).
Each factor is defined in greater detail in the Language Complexity Operational Definitions reference sheet. Standards and items are coded for each factor in , as well as with an overall holistic complexity rating of 1–4 or X. Although there is always a Language Complexity associated with a Language Demand, it may not always be possible to determine at what level of complexity to rate the Language Demand. In those instances, the letter X should be used to denote the complexity cannot be determined.
Procedures
Training and pre-coding procedures
1. In a group of 3 to 4 raters along with a facilitator, review procedures, coding criteria, rating sheets, and unit of analysis for each subject and/or grade level.
a. Review Language Demands—Academic English Language Functions reference sheet
b. Review Language Complexity Operational Definitions reference sheet
2. Review coded example standards and items that illustrate Language Demands and Language Complexity Levels.
3. As a group, apply the coding criteria described below to a few standards and items to verify understanding of the criteria and procedures.
4. Individually, continue to analyze a set of standards and items, applying the criteria, and recording individual judgments of Language Demands on the rating sheet.
5. Discuss individual analyses with respect to Language Demands and Language Complexity. You may use the discussion to inform your judgments of Language Demands, but it is not necessary to come to consensus about your judgments.
6. Make note of decision rules as necessary and record them on the Decision Rules page. Decision rules are guidelines that help ensure standardization and reliability of the coding process. They are created to address specific issues and circumstances that arise during the analyses as well as to document understanding and interpretation of Language Demands and Language Complexity in the standards and items. Decision rules may arise from group discussion; all analysts should apply these rules. Individual decision rules may also be created to ensure consistency of your coding (intra-rater consistency). Examples of decision rules will be presented during the training.
Training is complete when accuracy and consistency of the criteria’s application, and implementation of the procedures by each analyst, are assured.
Coding Procedures
Following training, you can begin independent coding using the steps below:
1. Review the standards and test items for your subject area.
2. Use the SEC-ELL rating sheet.
3. Determine whether the language in the standard or item has a socio-functional focus (the use of words, phrases or sentences for social or functional purposes, e.g., following directions) or an academic focus. If the standard or item has a socio-functional focus, then write “S” in the column “Focus: A/S.” S ratings are not given a Language Demand code or a Language Complexity rating.
4. If the standard or item has an academic focus, then write “A” in the column “Focus: A/S.”
5. Record the Mode(s) of the item or standard as described below.
Determining the Mode(s)
The Mode of the language of a standard or item is the method by which the information is accessed or processed and can be categorized as either Receptive or Productive. Each mode includes two Mode Categories.
The Receptive mode categories are Listening (LI) and Reading (RD), and may include language received through a variety of sources, such as lecture, observation, discussion, and all forms of text.
The Productive mode categories are Speaking (SP) and Writing (WR), and may include language produced in a variety of forms, such as presentation, response to questions, discussion, solving, and all forms of text.
Record the Mode Category(ies) present in the standard or item. The modes are not mutually exclusive, so a standard or item may be coded to more than one mode (i.e., both Receptive and Productive) and to multiple categories within a mode (i.e., both Reading and Listening).
6. Record the Language Demand(s) of each standard or item (A-P or Z) for each mode.
Determining the Language Demand(s).
Review the Language Demands—Academic English Language Functions reference sheet in this packet. Then choose the appropriate Language Demand(s) (A–P or Z) of each mode of the standard or item, by analyzing the specific tasks required to successfully complete the item or achieve the standard. As many as three Language Demand codes can be assigned to each mode for an item and as many as six Language Demand codes can be assigned to each mode for a standard. Should a standard or item be so complex as to suggest more than these limits, select the most dominant elements of the standard or item to code up to the accepted limit of Language Demands.
If you determine that a standard or item does not contain any language demands, then use the Language Demand code “Z” and explain in the comments field. Z ratings are not given a Language Complexity rating.
7. Record the Complexity of the language of each mode of the standard or item.
Determining Language Complexity
Language Complexity is comprised of two factors—density (coded as low or high) and construction (coded as simple or complex). Make a determination for each factor, and record the code. Then consider the interaction of the two factors to determine the holistic Overall Language Complexity (1–4 or X) of the standard or item. The two individual components, described briefly below, are defined in the Language Complexity Operational Definitions reference sheet. Each component should be considered in terms of the minimum language required to engage with and achieve in the content.
· Density (length; amount of variation/repetition; amount/level of detail; level of abstraction; amount of graphics/relevant non-text features);
· Construction (common/familiar vs. uncommon; simple vs. complex; level of sophistication; text structure; discourse style; genre characteristics).
For each Language Demand, record the letter that represents the level of Density (L or H) or Construction (S or C).
To determine the Overall Language Complexity, consider the interaction among the two components of density and construction as you answer the following question:
“Taken holistically, for this Language Demand, how complex is the language required by students to successfully complete this item or achieve this standard?”
Then rate the overall complexity on a holistic rating scale of 1–4, where 1 is low complexity and 4 is high complexity. If you are unable to determine the language complexity of the standard, first attempt to code it based on the minimum language required to engage with and achieve the content. If the standard does not contain enough information to determine the complexity level, record an “X” to indicate that the complexity cannot be determined. Then record a note in the comments section of the rating sheet to explain your rating. Language Complexity examples will be presented in the training.
Instructions for completing the rating sheet follow on the next page.
8. Document new decision rules as appropriate. All decision rules (group-determined and individually developed) should be turned in with the completed rating sheets.
9. Once standards and/or items have been rated, return your paperwork to the facilitator.
5 ©2008
Using the Rating Sheet
Below is a sample excerpt from the sheet on which you will record Language Demand codes.
Item Designation / Focus(A/S) / Mode
(RD/LI/ WR/SP) / Language Demands
(A–P, Z) / Density
(L/H) / Construction
(S/C) / Overall Complexity
(1–4, X)
1 / 1 / A / RD / B / H / C / 3
2 / 2 / A / RD / C / L / S / 1
3 / LI
4 / WR / C / G / J / H / S / 3
5
The rating sheet example above shows the correct coding format for the Language Codes. Note that the letters for the Focus, Mode, Language Demand, Density, and Construction and the number for the Overall Language Complexity are placed in separate cells. Every item or standard found to have an Academic focus should receive mode category letter codes. For each mode (Receptive [RD/LI], Productive [WR/SP]) the item or standard should receive letter codes for Language Demands and Language Complexity, and an Overall Language Complexity number code.
As mentioned earlier, up to three separate Language Demand by Language Complexity combinations per mode may be selected for any one assessment item, and up to six Language Demand by Language Complexity combinations per mode may be coded for standards. In the example listed as Item Designation 2 above, the item has an academic focus, and has language in both the receptive (Reading and Listening) and productive (Writing) modes. For the receptive mode, the item requires reading and listening in Language Demand C. The reading and listening in this item are rated as low density, simple construction, and received an overall complexity rating of 1. The same item also has language in the productive mode, in the form of writing, for Language Demands C, G, and J. The writing in this item are rated as high density, simple construction, and received an overall complexity rating of 3. Each applicable Language Demand should be listed only once for each mode of a standard or item. As mentioned above, if you determine that a standard or item does not contain any language demands, then use the Language Demand code “Z” and explain in the comments field. Z ratings are not given a Language Complexity rating. If the Overall Language Complexity cannot be determined, then use the Overall Complexity rating of X.
A rating sheet is “complete” for linguistic analysis when it contains all of the following information:
q Rater # (refer to the label on your folder)
q Document title (standards or assessment name)
q Date
q Rating form page # (if more than two rating forms are required)
q Strand IDs
q Content Codes
1. Topic
2. Cognitive Demand
q Language Codes
1. Focus code
2. Mode code(s)
3. Language Demand code(s)
4. Density codes
5. Construction codes
6. Overall Complexity codes
q Comments (as appropriate)
If you have questions or would like additional information, please contact Edynn Sato, Ph.D. .
6 ©2008