Worksheet for Design Phase of Engineering Course Design in
Tier 2 Institutions
The Worksheet enables a teacher or a group of teachers to design an engineering course in Engineering Undergraduate Programs offered in Tier 2 non-autonomous colleges affiliated to Universities. Instructional System Design Model ADDIE provides the framework for the process used here to design engineering courses. The Worksheet presented here is confined to the Design Phase of ADDIE model. It is assumed that the engineering program is being designed to meet the Program Outcomes(POs) as identified by National Board of Accreditation, India, and Program Specific Outcomes (PSOs), 2 – 4 in number, as identified by the Department offering the program.
A four year engineering program aims at meeting all the Program Outcomes and Program Specific Outcomes.It is assumed that a course can only meet a subset of POs and PSOs. While there are many dimensions of learning, the course design process presented here mainly focuses on Cognitive dimension.
The three key principles of course design are
- At the end of a course the students are to acquire stated Course Outcomes
- Assessment should be in alignment with the stated Course Outcomes
- Instructional activities are to be designed and conducted to facilitate students to acquire the stated competencies which are elaborations of course outcomes
At the end of Design Phase the course designer generates an item bank that is consistent with the assessment pattern the instructor chooses.
Design phase consists of
- Creating sample Test Items in complete alignment with Course Outcomes
- Setting targets for CO attainment
- Developing rubrics for assessment of activities associated with ACOs if any
- Determining the Assessment Pattern
- Creating the Item Bank
Development of Instruction Materials and Learning Materials for the Instructional Units (competencies) identified in the Design phase are undertaken during the subsequent Development Phase
Note:
- All materials presented in blue color are either instructions or assumptions. Background materials and samples are presented as pdf files which need to be consulted if in need of clarifications or this worksheet is used for the first time.
- When the final document is to be presented all the materials in blue color and the pdf files can be deleted.
Any interested teacher can freely use this worksheet to design his/her course. It would be appreciated greatly if you can give any comments and suggestions at on any aspect of this Worksheet.
Course:
Credits:
Course Designers:
SNo. / Name / Email IDSample Assessment Test Itemsfor Course Outcomes
Good learning requires addressing higher cognitive levels. It is the assessment that determines the quality of learning. TestItemsorAssessmentItems or simply Items are usedas measure ofstudents’ acquisitionofthecompetencies.Theycanbeeitherwrittentestitems(suchasquizzesandproblems)orperformancetestitems(suchasdiscussions, simulations, laboratory exercises, reports, field surveys, presentations, mini-projects etc.). A Test Item/Assessment Item is a unit that consists of a question, hints, sample answer, etc. Initially it will be tagged by the cognitive level and relevant categories of knowledge.
The test items are created in two stages.
- Create four sample test items for each Course Outcome and Additional Course Outcomes if any. These sample test items should be in complete alignment (same cognitive level and knowledge categories) with the COs/ACOs.
- Create the Item Bank as required by the Assessment Pattern.
Assessment Test items at the CO/ACO level should be written in a manner indicating the integration of knowledge and skills associated with it.
Test/AssessmentItems for CO1
Course Outcome / POs/PSOs / CL / KCCO1
TI1
Ans.
TI2
Ans.
TI3
Ans.
TI4
Ans.
Test/AssessmentItems for CO2
Course Outcome / POs/PSOs / CL / KCCO2
TI1
Ans.
TI2
Ans.
TI3
Ans.
TI4
Ans.
Test/AssessmentItems for CO3
Course Outcome / POs/PSOs / CL / KCCO3
TI1
Ans.
TI2
Ans.
TI3
Ans.
TI4
Ans.
Test/AssessmentItems for CO4
Course Outcome / POs/PSOs / CL / KCCO4
TI1
Ans.
TI2
Ans.
TI3
Ans.
TI4
Ans.
Test/AssessmentItems for CO5
Course Outcome / POs/PSOs / CL / KCCO5
TI1
Ans.
TI2
Ans.
TI3
Ans.
TI4
Ans.
Test/AssessmentItems for CO6
Course Outcome / POs/PSOs / CL / KCCO6
TI1
Ans.
TI2
Ans.
TI3
Ans.
TI4
Ans.
Test/AssessmentItems for CO7
Course Outcome / POs/PSOs / CL / KCCO7
TI1
Ans.
TI2
Ans.
TI3
Ans.
TI4
Ans.
Test/AssessmentItems for CO8
Course Outcome / POs/PSOs / CL / KCCO8
TI1
Ans.
TI2
Ans.
TI3
Ans.
TI4
Ans.
Rubrics for assessment of activities associated with Additional Course Outcomes (ACOs) if any
Tier 2 institutions need to identify curricular gaps and the departments have to make decisions with regard to additional modules that need to be included in some selected courses. Though CIE and SEE does not include any summative assessment of ACOs some assessment of acquiring of ACOs needs to be conducted, and the evaluation of student performance should be documented. There can be a much wider choice of assessment mechanisms besides written tests that may include writing term papers on topics associated with ACOs, assignments, discussions, and creating laboratory experiments. Identify the assessment mechanisms to measure the performance in activities associated with ACOs and the associated evaluation mechanisms.
Targets for CO Attainment
All the processes required for accreditation need to close the quality loop (Deming’s Quality Cycle)
Target levels of attainment of Course Outcomes (COs) are set; Course is delivered; actual attainment of COs is determined; AND
the loop is closed either by increasing the target level for the next offering of the course orby planning suitable improvements in the teaching/learning process to increase the actual attainment so as to reach the target
There are several ways of setting targets for COs. One method is to set target for each CO of a course separately as indicated in the following table.
CO / Target (Class Average Marks)CO1 / 70%
CO2 / 80%
CO3 / 75%
CO4 / 65%
CO5 / 70%
CO6 / 80%
This method does not directly indicate the distribution of performance among the students. It has the advantage of finding out the difficulty of specific COs. Course designer may choose his/her own way of setting targets. But one has to choose a method that does not become too complicated when applied to a few hundred courses in a College or is not so simple to render it inconsequential.
CO Target Matrix
CO / Target (Class Average Marks)CO1
CO2
CO3
CO4
CO5
CO6
CO7
CO8
ACO Target Matrix
ACO / Target (Class Average Marks)ACO1
ACO2
Assessment Pattern
The assessment pattern varies from institution to institution. It is broadly divided into Continuous Internal Evaluation (CIE) and Semester End Examination (SEE). The relative weightages, given to these two to determine the final grade, varies over Tier 2 institutions (20:80 to 30:70) in India. The final examination is conducted by the University to which the College/Institution is affiliated. Much of the assessment requires written responses.
Percentage WeightageCIE
SEE
Continuous Internal Evaluation: Delete/add rows as required
Item / WeightageQuiz 1
Quiz 2
Term Examination 1
Term Examination 2
Assignment 1
Assignment 2
Mini Project
Laboratory/Simulation
Term Paper/Report 1
Term Paper/Report 2
Presentation
Semester End Examination
Item / WeightageSEE
The instructors in Tier 2 Colleges have no direct say in the structure and content of the final examination papers. The Controller of Examinations of the affiliating University arranges for getting final examination papers designed as per the procedures of the University. These examination papers, at present, are not required to follow any assessment pattern of cognitive levels. The instructor at a Tier 2 Institution has freedom only with respect to CIE.
It should also be noted there will be no summative assessment associated with ACOs if any.
Pattern for Continuous Internal Evaluation
CIE activitieshave to be designed with specified weightages to different cognitive processes. The weightages will depend on
- Nature of the subject
- Course Outcomes that the students are supposed to achieve before the intended test
While it is essential that assessment should be in alignment with the Course Outcomes, that is, the test items should be located in the same cell as that of Course Outcomes, we may also have test items from the cells of the taxonomy table which belong to the same row but to cognitive levels lower than that of the cell where Course Outcomes is located. The weightage given to test items belonging to the lower cognitive levels is decided by the instructor. But the significant weightage should be given to the test items that are in direct alignment with the Course Outcomes (COi{Ai}). This is illustrated in the taxonomy table given below.Where {Ai} represents the set of Assessment items that are in direct alignment with the Course OutcomesCOi. {Ai1} represents the set of Assessment items that are at one cognitive level lower than that of competency COi, and {Ai2} represents the set of Assessment items that are at two cognitive levels lower than that of Course OutcomesCOi.
Knowledge Categories / Cognitive ProcessesRemember / Understand / Apply / Analyze / Evaluate / Create
Factual
Conceptual
Procedural / {Ai2} / {Ai1} / COi, {Ai}
Metacognitive
Fundamental Design Concepts
Criteria and Specifications
Practical Constraints
Design Instrumentalities
Note: In case of Basic Sciences, Humanities and Social Sciences engineering specific categories of knowledge, namely, Fundamental Design Concepts, Criteria and Specifications, Practical Constraints and Design Instrumentalities are to be ignored.
While assessment of a course outcome should be dominantly in alignment with cognitive level of the course outcome, assessment at lower cognitive levels is still required, especially if it is formative.A survey of End Semester Assessment Instruments indicates that significant percentage (anywhere between 60 and 100%) of assessment items belong to Remember and Understand cognitive levels. Besides, standard answers are expected most of the times in view of large number of students. This has a great influence on the quality of learning. In order to ensure that the students are motivated to learn well, significant percentage of items should be in alignment with the cognitive level of the course outcome.
All Course Outcomes of a course are not uniform (in terms of number of required classroom sessions) in scope. Therefore, the weightage in terms of number of marks associated with items related to a Course Outcome should be accordingly should be adjusted.
The following guidelines are suggested to identify the Assessment Pattern that needs to be communicated to the students
- The weightage to items that are in complete alignment with the COi should be at least60%.
- The remaining percentage of items can be distributed among the lower cognitive levels as decided by the instructorpreferably giving more weightage to higher cognitive levels.
- Percentage marks associated with a course outcome should be approximately proportional to the number of associated classroom sessions, with a provision for the instructor to modify.
- Weightage in marks should be treated as approximate.
Let us consider a sample course, with six COs, having two mid-term tests (Test 1 and Test 2) and one end-semester examination. Assuming Test 1 assesses course outcomes CO1, CO2 and CO3, and Test 2 with CO4, CO5 and CO6, their assessment pattern is determined as indicated in the following table.
CL / Class Sessions (approx.) / T1 (CO1, CO2, CO3)% wt / T2 (CO4, CO5, CO6)
% wt
Wt / R / U
0.6 Wt / Wt / R / U / Ap
0.6 wt
CO1. / U / 3 / 3/16~19% / 8% / 11% / 0 / 0 / 0 / 0
CO2. / U / 9 / 9/16~56% / 22% / 34% / 0 / 0 / 0 / 0
CO3. / U / 4 / 4/16~25% / 10% / 15% / 0 / 0 / 0 / 0
CO4. / Ap / 10 / 0 / 0 / 0 / 10/24~42% / 7% / 10% / 25%
CO5. / Ap / 8 / 0 / 0 / 0 / 8/24~33% / 5% / 8% / 20%
CO6. / Ap / 6 / 0 / 0 / 0 / 6/24~25% / 4% / 6% / 15%
Sessions / 40 / Total / 40% / 60% / 16% / 24% / 60%
CO / T1 (CO1, CO2, CO3) / T2 (CO4, CO5, CO6)
CO1. / R / U / R / U / Ap
CO2. / 8% / 11% / 0 / 0 / 0
CO3. / 22% / 34% / 0 / 0 / 0
CO4. / 10% / 15% / 0 / 0 / 0
CO5. / 0 / 0 / 7% / 10% / 25%
CO6. / 0 / 0 / 5% / 8% / 20%
CO7. / 0 / 0 / 4% / 6% / 15%
Total / 40% / 60% / 16% / 24% / 60%
The assessment pattern that will be communicated to students,with the SEE completely determined by the University,with some rounding will appear as
Cognitive Level / Term Exam 1 / Term Exam 2Remember / 40 / 15
Understand / 60 / 25
Apply / 0 / 60
Analyze / 0 / 0
Evaluate / 0 / 0
Create / 0 / 0
Determine the assessmentpattern of your course following similar procedures.If there are more summative CIE instruments, besides term examinationslike assignments, projects, reports and presentations involved than indicated in the table add the required columns
Cognitive Level / Term Exam 1 / Term Exam 2 / Assignment 1 / Laboratory / Mini ProjectRemember
Understand
Apply
Analyze
Evaluate
Create
Note:
- Modify the columns as required
- Percentages indicated need to be treated as approximate.
- Final examination set by the University does not normally have a declared Assessment Pattern
Item Bank
Item bank is a collection of test items from which specific assessment instruments required for limited time duration (mid-term tests, and quizzes) can be created. The test items can be different in their scope, and carry different marks. For example, most of the written examinations have 1-mark, 2-mark, 5-mark and 10-mark questions. Items, therefore, should be created with different marks for all course outcomes. The item bank should be planned as per the requirements of assessment pattern. The item bank, therefore, has to be defined in terms of number of questions under the relevant cognitive categories, and marks associated with them.
The item bank should be large enough to discourage memorizing. The active item bank to meet the requirements of CIE can start with about 500 items. It should be constantly updated, say retiring 10% of items and adding 10% new items every year. The number of items with numerics should be made as large as possible. Every time such an item is chosen to be included in an assessment instrument, the numerical values can be changed. This will discourage memorization.
The distribution of items in a 500 item bank as per the chosen assessment pattern will be
CO / CL / Class Sessions (approx.) / Wt. to CO / Items distribution in a 500 item bankR / U / Ap
CO1 / U / 3 / 3/40~7% / 14 / 21 / 0
CO2 / U / 9 / 9/40~23 / 46 / 69 / 0
CO3 / U / 4 / 4/40=10 / 20 / 30 / 0
CO4 / Ap / 10 / 10/40=25 / 20 / 30 / 75
CO5 / Ap / 8 / 8/40=20 / 15 / 25 / 60
CO6 / Ap / 6 / 6/40=15 / 10 / 20 / 45
Total / 40 / 100 / 125 / 195 / 180
These are distributed according to marks associated with the test items and the sample item bank is now shown as
CO / Proposed item distribution as per marksR / U / Ap / An / E / C
CO1 / 14(6+8+0+0) / 21(8+13+0+0) / 0 / 0 / 0 / 0
CO2 / 46(15+31+0+0) / 69(25+44+0+0) / 0 / 0 / 0 / 0
CO3 / 20(10+10+0+0) / 30(15+10+5+0) / 0 / 0 / 0 / 0
CO4 / 20(10+10+0+0) / 30(10+10+10+0) / 75(0+30+25+20) / 0 / 0 / 0
CO5 / 30(20+10+0+0) / 50(10+20+20+0) / 60(0+20+30+10) / 0 / 0 / 0
CO6 / 20(15+5+0+0) / 40(20+20+10+0) / 45 (0+10+15+20) / 0 / 0 / 0
Total / 125 / 195 / 180 / 0 / 0 / 0
The numbers in the brackets indicate distribution of questions as per marks. For example 20(6+14+0+0) means 20 items are to be created with six 1-mark items, fourteen 2-mark items, zero 5-mark items and zero 10-mark items. The item numbers shown should be treated as approximate.
Item Bank Structure
CO. / CL / Class Sessions (approx.) / Wt. to CO / Items distribution in a 500 item bankR / U / Ap / An / E / C
CO1
CO2
CO3
CO4
CO5
CO6
CO7
CO8
Total
Item Bank structure including the marks distribution
CO / Proposed item distribution as per marksR / U / Ap / An / E / C
CO1
CO2
CO3
CO4
CO5
CO6
CO7
CO8
Total
Each test item in the Item Bank is to be tagged with Course Outcome, Cognitive Level, Knowledge Category, Marks assigned, Time taken in minutes by the average student to answer, and Difficulty Level.
Cognitive Level: Remember, Understand, Apply, Analyze, Evaluate or Create
Knowledge Categories: Factual, Conceptual, Procedural, Metacognitive, Fundamental Design Concepts, Criteria and Specifications, Practical Constraints and Design Instrumentalities.
Note that an Item can address more than one category of knowledge.
Marks assigned: 1, 2, 5 or 10
Time in Minutes: It indicates the time likely to taken by an average student to answer the question. Generally this time is considered to be three times time taken by the instructor who designed the test item.
Difficulty Level: Three difficulty levels are considered. 1: Slightly, 2: Moderately and 3: Very
Note: Some of these tags and numbers associated with them are determined by the University/Schools/Departments and sometimes by the Instructors
Items in the Item Bank are to be created as per the following format. (Comp. – Competency; CL- Cognitive Level; KC- Knowledge Categories)
Test Items
Create test items for the Item Bank as per the pattern created
ItemCO / CL / KC / Marks / Time in Min. / Difficulty Level
Sample
Answer
Design Phase is followed by Development Phase consisting mainly of developing instruction material for all instructional units, and selecting or developing learning material.
N.J. Rao/April 20161