Stevens BS-CS Program Evaluation Report for 2009-10 June 14, 2010

Stevens BS-CS
Program Evaluation Report for 2009-10
June 14, 2010

The report is divided into two sections, process and results. The process section evaluates the process used to assess the program. The results section evaluates the extent to which the program is meeting its goals (i.e., objectives and outcomes). Each section contains two subsections: discussion and planned improvements.

1. Process

1.1. Discussion

This year we succeeded in fully enacting our assessment process. We performed all planned surveys: senior exit survey, employer objectives, alumni objectives, and alumni outcomes. We also quantitatively assessed all required courses and all ABET outcomes A through K. The results of all surveys and quantitative measures are available at Per-course information is available

In October we had an ABET site visit and the site visit team suggested two improvements to our process:

1. To write and publish a "continuous improvement plan" that details all aspects of our assessment plan.
   We have done this. The document is available at
2. To completely plan a course before its commencement, including the complete text of all instruments and grading rubrics.
   For this purpose we have designed a new form, the "Course Plan" (CP) form. It is available at started planning courses this way in the Spring 2010 semester. It is a great deal of work but once the course is underway assessment becomes mechanical. Although we publish all our course forms at the CP forms are not visible there because instructors do not want answers and grading rubrics posted publicly. We are debating whether password protection provides sufficient security for such information or whether it is wise to keep such material off the Web altogether.

1.2. Planned Improvements in the Assessment Process

At this time we are happy with our assessment process which combines direct data (measurement of student performance) with indirect data (surveys of graduating seniors, alumni, and employers). We plan no changes until future evaluations suggest that improvements are needed.

The primary concern about the process is the difficulty of obtaining “objectives survey” responses from employers. Last year we received only 12 responses and this year the number dropped to 3, a number too low to be an effective measure. We are working with the Stevens Career Services Office (CSO) to increase the number of responses. The CSO has contact information for many companies that hire our graduates year in and year out. However, the CSO often has contact information for Human Resources personnel or the campus recruiter. Neither is necessarily the type of person who can evaluate the job performance of our graduates; some are, some aren’t. Therefore, another step we are taking is to enlarge the department’s Industrial Advisory Board (IAB). The IAB is a set of employers who have a long term record of hiring Stevens BS-CS students. We are in email contact with the board throughout the year and the board meets once per year in the September/October time frame. The board presently consists of 6 members. We plan to increase the board’s size to 12 or more. Board members have a much tighter bond to the department than most employers; for example, this year only IAB members responded to the employer survey. We are hopeful that expanding the IAB to 12-15 members will guarantee a steady state response of 10 or more to the yearly employer objectives survey. While 10 is not enough, it would serve as a useful base so that if CSO efforts are successful perhaps we can expect 20-25 responses per year.

2. Results

We evaluated our program according to the following process. First we examined our 5 data sources[1] to determine which program outcomes were poorly achieved or lightly covered. Second, for poorly achieved outcomes we drilled down into the per-course SPAD data to determine which assessment instruments had unacceptably low student performance scores. We consulted the instructors of those courses and asked them how they planned to improve curriculum, delivery, the instrument, or anything else in order to achieve better student performance in the future. Third, for outcomes that are too lightly covered we solicited opinions from the faculty for how our curriculum could be augmented to include additional coverage of these outcomes. The sections below provide additional detail on the process, our conclusions, and our plan for improving in the coming academic year.

2.1. Discussion

2.1.1. Direct Evidence

The program summary SPAD data (from is shown below:

All but four outcomes achieved 77% success or better. The four laggards are outcome #4 (application of discrete math, ABET A, 64.0%), outcome #16 (application of software engineering best practices, ABET C, 64.9%), outcome #8 (communication, ABET F, 65.3%), and outcome #6 (runtime organization, not an ABET requirement, 70.8%).

In addition, we note that outcomes #7, #10, and #11 are lightly covered: judging by the number of students assessed, seemingly a single instrument in a single course served to establish each outcome across the entire curriculum. This is insufficient coverage, especially considering that all three are ABET requirements (#7 is ABET D, #10 is ABET E, and #11 is ABET G).

Therefore, analysis of 2009-10 SPAD data yielded 7 program outcomes (out of 19) potentially in need of improvement: 4, 6, 7, 8, 10, 11, and 16. Because there is some variation in results year to year, we examined the SPAD data from 2008-09. In that year all but three outcomes were established with at least 79% success. The three outliers that year were outcomes 4, 14, and 18; in 2008-09 outcome #4 was once again the outcome with the lowest success rate (65.3%). In 2008-09 outcomes 9 and 10 were lightly covered by our curriculum. By this line of thinking—looking at two years of SPAD data—we decided to focus on outcomes #4 (application of discrete math, ABET A) and outcome #10 (ethical issues, ABET E) as particular concerns, Outcome #4 suffers from continuing insufficient performance while outcome #10 suffers from continuing light coverage.

2.1.2 Indirect Evidence

Incorporating indirect evidence (survey results) into our analysis, we saw thatour 1-year-out alumni rated outcomes 4, 8, 10, 11, and 15 as those least well established in their opinions. Of these, outcome #4 stands out as clearly the weakest. All of these but #15 map to ABET-required outcomes.

Looking at the past three senior exit surveys, outcomes identified as weak in at least two of the three years are: 4, 14, 15, 18, and 19. Of these, #4 is the most worrisome because it is the only one of these that maps to an ABET-required outcome and because it has already been identified as a problem by every other measure.

We also have results from two objectives surveys: those of employers and three-years-out alumni. In order to relate performance on objectives to program outcomes, we had to create a mapping. The mapping is intended to show which program outcomes support which objectives. If an objective is identified as a weakness then we will investigate how to improve our program's ability to establish all the supporting outcomes. Our mapping is shown on the next page.

Results from both alumni and employer objectives surveys were consistent in identifying objectives #3 and #5 (communication and ability to evaluate impact) as the weakest abilities of Stevens BS-CS graduates. We map these objectives to outcomes 8, 10, and 11. As noted above, 1-year-out alumni also identified 8, 10, and 11 (and two others) as problem areas even when given a different survey that asked about outcomes rather than objectives.

2.1.3 Conclusion

Based on the above analysis, we concluded that the program outcomes that most require remediation are:

• Outcome 4 (ABET A), application of discrete math
• Outcome 8 (ABET F), oral and written communication
• Outcome 10 (ABET E), ethical problems facing computer scientists
• Outcome 11 (ABET G), analyze local and global impact of computing

2.2. Planned Improvements in Curriculum and Teaching in 2010-11

Having identified program outcomes 4, 8, 10, and 11 as those most in need of remediation, we then examined the rows of the summary SPADs posted at to determine which courses and which instruments require improvement.

There are two situations to consider: outcome #4 is adequately covered (coverage in 5 courses, often with multiple instruments per course) but must be taught more effectively, while the other three outcomes are inadequately covered (only 3 instruments acrosstwo required courses for outcome #8; only a single instrument across the entire curriculum for each of outcomes #10 and #11). We consider each situation separately in the two sections below.

2.2.1. Improvement of Instruction for Outcome #4, Application of Discrete Math

Course CS 135, Discrete Structures, is our primary discrete math course. Most students take it in the second semester of the freshman year. Most instruments in this course achieve high success rates of 80% or more, up to 100%. However, two instruments had success scores below the 70% mark. To improve scores on these instruments, the instructor plans to reorder the coverage of topics so that students are better prepared by the time the instrument is given. He writes:

I think it would be good to re-order the topics so that the basics about divisibility and modulo arithmetic are introduced around the 4th week. This actually fits with the Rosen textbook. I had chosen to cover topics that are later in the book first, in order to get to induction sooner. By covering the number theory basics sooner, we can use those topics as additional examples and exercises on subsequent topics (relations, etc), and hopefully this will reinforce this course outcome.

CS 496, Principles of Programming Languages, is a sophomore-year course that introduces the theory of programming languages and shows how to apply this theory to translator design and reasoning about programs. The two instruments in this course related to outcome #4, each a question on the final exam, generated extremely low success scores of 22% and 9%. The instructor plans to move the material earlier in the course, supposing that students' attention trails off toward the end of the semester. She writes:

Outcome 12 was not achieved to my satisfaction. It is the last topic of the semester and students tend to neglect it for the final exam. To improve the achievement level of Outcome 12, type systems will be introduced earlier in the term and a homework or quiz will be administered before the final exam.

CS 511, Concurrent Programming, is a junior-year course that concentrates on the practical aspects of programming concurrent and parallel systems. The course includes a "discrete math" section on the automated analysis of concurrency properties. The course contained two instruments related to this discrete math outcome. One had a success score of 100% while the other's score was only 33%. The instructor attributes the low score to the "surprise quiz" assessment instrument. He will consider discontinuing surprise quizzes next year since scores on these quizzes are significantly lower than scores on other types of instruments, across multiple topics. He writes:

The underperforming objective #13 of CS 511 was tested in the first surprise quiz of the course. Based on the overall performance of the students, I believe that this underperformance is caused by the measurement instrument.

To test this hypothesis, the SPAD data was analyzed with respect to measurement instruments. The aggregate proportion of acceptable performances is 83% for non-surprise quiz as compared to 72% for surprise quizzes. The corresponding Chi-square value is 1.4e-13, which indicates a strong relationship of performance to measurement instrument.

Because the measurement instrument biases our measurement, we will repeat the evaluations for this objective #13 in a non-surprise format during the next offering. We will also consider eliminating the surprise quiz measurement instruments.

The other two courses that contribute to the establishment of the discrete math outcome are CS 334 and CS 442. CS 334, Automata and Computation, is a course that covers the mathematical theory of computation. CS 442, Database Management Systems, mostly covers practical aspects of SQL programming but also includes material on normalization theory. In both courses, instruments covering outcome #4 had adequate success scores.

2.2.2. Improvement in Coverage for Outcomes #8, #10, and #11

Outcome #8 (oral and written communication) can easily be covered more. Many points in our curriculum include project assignments, and often the students are required to make a presentation and/or write a report about their work. By formalizing these requirements, we can add instruments for this outcome. Required courses that include project components are CS 146 (Web Fundamentals, freshman year), CS 442, CS 551, and CS 552 (551 and 552 together form the year-long senior course). For next year the CS 146 instructor plans to make use of the expertise at the Stevens Writing and Communication Center (

I was thinking of having the writing center make a presentation on business reporting and presentation skills. I think I will also require the students to have their first written assignment reviewed by the writing center. I also hope to have them do mini presentations every few weeks.

Similarly, project reports (written and oral) will be added to CS 442, the junior-year database course.

We have ideas but not yet any concrete plans for increasing the coverage of outcome #10 (ethical problems facing computer scientists). These ideas include:

• Adding ethics segments to one or more required courses. These segments would highlight ethical challenges in the topic areas covered by these courses.
• Creating a 1-credit course taught in computer science focusing on computer issues such as privacy vs. law enforcement, software piracy, jurisdictional issues, patent controversies and intellectual property, and so on.
• Using Stevens's College of Arts and Letters (CAL). One of CAL's major focus areas is ethics as applied to technological situations. The instructors in CAL are highly educated and capable. However, there is some concern that CAL courses might be "overkill," being so broad, or drilling down into bioethics or other non-computer technologies such as energy production, that the application of the principles of ethics to computer science might be lost on students. There is a school of thought that the 1-credit approach, even when taught by ethics non-specialists in computer science, might be preferable.

How to improve computer-ethics instruction will be a major topic of discussion for the CS department curriculum committee in the coming year.

Regarding outcome #11 (local and global impact of computing), we have an entire CAL course, HSS 371 Computers and Society, on this topic. This course has previously been taught by an adjunct instructor who, while dedicated and energetic, is neither a specialist in the area nor as well connected to the needs of Stevens undergraduate curricula as a full time faculty would be. Beginning in 2009-10, this course was transferred to a highly capable instructor full time tenure track professor in CAL. This instructor is cooperating fully with us with respect to topic coverage, instruction, and assessment procedures. Although only a single instrument was documented this year, we believe the "impact" program outcome is now being well met by the current offering of this course and that this fact will be well documented in future years.

[1] Employer objectives survey, alumni objectives survey, senior outcomes survey, alumni outcomes survey, and student performance data collected on a per-instrument basis then combined into course and program SPAD forms.