89
The Role of Search Strategies and Feedback on a Computer-Based Collaborative Problem Solving Task
A Proposal Presented to the
Faculty of the Graduate School
University of Southern California
San-hui (Sabrina) Chuang
University of Southern California
to
Dr. Harold O’Neil, Jr. (Chair)
Dr. Robert Baker
Dr. Dennis Hocevar
Dr. Edward Kazlauskas
Dr. Audrey Li (Outside Member)
2677 Orchard Ave. #1
Los Angeles, CA 90007
(323) 731-8344
Table of Contents
ABSTRACT………………………………..…………………………………… 4
CHAPTER I: INTRODUCTION……………………………………………….. 6
Background of the Problem…………………….………………………………. 6
Purpose of the Study……………………………………………………………. 8
Significance of the Study……………………………………………………….. 10
CHAPTER II: LITERATURE REVIEW….……………………………………. 12
Relevant Studies………………………………………………………………… 12
Collaborative Problem Solving…………….………………………………… 14
Assessment of Collaborative Learning Processes………………………. 15
Assessment of Problem Solving………….………………….………….. 18
Measurement of Content Understanding..………………………. 18
Measurement of Problem Solving Strategies…..………………... 21
Measurement of Self-Regulation…………………....…………… 24
Summary………………………………………………………..……….. 24
Information Seeking……………………………………………………..………. 26
Information Seeking in Education…..…..………………………………. 26
The Information Age and its Impact…………………………….. 26
Information Seeking and Help Seeking………………………….. 28
Curriculum Change for Tomorrow’s Workforce Readiness.……. 30
Electronic Information seeking/searching…………………………..…… 32
The Information Age and Computer Technology……………..… 32
Search Strategies on the World Wide Web……..……………..… 33
Technical Considerations in Electronic Information ……………. 38
Summary…………………………………………………………………. 39
Feedback…………………………………………………………………………. 40
Definition and Feedback Type……………………………..…..………… 40
Feedback Characteristics ………………………………………………… 43
The Amount of Information……………..……………………….. 43
Timing of Feedback ……………………………………………… 46
Representation of Feedback……………………………………… 49
Summary…………………………………………………………………. 52
Summary of the Literature……………………………………………………….. 53
CHAPTER III: METHODOLOGY………………………………………………. 59
Research Hypotheses…………………………………………………………….. 59
Research Design……………………………………………………………….…. 59
Pilot Study…………………………………………..…………………………… 59
Method of the Pilot Study……………………………………………….. 62
Participants…………………………………………………….… 62
Networked Knowledge Mapping System……………………….. 62
Simulated World Wide Web Environment……………………… 67
Table of Contents (Cont.)
Measures………………………………………………………………… 71
Group Outcomes Measures……………………………………… 71
Individual and Group Teamwork Process Measures……….…… 72
Relevant Information Found…………………………………….. 72
Information Seeking and Feedback Behavior Measures………… 73
Data Analysis……………………………………………………………. 74
The Main Study………………………………………………………………….. 74
Method…………………………………………………………………... 74
Participants………………………………………………………. 74
Measure………………………………………………………………….. 74
Procedure………………………………………………………………… 74
Collaborative Training Task…………………………………….. 75
Boolean Search Strategies Training Section…………………….. 75
Data Analysis……………………………………………………………. 77
REFERENCES ………………………………….……………………………… 80
Appendix A Self-Regulation Questionnaire……..……………………………… 92
ABSTRACT
Collaborative problem solving and cooperation skills are considered necessary skills for success in today's world. Cooperative learning refers to learning environments in which small groups of students work together to achieve a common goal, and problem solving is “cognitive processing directed at achieving a goal when no solution method is obvious to the problem solver” (Mayer & Wittrock, 1996, p.47). Thus, collaborative problem solving is defined as problem solving activities that involve interactions among a group of individuals.
Hsieh (2001) successfully improved a computer program to evaluate student collaborative problem solving and team processes on a computer-based knowledge mapping group task with a simulated Web space as an information source. The program also provides feedback to the participants. The feedback content tells the collaborative group how much they need to improve each concept in their map by three feedback groups: a group that needs a lot of improvement, a group that needs some improvement and a group that needs a little improvement. Even though her feedback provided participants a direction as to “what” area to improve for search and task performance, her feedback did not provide practical tips on “how” to improve the performance. In addition, in her study, “search” was significantly negative related to performance.
This proposal argues that by providing search strategies training and feedback, students will become more efficient in locating the information needed. Therefore, this study seeks to demonstrate that with search strategies training and feedback, students should perform better than Hsieh’s (2001) results in general.
There are three research hypotheses in this study. Hypothesis 1 is that students with task-specific/dependent adapted knowledge of response feedback will be more likely to use decision-making and leadership messages in their communication than students with task-general/independent adapted knowledge of response feedback. Hypothesis 2 is that students will perform better in the concept mapping task by receiving task-specific/dependent adapted knowledge of response feedback than by receiving task-general/independent adapted knowledge of response feedback. The last hypothesis is that information seeking using search strategies such as Boolean operators will have positive effects on problem solving strategies and group outcome performance on the concept map.
The research design is an experimental one. Students will be randomly assigned into two person groups as either searcher or leader. Then the groups will be randomly assigned into two feedback treatment groups (task specific versus task general). The dissertation will consist of a pilot study and a main study. There are two purposes of the pilot study. First, it will be used to assess the functionality of the computer system with the newly implemented feedback treatment. Second, the pilot study will be used to assess if the revised predefined messages will permit students (leaders and searchers) in each group to construct the knowledge map collaboratively after deleting the messages of low reliability. The pilot study will be done during October while the data collection of the main study will be done in December. In general, descriptive statistics (e.g., means, standard deviations) will be used to estimate all measures. In addition, MANOVA will be used to examine the relationships between outcome and process variables (i.e., variables of teamwork process and information seeking process with concept maps).
CHAPTER I
INTRODUCTION
Background of the Problem
People in the 21st century face inevitable and rapid changes in every aspect of their lives. Traditional education systems do not sufficiently address the needs of our students in this ever-changing society. Whereas teachers were once viewed as knowledge transmitters, in this new information age it is never possible to transmit enough updated knowledge to students for solving the problems they face throughout their lifetimes. In this era where change is the only constant, the greatest legacy education can provide for students is a will to learn and to continue learning as personal circumstances change throughout life (Covington, 1998).
In view of the changing needs in the workforce, the National Center for Research on Evaluation, Standards, and Student Testing (CRESST) examined five studies on workforce readiness skills. All five studies identified higher order thinking, interpersonal and teamwork skills, and problem solving as the necessary generic skills needed for success in today's world (O’Neil, 1997, Allread, & Baker, 1997). Among these skills, graduates and employees rate thinking, decision making, communications skills, teamwork and cooperation skills as the most important ones (Sinclair, 1997).
Teachers and professors at all educational levels have recognized the changes in the requisite work force skills and have tried their best to prepare their students. They incorporate problem solving and cooperative learning in daily instruction. They encourage students to develop higher order thinking skills through problem solving, and they hope that through constant practice students will harness cooperative skills. At the end, they hope to stimulate students’ motivation in learning during school and self-learning after graduation (Shapiro, 1999).
Recently, many educational assessment programs have also used collaborative small-group tasks in which students work together to solve problems or to accomplish projects to evaluate learning results (Webb, Nemer, & Chizhik, 1998). In reality, when students enter the work force, they inevitably have to work in groups. Empirically, collaborative problem solving has been shown in educational research to enhance students’ cognitive development (Zhang, 1998; Webb et al., 1998).
O’Neil (1999) defines problem-solving as consisting of three facets: content understanding, domain specific problem-solving strategies, and self-regulation. A good problem-solver understands the content well (content knowledge), possesses specific intellectual skills (problem-solving strategies), is able to plan use of her resources and skills and, during the process, monitors her own progress toward the end goal of solving the problem (self-regulation).
Content understanding can be evaluated using concept maps, which are graphical representations consisting of terms (concepts) and links (interrelationships between concepts). It is a useful tool in understanding a complex content area, especially in science learning (Hurwitz & Niemi, 1999). Self-regulation, which includes planning, self-checking, self-efficacy, and effort, can be measured by questionnaires (O’Neil, 1999). Assessment of problem solving strategies is more problematic. Most traditional assessment relies on questionnaires, self-reporting, interviews or natural observations. However, those methods are not able to capture the essence of the problem solving process (Chung, O’Neil, & Herl, 1999). Recently, with the emergence of computer technology, researchers have used computer-simulation as an assessment tool for problem solving skills (e.g., Herl, O’Neil, Chung, & Schacter, 1999; Hsieh 2001; Schacter, Herl, Chung, Dennis, & O’Neil, 1999). Computer technology with its powerful relational database makes capturing problem solving processes easier and less costly, especially in a large scale test setting. For example, O’Neil, Wang, Chung, & Herl (1998) and Hsieh (2001) used a computer-simulated teamwork task to evaluate problem solving and measure the thinking processes involved. These processes were recorded by the computers and through predefined messages that participants used to communicate with their team members. However, Hsieh’s (2001) study showed that one of the problem solving strategies, searching, was significantly negatively related to team performance.
Search strategy is one of the problem solving strategies attracting considerable attention in educational research. Cyveillance (2000) estimated in 2000 that the World Wide Web contains more than two billion publicly accessible information pages and it will continue to grow exponentially. The World Wide Web (WWW) has the advantage of being easily updated and being assessed in real time. In schools, students use the Internet more and more each day, either for email or for academic projects (U.S. Census Bureau, 1999). Currently, over 95 percent of public schools have computers (National Center for Education Statistics, 2000). According to Becker (1999), at least 40-45% of American classrooms are linked to the Internet, and almost 90% of teachers perceive the World wide web as an important research source for their teaching. More and more schools demand that students do research on the World wide web so that they can satisfactorily complete their projects (Breivik, 1998; Ercegovac, 1998; Roblyer, 1998). In fact, research is the most common classroom Internet use (Becker, 1999).
Purpose of the Study
The primary purpose of this study is to improve Hsieh’s (2001) results by providing more task-related feedback to the participants. Hsieh provided feedback on the task progress. A simulated Web space used for information seeking and feedback accessing was used in Hsieh’s (2001) study. She demonstrated the following: (a) students who received adapted knowledge of response feedback significantly outperformed students with knowledge of response feedback, and (b) information seeking and feedback accessing were positively correlated with students’ outcome performances. However, surprisingly, within information seeking, searching for the adapted knowledge of response groups was negatively related to group outcome. She did not offer an explanation.
In order to improve searching, this author suggests including searching strategies training and including them in the feedback. The rationale is that even though Hsieh’s (2001) feedback provided participants information about how much they accomplished the task, and new direction as to “what” area to improve their search and their concept map, her feedback did not provide practical tips on “how” to improve the map. This proposal argues that by providing search strategy training and feedback, students will become more efficient in finding the relationships between the concepts and in turn improve the overall result of the map. Therefore, the author will modify the original task by providing examples on one specific type of search strategy, use of Boolean operators, in addition to the original information on what area the participants should improve next. Because the participants were high school students, they were considered as lacking the perquisite knowledge to do the task. By providing search tips that have proven to be effective, participants in the proposed study should perform better than Hsieh’s (2001) students in general. In addition, increased use of the Boolean operators when doing the simulated web search is expected. With the search tips, students should be able to get more relevant relationships among the concepts and should be able to construct a better map through efficient information seeking and self-monitoring of their performance through feedback.
Based on Hsieh’s (2001) study and adding search training and feedback, the proposed study will evaluate student collaborative problem solving and team processes on a computer-based knowledge mapping group task with special attention to the effectiveness of information seeking/searching. Hsieh (2001) compared the effects of two levels of feedback, i.e., knowledge of response feedback and adapted knowledge of response feedback. Her study indicated that the group with adapted knowledge of response feedback outperformed the group with knowledge of response feedback. The proposed study will compare two levels of adapted knowledge of response feedback. One is Hsieh’s (2001) general task-progress feedback (how much progress is made) and the other one is the improved one with more specific task feedback (how to better search the simulation web).
The dissertation will consist of a pilot and a main study. A pilot study will be run because even though Hsieh (2001) has proven the feasibility of the computer simulation program and her participants were able to successfully complete the task with the predefined messages, some scales and messages had unexpectedly low reliability. Those messages will be deleted and a pilot study will be run to test whether a group of a searcher and a leader can still complete the task using the remaining messages.
Significance of the Study
Problem solving skills and teamwork skills have been regarded as the most important skills for both the workplace and for the school context. A new problem solving skill coming from the advance of computer technology and the World Wide Web is search strategies. The present study is significant for the following four reasons. First, it represents one of the earliest attempts to use computer simulations to report and capture the process of cooperative problem solving with an emphasis on search strategies. Second, while some studies have been done in the area of feedback and collaborative problem solving (Tudge et al., 1996), very few studies have explored the effects of types of feedback on collaborative learning in computer related instruction. The present study will investigate the effects of two different types of feedback (task general/dependent and task specific/independent) on collaborative problem solving. Third, use of search strategies has been considered in education as a characteristic of expert problem-solving. Research has shown that with simple problems, there is not much difference between a novice search and an expert search (Kubeck, Miller-Albrecht, & Murphy, 1999). However, when it comes to more difficult problems, expert search is by far more efficient and more accurate in locating the information needed. With the participants in Hsieh’s (2001) study achieving less than satisfactory results (the highest score possible = 20, Mean for all students = 9.9) in the map construction, the current task should be considered as a more difficult one. Therefore, by adding the feedback on search strategies, this study seeks to demonstrate that students should be able to locate the information needed in a timely fashion and, at the end, achieve better results in the map construction due to better problem solving strategies. Fourth, the computer simulation used in this study not only records but also evaluates students’ process and performance on cooperative problem solving. In addition, it also reports back to students during the process on how they are doing and how they can improve. It is a dynamic assessment tool (Grigorenko & Sternberg, 1998). The study also represents an attempt to improve the current computer simulation program. It does so by identifying searching as one of problem solving processes and adds it to the program. Hopefully, it eventually will become an automatic learning and an assessment tool.