1
Raising awareness of metacognition for HE students:
A questionnaire approach to assessing metacognitive patterns
Paper presented at the British Educational Research Association Annual Conference, University of Glamorgan, 14-17 September 2005
This is a draft paper and the permission of the authors
should be sought before using any part for reference purposes
Jacquie Turnbull Education & Training Consultant,
The Business School,
University of Glamorgan
email:
Nigel Brown Senior Lecturer in Accounting
The Business School
University of Glamorgan
Tel: 01443 483726.
email:
Introduction
This paper reports on the initial findings from a questionnaire administered to 862 students at the University of Glamorgan, and discusses how analysis of follow-up interviews with students supports the results of the questionnaire. The purpose of the questionnaire was to identify metacognitive patterns that had a potential influence on students’ educational experience. The concept of metacognitive patterns has been developed from a research stream that has considered approaches to developing higher thinking skills in students.
Background
Several factors had contributed to the initiation of this research stream. First, there was the background of the endorsement of a policy of Personal Development Planning (PDP) by the representative bodies within higher education (HE) (University UK 2000). This established a requirement that HE institutions set up policies and structures to provide guidance and support for students, so that they become more effective, independent and confident learners. In particular, Progress Files were seen as a vehicle to enable students to improve their capacity to reflect on what and how they were learning, and to take responsibility for their own learning (Universities UK: para 29). Nevertheless, the authors had some concerns about the perceived focus of the Progress File format on measurable task outcomes. This seemed in line with a more general concern that learning in HE had taken something of a rational-instrumental turn in recent years, with notions of self and emotion being backgrounded (Yorke & Knight 2004).
A second influence was the increasing volume of writing on the subject of ‘higher’ thinking processes, whether termed meta-processes (Eraut 1993), metacomponents (Sternberg 1990:121), metacognitive processes (Klenowski 2002:34), or metacognitive skillfulness (Veenman & Verheij 2003:260-261). However titled, the reference appeared to be to thinking processes - of which the individual can initially be unaware - that perform an executive function in organising and monitoring problem-solving abilities, influencing motivation and the performance of tasks. The developing interest in metacognition could be linked to the interest in ‘learning to learn’ that was influencing educational policy, such as in PDP. Significantly however, Gunstone (1994) stresses that it should be a pedagogical goal to enhance metacognition itself: that, as well as having a critical impact on the achievement of content-based learning outcomes, enhanced metacognition should be viewed as a learning outcome.
Thirdly, the authors had a particular interest emerging from their experience as Master Practioners and Certified Trainers in Neuro-linguistic Programming (NLP). NLP evolved from practice as a therapeutic tool to widespread use in training, coaching, management and educational contexts. The main attraction has been the practical use that can be made of the collection of concepts and models that comprise NLP. One of these models has been meta programmes (Charvet 1997:11), which are conceived of as sorting principles by which people filter and organise sensory data to create and sustain their mental re-presentations of reality. Since meta programmes are conceived as operating at a level above or ‘meta’ to conscious thoughts, it was felt the concept could be aligned with notions of metacognition as described above. For the authors, there were particular advantages in the model as a tool for raising student awareness of metacognition, particularly the fact that the model provided an accessible language with which to describe and identify metacognitive processes. It was thought this accessibility would enhance the ability of students to ‘think about their thinking’ and be able to improve self-regulation.
In the earlier stages of the project the potential benefits of a knowledge of meta programmes for both lecturers and students had been identified (Brown 2002, 2003, 2004a, 2004b, Brown & Graff 2004). As this work developed it also became clear there were other aspects to be considered in developing a metacognitive assessment tool for use with HE students. First, the fact that meta programmes were thought to be amenable to change, and to vary in the individual according to context (James & Woodsmall 1988:92). As the commercial tools that had been tested so far had been general in nature, this suggested there would be a value in developing an assessment tool that related specifically to the education context. Secondly, the work of other theorists provided valuable insights that could not be ignored. In particular, the work of Sternberg (1997) provided theoretical support for a model of thinking ‘styles’ that were amenable to adjustment and evidenced in different combinations within individuals. In addition, the work of Dweck (2000) had established how beliefs about intelligence could have a direct effect on student goals and concerns relating to education. Overall, it was felt important to adopt a new term ‘metacognitive patterns’ to reflect the combination of influences that were being incorporated into the development of a questionnaire relating specifically to higher educational experience.
The Questionnaire
The questionnaire was designed to establish a numerical score for 24 different metacognitive patterns, that had been identified from interviews to be relevant to student educational experience (Brown 2004b). The majority of the patterns were linked as pairs conceptually opposite in nature (e.g. Towards / Away From) and could therefore be viewed as two ends of a continuum. Completion of the questionnaire resulted in a score for each individual pattern which was totalled and expressed as a percentage of the maximum score possible for that pattern. This was derived according to the responses to the Likert scale for each statement (see Fig. 1)
Fig. 1 Extract from the Metacognitive Pattern Questionnaire to illustrate the format
Strongly agree / Agree / Tend to agree / Tend to Disagree / Disagree / Strongly Disagree / Unable to Answer1. I like a college course that allows me to do a bit of everything / 6 / 5 / 4 / 3 / 2 / 1 / U
2. I find it hard to take criticism / 6 / 5 / 4 / 3 / 2 / 1 / U
3. I am good at deciding what I do not want to do for a career / 6 / 5 / 4 / 3 / 2 / 1 / U
4. I’m doing this course to give me lots of opportunities / 6 / 5 / 4 / 3 / 2 / 1 / U
The resulting score was sometimes confusing for the participants as the scores for a ‘pair’ did not add up to 100, but gave a percentage score for each pattern, for example, 69% for Towards and 53% for Away. A fuller explanation of this will be needed in future testing of the questionnaire. The original draft questionnaire aimed to test 10 statements for each pattern, but this took an estimated 40 minutes to complete. Following a round of feedback from both non-NLP and NLP-trained adults, the pilot questionnaire was reduced to a more manageable 142 statements with four to six statements per pattern. All statements were written to relate to an educational context or activities.
The sample of 862 students was drawn from 6 schools within the University, and included levels of study from Foundation to Masters level. The questionnaire was made available in paper copy and an on-line version. Students who completed the electronic version of the questionnaire received results immediately, whilst the remaining students were informed of their profile after completion of manual data inputting. Students also received a text explanation of the patterns, showing how the particular thinking style could potentially be evident in behaviour.
Statistical analysis of the Questionnaire
Initial statistical analysis of the responses indicated that 19 of the 24 question categories had a sufficiently high internal reliability coefficient (>0.53) for an instrument in its early stages of development (Duff, 2001). For the remaining 5 patterns with poor internal reliability, further work will be needed, either to eliminate questionnaire bias or to consider whether the particular pattern is a reliable conceptual interpretation of a metacognitive preference. The Cronbach’s alpha coefficients are shown in Figure 2 below. There was an additional consideration in that, of the 5 patterns that showed poor internal reliability, certain ones were nevertheless identified consistently in the follow-up interviews (e.g. Kinaesthetic, General). This would suggest further work is needed on the construction of the question items with higher internal consistency reliability.
Figure 2 Table 5 Revised alpha coefficients post factor analysis
Pattern / AlphaBased on all items
(n=862) / Alpha post factor anal.
selected items / Qs on which it is based
Towards / 0.6014 / 0.665 / Q19, 35, 68
Away from / 0.2001 / 0.346 / Q101, Q139
Through time / 0.6914 / 0.692 / Q 16, 42,52, 106
In time / 0.4547 / 0.4755 / Q 36, 74, 97, 108, 122
Proactive / 0.3705 / 0.57 / Q6, 25, 60
Reactive / 0.4022 / 0.373 / 80, 98, 137
Options / 0.5297 / 0.614 / Q44, 84, 96, 110
Procedures / 0.6463 / 0.7474 / 51, 59, 71, 87, 127
General / 0.3035 / 0.347 / Q7, 26, 61, 81
Detail / 0.5417 / 0.584 / Q27, 78, 124, 136
Internal / 0.4313 / 0.5305 / Q18, 65, 99, 104, 131
External / 0.5958 / No improvement
Visual / 0.4577 / 0.607 / 37, 62, 114
Auditory / 0.4001 / 0.698 / 79, 138
Auditory digital / 0.5373 / No improvement
Kinaesthetic / 0.3077 / 0.301 / 86, 95
Independent / 0.826 / 0.823 / 29,113, 132, 141
Co-operative / 0.734 / 0.8289 / 48, 63, 82, 91, 118
Sameness / 0.5013 / 0.581 / 28, 54, 93, 105, 115
Difference / 0.6203 / No improvement
Entity / 0.6494 / 0.745 / 31, 66, 100, 134
Incremental / 0.7766 / 0.8136 / 34, 50, 73, 103, 130
Aristotelian / 0.5467 / 0.688 / 30,49,116
Non-Aristotelian / 0.7424 / 0.80 / 32, 58, 94
Source: Brown (2005) extract from Table 5.
For those question categories with a high internal reliability coefficient, factor analysis was applied and from this areas for further research were identified. First, it became apparent that certain metacognitive patterns exhibited a tendency to cluster together to form a particular ‘profile’ of thinking patterns and behaviour. This is consistent with Sternberg (1997) and had also been a finding during the interviews with students that preceded the development of the questionnaire (Brown 2004b). Secondly, there are interesting indicators of the incidence of particular patterns relating to subject of study and gender that will require further analysis. The full statistical analysis is presented in Brown (2005) and follow-up interviews were held with students in order to further validate the statistical findings by assessing metacognitive patterns evidenced in verbal and non-verbal indicators (see below).
Student Feedback on Completing the Questionnaire
An important outcome for the development of the questionnaire was that it would be capable of raising student awareness of their thinking patterns and improve their understanding of themselves. Students who completed the paper version of the questionnaire were asked to complete a short evaluation questionnaire immediately after completing the questionnaire. These students were therefore providing feedback prior to receiving any results. Fig. 3 includes a summary of the student responses. These results suggest that the students were able to understand the questionnaire without having had any briefing on the content of the questionnaire before completion.
Figure 3 Student paper-based evaluation BEFORE they received their results.
Circle ONE number to indicate your views on this questionnaire statementsItem 1 / Easy to Understand / Hard to understand
5 / 4 / 3 / 2 / 1 / Not answered / Total
Totals for item 1 / 245 / 294 / 94 / 14 / 1 / 0 / 648
Item 2 / Relevant to my education / No relevance to my education
5 / 4 / 3 / 2 / 1 / Not answered / Total
Totals for item 2 / 108 / 284 / 196 / 44 / 15 / 1 / 647
Item 3 / Easy to relate to myself / Difficult to relate to myself
5 / 4 / 3 / 2 / 1 / Not answered / Total
Totals for item 3 / 163 / 328 / 126 / 26 / 5 / 0 / 648
Item 4 / Helpful in making me think about myself / No help in making me think about myself
5 / 4 / 3 / 2 / 1 / Not answered / Total
Totals for item 4 / 180 / 271 / 136 / 48 / 13 / 0 / 648
Source: Brown (2005) Table 8
A further way of evaluating the questionnaire was to ask the students who completed the online version, to complete a short evaluation questionnaire immediately after they received their results. The results comprised their scores for each set of metacognitive patterns, based on all questionnaire items and expressed as a %, together with a text description of the potential behaviours that relate to each of the patterns. The summary of the responses in Fig. 4 confirms the value of the actual completion of the questionnaire in raising student awareness.
Figure 4 Student online evaluation AFTER they received their results.
6: Strongly agree 5: Agree 4: Tend to Agree
3: Tend to Disagree 2: Disagree 1: Strongly Disagree
U: Unable to Answer