IMS 3801 Lecture 4
Knowledge elicitation
The knowledge crew
•The knowledge practitioner; may be either a knowledge operator or a knowledge specialist. Operational level of management.
•The knowledge officer/ knowledge manager at strategic management level, who establishes the knowledge vision, and sets standards for justifying the value of the knowledge base.
And us, knowledge workers who are the information managers in the organisation, maintaining the knowledge bases
or knowledge engineers, who engineer new knowledge according to the company’s or organisation’s vision, and elicit the knowledge of knowledge practitioners
Knowledge Acquisition
What is Knowledge Acquisition?
•knowledge extraction view:
•"mining the jewels of knowledge out of [the experts'] heads one by one”
•assumes
•this view is over-simplistic(Feigenbaum & McCorduck, cited in Kidd, 1987)
Another viewof Knowledge Acquisition
•employ a technique to elicit data from the expert
•interpret the data to infer the expert's underlying knowledge and reasoning processes
•use the interpretation to guide the construction of a model (prototype)
•use the model to guide further interpretation of data
Developing a Knowledge Base
Stages in Development of a Knowledge Base System
•Select the problem
•Elicit the knowledge from the expert
•Model knowledge(represent the knowledge in an explicit form)
•Validate the knowledge
•Refine the knowledge base
•Apply the knowledge base
•Maintain the knowledge base
Knowledge Acquisition Techniques
•interviews
•presentations by the expert
•verbal protocols
•observation
•simulated consultations
•teachback techniques
•psychological techniques (e.g. repertory grids)
•induction from examples
Interview techniques
Knowledge to Acquire
•strategic knowledge -what steps does the expert take in performing the task?
•judgemental knowledge -how does the expert reason from the input data to the output results?
•factual knowledge - what characteristics of the case does the expert use in performing the task?
•subject representation - metaknowledge - how should the knowledge manager identify and classify the knowledge being acquired?
Ancillary Knowledge to Acquire
•terminology of the expert and the users
•structure of the knowledge
•reference items in expert’s knowledge base
•mode of reasoning, explanations of reasoning
•uncertainty measures and combinations
Unstructured Interviews
useful for:
•terminology;
•identifying general characteristics of the domain;
•setting scope and objectives of the proposed knowledge base;
•gaining confidence of expert (personal relations)
Structured Interviews
•describe typical cases
•describe interesting cases (critical incident technique)
•match sets of characteristics with decisions
•distinguishing goals:
•work backwards, to identify the evidence necessary and sufficient to distinguish one goal from the others
or
•work forwards, classifying evidence successively until final goal is reached
Problems with Interviews
Experimental results : a case study
•140+ knowledge engineers listened to an audio tape of interview with an expert, and took notes as if to develop a prototype;
•noted only 30% of topics the expert mentioned (on average). Of these, about 80% contained analysis errors;
•i.e., only about 6-7% of what the expert said was captured correctly.
source: Naughton, M.J. (1989)
Reference interview techniques
•Closed question (yes/no or brief answer expected)
•Multiple choice (offering of alternatives)
•Open question (interviewee can interpret question and give descriptive answer)
•Rankings of response on a scale (qualitative studies... not necessarily written. “How much”, “how little”.
•Exploratory questions to uncover purpose and assumptions (eg why do you use this information, how do you use it, when do you use it, when is it inappropriate)
Verbal Protocol
•a verbal protocol is a record of the comments made when an expert is performing the task, stating his thoughts aloud, i.e. a "thinking aloud" experiment;
•may be complemented by a "cross-examination" experiment;
•an alternative may be for the expert to comment on a videotape of his behaviour;
•often uses simulated consultations ("familiar tasks")
Protocol Analysis
•identify the objects and relations the expert refers to;
•identify the causal relationships (or influences) described:
–e.g.when deciding between an applicant with a previous record of study and one with work history, highly relevant work experience is preferable, particularly when the record of study is poor.
Observation
•an extension of the protocol analysis technique;
•expert comments on his behaviour;
•knowledge engineer asks for explanations of behaviour;
•observation may often illustrate situations where there is a difference between what the expert does and what he says he does
Simulated consultations
•a technique for adding structure to the interview;
•the expert solves typical problems in front of the knowledge engineer (“familiar tasks” - Hoffman,1987);
•the expert attempts to solve the problem aloud
•the knowledge engineer may ask "why" and "what if" type questions to explore the reasoning further
•later interviews may also use consultations on limited information tasks, constrained processing tasks or "tough cases" (Hoffman, 1987)
Teachback
•the knowledge engineer pretends to be the expert;
•the knowledge engineer "teaches" the expert how to solve the problem under study;
•the expert suggests changes to the reasoning;
•of course, requires that the knowledge engineer has some understanding of the reasoning process before undertaking this activity
Problems with verbal techniques
•the expert may not be able to verbalise knowledge.
•there is often a difference between what the expert does and what he thinks he does.
•difference in performance when under observation.
•difficulties in identifying key concepts (structuring the knowledge) although subject representation and modelling decision making with decision trees can help
•the knowledge engineer interprets the expert
•the assumed form of knowledge representation affects the knowledge acquisition process.
Caution on Introspection
•in knowledge elicitation, the expert is often asked to describe what he or she is or was thinking;
•Anderson's model of human memory (production memory) suggests that experts have no direct access to this knowledge
•and knowledge engineer’s preconceptions influence recording of knowledge
•as a result, questioning techniques may be suspect and observation should be used as a validation technique
Knowledge acquisition summary
•Knowledge acquisition is not a simple transfer of information
•Knowledge acquisition is a constructive modelling technique requiring the active participation of both the knowledge engineer and the expert
•Knowledge acquisition requires metaknowledge as well, if the knowledge manager is to use the elicited knowledge as a resource... Ford et al speak of “mediating representations” and “conceptual schemes”. In information management we commonly use subject representations and classifications of knowledge.
References