Intelligent Tutoring Tools for Cognitive Skill Acquisition in Life Long Learning

 

Educational Technology & Society 3(1) 2000 ISSN 1436-4522 Intelligent Tutoring Tools for Cognitive Skill Acquisition in Life Long Learning Ashok Patel CAL Research & Software Engineering Centre De Montfort University, Leicester, United Kingdom Tel: [phone] Fax: [phone] [e-mail] Kinshuk Information Systems Department Massey University, Palmerston North, New Zealand David Russell Leicester Business School De Montfort University, Leicester, United Kingdom ABSTRACT Cognitive skills are concerned with analysis, interpretation and decision making required for performing the procedural tasks. Since much of its process runs inside a human mind, the development of cognitive skill may largely rely on self-explanation as a mechanism for forming better procedures by clustering elementary operations and pushing them in to subconscious level. It is important for a tutoring system designer to consider acquisition of cognitive skills, particularly by life long learners who may not be learning in the traditional setting. The paper describes possible categorisation of learning resources to match the different phases of skill acquisition. It discusses an implementation of a cognitive apprenticeship-based learning environment and an independent feedback on its use in real classroom. It also touches very briefly on the far-reaching potential of such methodology when extended to the Internet. Keywords: Cognitive skills, Intelligent tutoring, Life-long learning 1. Introduction Domain competence for any domain consists of domain knowledge, physical skills and cognitive skills. The physical skills constitute physical expertise of the procedural tasks including the use of any tools. Cognitive skills are concerned with the cognitive processes of analysis, interpretation and decision making required for "carrying out" the procedural tasks. Although both physical and cognitive skills demand a fair amount of learning and rehearsing, physical skills are relatively easier to acquire due to their external visibility. The development of cognitive skill demands a more sophisticated learning process as much of its process runs inside a human mind and may largely rely on self-explanations as a learner studies an example and attempts a given problem. Though the underlying mechanism is not fully understood, self-explanations not only construct better problem solving procedures but they also help the learners in understanding the underlying principles more completely (Chi et al, 1989). The discussion in this paper is mainly based around those domains, where the physical reality is represented as numeric models and the outcomes are determined by manipulating the numeric model. Such domains are second order in nature as the practitioners work with concepts that are already abstracted to a greater or lesser degree. For example, engineers deal with the physical conditions as represented by suitable mathematical measures such as force, gravity, velocity etc. and accountants deal with all business realities reduced to money - a social construct for measuring utility and wealth. Perhaps it is the second order nature of the domains and the consequent requirement of greater emphasis on cognitive skills, that makes them more difficult to grasp and frequently the learners in the traditional educational setting are found to under achieve in these domains. Besides these factors, the traditional learners also experience perhaps greater problems relating to motivation and failure to grasp the relevance of learning. This issue is briefly discussed later in the next section. A major part of learning in a numeric domain involves working with a set of variables representing the various concepts of the domain. A learner needs to understand the inter-relationships between these variables; recognise the available instances of variables for a particular problem space (the independent variables); determine the proper sequence of resolving the dependent variables and resolve them employing forward or backward chaining as required to attain each sub-goal for arriving at an overall solution. The elementary operations carried out on individual variables are straight-forward and simple. However, performing them on a set of variables in an appropriate sequence helps in forming procedures by clustering elementary operations and pushing them in to subconscious level. It is the procedural knowledge that is critical and separates an expert from a novice. This paper suggests ‘cognitive apprenticeship’ (Collins, Brown & Newman, 1989) based learning as a possible solution towards the achievement of adequate skills and knowledge in such domains. It also suggests that the theoretical knowledge about cognitive skill acquisition can inform the design of successful computer based tutoring systems. The paper is based on the experience of the design, development and implementation of Intelligent Tutoring Tools (ITTs) developed by the Byzantium project, a consortium of six Universities funded under the Teaching & Learning Technology Programme (TLTP) of the Higher Education Funding Councils of the United Kingdom. The ITTs have been gradually implemented at a number of institutions as the beta testing versions became available and till date more than five thousand students have used one or more ITTs in the cost engineering/accounting domains. For detailed description of the structure and functionality of the ITTs, please see Patel & Kinshuk (1997a). 2. The learners and the learning The learners in the traditional educational setting predominantly constitute of students preparing for a career. Though this sector has diversified into adult education and distance education, it is the predominance of classroom based students that overarches the design and operation of the teaching and learning systems. The courses are designed to provide a breadth and depth of knowledge, the relevance of which may not be fully understood by the students. The students are also aware that the possible application of any knowledge gained is at a distant horizon, leading to a perception of low relevance and attendant low motivation. Their focus frequently shifts to the skills that are likely to yield higher grades as an immediate objective. Cognitive skills related to ‘examination techniques’ acquire importance though there are not many situations in life that resemble the typically contrived nature of an academic assessment system. Frequently, the abundance of the assessment tasks set for the different subjects overwork skills such as report-writing and presentation at the expense of adequate time for reflection on what is learnt (Patel, Russell & Kinshuk, 1999). The learning, in many cases, is reduced to assignment hopping with ‘just-in-time’ and ‘just-enough’ learning to fulfil the assessment tasks, thus defeating the objective of providing a well-balanced learning experience. The life long learners, on the hand, constitutes of those who continue their overt learning activities beyond the student life and many of them are likely to be much more sophisticated due to prior education and life experiences. These obviously include those practitioners who have to learn continuously due to rapid changes in the technology employed for their work. The different categories of life long learners also include the learner groups who traditionally pursued adult learning or distance learning avenues to advance or complement their current knowledge or to gain new knowledge in pursuit of career advancement or recreation activities. The motivation, in case of all categories of life long learners is likely to be much higher than the traditional students, since they have perceived the relevance of such learning and decided to acquire it. While research in distance education has revealed a non-completion or ‘dropout’ rate in the range of 30 to 50%, it has also revealed that the best predictors of success in distance education are a greater length of exposure to formal education and less length of time since completing a formal education course (Moore & Kearsley, 1996). Perhaps these findings are indicative of the shortcomings of the distance learning curriculum due to the hangover of the traditional model rather than the learners. For instance, the successful use of the life long learning resources such as FAQs, Technical Databases, Bulletin Boards or News Groups etc. by the software programmers demonstrates that where the learner can clearly see the relevance, there is no lack of motivation to use even ill-designed and cumbersome sources of information. None of these sources are likely to be designed on a pedagogically sound basis and the learner learns by studying and self-explaining the pieces of program codes that work and those that fail to work. The advantage to the learner, in this case, is that the code can be tried out, modified and again tried out to test whether it works, and this provides the feedback, which is very helpful in the process of self-explanation A life long learning resource for other domains, however, is much more beneficial to the learner if it aids in self-explanation through feedback, suggestions and guidance. Before moving on to discussing a practical implementation of such design, it would be useful to briefly consider the constructivist notion of the situatedness of knowledge. 3. A brief consideration of the situatedness of domain knowledge and skills Considering the predominantly practical dimension of the task oriented professions, it is important to consider whether learning in an academic institution and learning in a real work environment are competitive or complementary approaches for facilitating acquisition of domain competence. Real work environments provide a strongly situated and therefore highly relevant learning of the tasks that are of immediate practical use at the workplace. However, if all learning can occur only in the context of use, learners would need to prematurely commit themselves to a career in a very narrow sense and train in the specific workplace. It also fails to satisfactorily address the increasing emphasis on multiple skilled employees and changes in occupation structure brought about by technological development. The argument is also relevant for the design of Web based Life Long Learning systems. If the purely situated learning argument is followed zealously, the learner will have to be immersed in the virtual ‘real work’ environment. While this approach can be beneficial in developing critical skills (e.g. flight control simulation) or tolerance to different physical conditions (e.g. space travel simulation), it is very costly to develop and implement such learning environments. While recognising that learning is progressive and a life-long process and that some aspects of learning are strongly situated, there does not appear any need to take an extreme view that all learning cannot be decomposed and decontextualised. For instance, Baumgartner & Payr (1996) divided the learning process into various layers (such as learning of facts and context free rules, context sensitive rules, problem solving, and so on) that are in principle but not necessarily sequential. There are overlaps among these layers; nevertheless they represent different stages of the learning process. Each layer of learning process may demand different technological approaches and media to support effective and efficient learning, ranging from traditional books to more sophisticated techniques such as simulations, games and microworlds. Some of these techniques are more suited to contextualised components of the learning process whereas others are better facilitators of decontextualised components. Anderson et. al. (1996) observed, "While some context will often be required … there are always bounds on how complex such a context need be. It is a well-documented fact of human cognition that large tasks decompose into nearly independent subtasks, so that only the context of the appropriate subtask is needed to study its components." They claimed that it is better to train independent parts of a task separately because fewer cognitive resources will then be required for performance, thereby reserving adequate capacity for learning. An intelligent tutoring system can be a useful tool to provide efficient learning by representing the domain content at suitable granularity and providing interactive guidance to the learner at the level of independent parts of the domain tasks. The necessity of some context also raises an important issue about the selection of such context. In the traditional classroom based educational system (excepting the adult learners) the learners are less likely to have any significant work experience. The context provided for their learning is determined by the authors of learning materials and teachers who provide examples to describe a concept. The learning process involves the appreciation of the context as a first step. In the case of life long learning, the learner may already have a good understanding of a particular work environment and will benefit significantly if the context is based on this work environment. For example, if a Chemical Engineer and a Sports Facility Manager are learning Business skills, it is apparent that they would find it easier to understand a concept if it is presented in the first instance, in the context of a Chemical Plant and Leisure Business respectively. Thus, the life long learning resource benefits from a consideration of multiple representations of the contextual information. Keeping the foregoing considerations in the background, it will be useful to consider how our understanding about the acquisition of cognitive skills can help in designing tutoring systems for numeric domains. 4. Phases of skill acquisition and the cognitive apprenticeship framework Kurt VanLehn (1996) has discussed a framework for reviewing cognitive skill acquisition. VanLehn suggested that Fitts’ categorisation of the three phases of motor skill acquisition (early, intermediate and late) also aptly described the course of cognitive skill acquisition. In the early phase, dominated by reading, discussing and other general information acquiring activities, the learner is trying to understand the domain concepts without yet trying to apply the acquired knowledge. The primary focus is on studying expository instructional material. The intermediate phase begins when the learner turns the attention to solving problems. In most cases such attempt is made after studying a few problems that have already been solved. Though the learner may refer back to the expository material, the primary focus at this stage is on solving problems. At the intermediate phase, the learner removes misconceptions and acquires missing conceptions as related to the domain knowledge as well as acquiring heuristic, experiential knowledge to expedite problem solving. At the end of this phase, the learner can solve problems without conceptual errors though they may still commit unintended errors or slips (Norman, 1988). The slips are generally indicative of the lack of attention arising from the increasing confidence of the learner. Such slips may remain uncorrected as the learner may not have adequately developed a sense of judgement about the overall solution and does not ‘feel’ that something isn’t right! During the late phase, the learners improve in accuracy and speed through practice. Their understanding of the domain and their basic approach to solving the problems do not change significantly at this stage. The three-phase distinction, though being an idealisation as the boundaries between the phases are not sharp and clear, is nevertheless useful as it indicates that learning resources for cognitive skill acquisition in any domain, needs to be categorised into three types. The first category is the expository material that may contain hyperlinks to facilitate movement between hierarchically, semantically or laterally connected notions. The second category consists of formative assessment material with immediate and dynamic feedback at each step. The immediate feedback ensures that the learner’s error is corrected as soon as it occurs and there is no danger of rehearsing an incorrect conception. The dynamic nature of feedback ensures that the immediate feedback is provided for each sub-goal. The learner is thus assured of better interactivity with the learning resource and is freed from the cognitive load of being concerned about a chain of sub-goals at this stage of learning. The final category is based on summative assessment to enable free application of knowledge gained by the learner, allowing any slips and errors. A delayed feedback after completion of one or more summative problems can then be provided for the learner to inspect. A practical application of this categorisation can be found in the Byzantium ITTs. The first screen in all the ITTs offers a menu shown in Figure 1. The learner selects ‘Basic Concepts’ for expository material, ‘Interactive Mode’ for formative assessment and ‘Assignment Mode’ for summative assessment. The fourth option, ‘View Marked Work’ is selected for inspection of marked assignments and provides feedback on the attempted solution for each assignment problem. Figure 1. First screen in ITTs While the foregoing discussion is based on the phases of cognitive skill acquisition as idealised distinct categories, the continuum between the early and intermediate phase is perhaps better captured in the functionality oriented cognitive apprenticeship framework suggested by Collin, Brown and Newman (1989). The framework requires that the following functionality should be present in a tutoring system: The learners can study task solving patterns of experts to develop their own cognitive model of the domain (modelling). The ITTs provide a Basic Concepts mode presenting textual/graphical explanations and solved examples. The same material is also available through the Help button in the interactive learning mode. The learners can solve tasks on their own by consulting a tutorial component (coaching). The ITTs offer qualitatively better coaching through interactive guidance and dynamic feedback while a learner is attempting to solve a problem. The tutoring activity of the system is gradually reduced with the learner’s improving performances and problem solving (fading). The ITTs provide help ‘by exception’ and the tutoring activity is triggered by an illegal or incorrect attempt. With the improvement in performance there is less tutoring intervention. The next section briefly discusses the cognitive apprenticeship-based learning environment provided by the Byzantium ITTs. 5. An implementation of cognitive apprenticeship-based learning environment As discussed in section 3, it is possible to decompose a large task and only the context of the sub task is required for learning. Therefore, with well-designed learning resources employing granular interface, it is possible to learn from much simpler interactions. The learning tasks are decomposed into smaller components at varying levels of granularity with the perspective shift enabled through the user interface (Patel & Kinshuk, 1997b). There is no need for the system to engage in complex inferencing about user knowledge as the system can provide a simple correct/incorrect feedback at a coarser grain size. Where necessary, the system can advise the learner to use a fine-grained interface for more detailed interaction, as shown under ‘Interactive Messages’ in Figure 2. The figure shows a problem space in the Capital Investment Appraisal ITT. The proposed investment into a project can be evaluated using one or more of the four techniques, each of which can be selected by one of the pushbuttons on top of the panel on the left-hand side. The learner is given some information, shown in blue, which is sufficient to solve the problem. The derived values entered by the learner are shown in black. The learner is using the third technique of evaluation, namely NPV or the Net Present Value and is attempting the discount factor for end of Year 1. The value of the discount factor attempted by the student was incorrect so the interactive message advises the use of the Formula feature. The feature employs a fine-grained interface, shown in Figure 3, called up using the Formula pushbutton on the left-hand side panel. Figure 2. Example of interactive message Figure 3. Fine grained interface The ITT has ‘Just in time’ scaffolding and ‘Built in’ fading as demonstrated by the following attributes: The system does not force a student to use a rigid sequence of data entry or a specific path to solution when multiple paths are possible. Though the expert solution records its own path to the solution, the system recognises all the alternative valid paths to solution and alters its guidance in line with the path chosen by the learner. The system offers scaffolding only when the student demonstrates a need for it through an erroneous action. If the attempted value cannot be obtained directly from the given information and that already derived by the learner, the system suggests that the learner first attempts an intermediate step. This feedback helps in clearing any misconceptions and in acquiring any missing conceptions. If the learner has performed some mental operations and attempted a correct value, the system does not insist that the intermediate step must be carried out first. If the attempted value is incorrect at the first attempt, the system merely notifies that it was incorrect and allows re-examination. The frequency of this, the first level of feedback, has been observed to diminish to almost zero with the increase in learning and then rise again due to the slips arising from overconfidence and increased cognitive load as the student attempts to tackle a larger chunk of problem by performing several mental operations. On second incorrect attempt, the system advises the correct relationship to use. This, the second level of feedback, is the workhorse of intermediate phase of cognitive skill acquisition as it indicates misconceptions and helps in getting rid of them with the immediate feedback. On third incorrect attempt, the system shows the calculation using the actual data. This level of feedback has been observed only at the initial stage of the intermediate phase as the learner is still grasping the various domain concepts by placing them in relation to each other. In the Interactive mode, the learner has to initiate some action before the system offers any guidance. The feedback’s purpose is to spur the learners’ own self-explanations by pointing out the correct action. At any stage of interactive learning, the learners can refer back to the expository material by using the Help pushbutton and do not have to return to the menu and select the Basic Concepts option. This allows the learners to navigate back and forth between the work they are doing and the textual explanations and solved examples provided in the expository material. The system requires greater engagement by the learners while giving them a greater control over the learning actions. It harnesses the natural learning capabilities of an intelligent being by giving enough feedback to prevent an impasse. A positive motivational impact from this approach has been observed among learners and many students have commented that they find it motivating and positively challenging rather than the ‘patronising’ approach they saw in some tutoring systems. Since the learners may experience different levels of difficulty with different topics based on their background and prior exposure to some of the notions, the system offers an almost infinite bank of questions for practice through its capability to randomly generate problem data. It does not hold problems and solutions but holds the knowledge of the inter-relationships of the variables. It can, therefore, randomly select some variables as independent variables and assign random values within the programmer specified bounds. On generating the problem data, the system applies its knowledge to derive an expert solution and if all the remaining variables are found to have a legal value in the expert solution, the problem is presented to the learner. Thus there is ample scope for practice at the intermediate phase of cognitive skill acquisition. Towards the end of the Intermediate phase, a teacher may wish to introduce problems in a narrative form for the learner to experience more authentic situation and interpret the data provided in a raw form. The system provides an ‘Enter Your Own’ problem data option. However, some variables have a fixed value in the interactive mode to prevent misuse of this option for solving the assignment problems with interactive guidance. The problem narration, therefore, need to be designed around these fixed values. It still offers ample scope of variation and return for this handicap, the system offers a rich scope of summative assessment that can be computer marked. The combination of the ITT and the Byzantium Marker software is capable of identifying and giving partial score for ‘incorrect interpretation but application of correct method’, just like a human tutor marking the work. For more details on the Byzantium intelligent assessment system, please see Patel, Kinshuk & Russell (1998). The computerised marking enables a very fast turnaround and more frequent summative testing can be employed without burdening the tutor. The advantages of such frequent summative testing are three folds. Firstly, it motivates the students in the traditional educational setting to be more attentive to the interactive learning as they are aware that they have to take a test on what they have learnt. Secondly, such increased attention shortens the intermediate phase of skill acquisition and finally there is a greater amount of and more frequent feedback to support the late phase of skill acquisition. The objectives of the Byzantium learning environment reflect Collins’ (1990) recommendations for constructing robust domain competence. Accordingly, the Byzantium environment aims to facilitate the learners in: acquiring the basic domain knowledge which can be used subsequently as a base to integrate all the bits and pieces of knowledge gained from specific situations; applying the basic domain knowledge in abstract and contextual scenarios to generalise the knowledge and skills to be able to apply them in real world situations. The learning path consists of transition through observation, interactive learning, simple testing, learning and testing involving multiple contexts and/or interpretation of text narrative. The ITT architecture provides customisation facility for creating appropriate templates as shown in the example from the absorption costing ITT given in Figure 4. Figure 4. Example of a template A teacher or the designer of Life Long Learning material can specify various parameters to create a replica of simple real world scenarios. The templates can then be used for the purpose of both the structured and non-structured problems. This facility can be used both for providing a situated learning through a particular context (say, for those learning costing as a part of workshop management course or those who have a workshop background and desire to acquire business skills) and for helping in generalising the knowledge through multiple contexts. 6. Feedback from independent evaluation of the ITTs Stoner & Harvey (1999) carried out an independent evaluation at the University of Glasgow, involving Byzantium and another widely used traditional Computer Based Learning package. They found that the results indicated that students’ performance had improved significantly over the period since Learning Technology materials were introduced and that this improvement appeared to be mainly reflected in the students’ ability to complete numeric questions. Their student feedback focus groups observed: Byzantium was useful because you could go over bits you were unsure about. It was better than a book because it was interactive. With the interactive questions you tend to pay more attention than you would to a book. Prefer Byzantium because the other package waffles on about what you already know and provides no incentives to pay attention to what it says. Byzantium offers instant feedback, is more involving and you can do as many questions as you like. Of the two tutoring systems, 71% students showed a preference for Byzantium material while 8% indicated no particular preference. The students wanted more tutoring systems, similar to Byzantium material, for other topics and were positive about Computer Aided Learning (CAL) in general, observing that it was good to use CAL if the tutoring software was good. 7. Extending the ITT methodology to the Internet The success of the stand-alone ITTs has opened the way to create Intelligent Tutoring Applications (ITAs) over the Web, using the same underlying methodology but written in Java or other such platform independent language suitable for the Web. With the help of suitable authoring tools and a modular software structure, the Web permits the tutoring modules to be created, held and accessed in a structured manner across vast distances. With the appropriate structuring parameters, the ITAs created by different teachers build up to a large inventory of accessible knowledge that can be utilised by all the teachers in various configurations of single or multiple ITAs to create a Hyper-ITS. The development experiences of ITTs have confirmed that the object oriented and data driven approaches to module development are required to provide the necessary ease and flexibility that is required by a typical teacher. For example, hard coded knowledge base makes the teacher dependent on a software engineer. If the system can acquire the knowledge base from a teacher through an interactive dialogue, then it would be possible for any teacher to produce the ITAs for any numeric discipline relatively easily. For more details on the proposed extension of the ITT methodology to the Web, please see Patel, Russell & Kinshuk(1998). 8. Conclusion The ITTs, even in their current state of implementation, eliminate a deficiency of distance education by providing an interactive learning environment for the numeric disciplines. A very high proportion of learners tend to have problems with numeric disciplines and can greatly benefit from the cognitive apprenticeship approach of the Byzantium ITT. Based on these small knowledge based tutoring tools, this paper briefly presents the exciting potential of designing Web-based ITA and Hyper-ITS that support sharing of development efforts as well as the learning resources so developed. The proposed architecture benefits from the bottom-up approach of system development to provide an incremental path to building up a large inventory of knowledge entities on the Web. These entities can be linked to construct large tutoring systems without too much effort on the part of the individual contributors. References Anderson, J. R., Reder, L. M. & Simon, H. A. (1996). Applications and Misconceptions of Cognitive Psychology to Mathematics Education. Internal paper, Carnegie Mellon University, Pittsburgh, USA. Baumgartner, P. & Payr, S. (1996). Learning as action: A social science approach to the evaluation of interactive media. In Carlson P. & Makedon F. (Eds.) Proceedings of Ed-Media 96 - World Conference on Educational Multimedia and Hypermedia, June 17-22, Boston, Mass., USA. Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P. & Glaser, R. (1989). Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems. Cognitive Science, 13, 145-182. Collins, A. (1990). Generalising from situated knowledge to robust understanding. American Educational Research Association Annual Meeting, April, Boston. Collins, A., Brown, J. S. & Newman, S. E. (1989). Cognitive Apprenticeship : Teaching the crafts of reading, writing and mathematics. In Resnick L. B. (Ed.) Knowing, Learning and Instruction, Hillsdale, NJ: Lawrence Erlbaum Associates, 453-494. Moore, M. G. & Kearsley, G. (1996). The Distance Education Student. Distance Education – A Systems View, Belmont, CA: Wadsworth Publishing, 153-171. Norman, D. A. (1988). To Err is Human. The Psychology of Everyday Things, USA: Basic Books, Perseus Books Group. Patel, A. (1997). Learning by solving examples through data driven Internet Based Intelligent Tutoring Systems. TLTP Newsletter, 10, http://www./tltp Patel, A. & Kinshuk (1997a). Intelligent Tutoring Tools in a Computer Integrated Learning Environment for introductory numeric domains. Innovations in Education and Training International Journal, 34 (3), 200-207. Patel, A. & Kinshuk (1997b). Granular Interface Design : Decomposing Learning Tasks and Enhancing Tutoring Interaction. In M. J. Smith, G. Salvendy & R. J. Koubek (Eds.) Advances in Human Factors/Ergonomics - 21B - Design of Computing Systems: Social and Ergonomic Considerations, Amsterdam: Elsevier Science B. V., 161-164. Patel, A., Kinshuk & Russell, D. (1998). A computer-based intelligent assessment system for numeric disciplines. Information Services & Use, 18 (1-2), 53-63. Patel, A., Russell, D. & Kinshuk (1999). Assessment in a Cognitive Apprenticeship-based Learning Environment – Potential and Pitfalls. Brown S., Race P. & Bull J. (Eds.) Computer Aided Assessment in Higher Education, London: Kogan Page, 139-147. Patel, A., Russell, D. & Kinshuk (1998). Collaborative Teaching in Distance Education – An Innovation in Intelligent Tutoring. Exploiting Learning Technology: Issues for Learners and Educators, IEE Informatics, 98/453, Savoy Place, London: The Institution of Electrical Engineers, 21-29. Stoner, G. & Harvey, J. (1999). Integrating learning technology in a foundation level management accounting course: an e(in)volving evaluation. CTI-AFM Annual Conference, April, Brighton, U.K. VanLehn, K. (1996). Cognitive Skill Acquisition. In Spence J., Darly J. & Foss D. J. (Eds.) Annual Review of Psychology, 42, Palo Alto, CA: Annual Reviews, 513-539.

---