Interoperable Performance Assessment

Interoperable Performance Assessment within Intelligent Tutoring Systems Tiffany Poeppelman1, Mike Hruska2, Rodney Long3 and Chuck Amburn3 Aptima, Inc.1; Problem Solutions2, U.S. Army Research Laboratory3 tpoeppelman@aptima.com mike@problemsolutions.net rodney.long@us.army.mil charles.amburn@us.army.mil Abstract. The domains in which Intelligent Tutoring Systems (ITS) are being used in single learner environments are well-defined and have been shown to be effective. However, current data tracking and interoperable data sharing across those environments and domains are still in their infancy. Growing specifica- tions like Experience API (xAPI) from the Advanced Distributed Learning (ADL) Initiative are gaining traction due to their ability to capture data about experiences across a variety of environments. However, determining the right level of data is still a challenge. This paper explores the current research and use of xAPI within the Generalized Intelligent Framework for Tutoring (GIFT) for macro-adaptation methods. To address current gaps in research, recent ef- forts have used GIFT as a test bed for analyzing the effect of data descriptions at various levels for tutoring technologies. Keywords: Adaptive Training, Interoperable, Performance, Experience API, Intelligent Tutoring Systems 1 Introduction Built with the intention of providing highly individualized learner feedback, Intel- ligent Tutoring Systems (ITS) are still being researched and developed to ensure usa- bility and reliability of adapting instruction at the right levels. Effective human tutors ask questions, tailor feedback, provide opportunities for reflection, and change the content, direction, pace, and challenge level of instruction to optimize learning [1]. Adaptive ITSs also attempt to select optimal instructional strategies to meet the spe- cific learning needs of individuals or teams [2]. Yet several issues still remain with current authoring practices. They are expensive, require expertise, lack extensibility and reuse, and remain hard to integrate into daily practices. As a result, the ITS com- munity continues to explore trends in domain-independent authoring environments to facilitate learning that typically occurs between teachers and students, as well as cost- effective approaches to integration. adfa, p. 1, 2011. © Springer-Verlag Berlin Heidelberg 2011 Additionally, interoperable assessment capabilities are continuously being exam- ined to provide the appropriate internal and external learner data to each individual tutoring system at the correct time or optimal teachable moment. Current efforts are guided by the concept of the Interoperable Performance Assessment (IPA) which is a method of uniformly defining and describing experience and context data in order to assess learning and performance over time and adapt training across a variety of envi- ronments, systems, or modalities; where performance is observed, assessed, evaluat- ed, or asserted by systems or observers [3]. Within various training environments and other tutoring systems, data is currently collected in a black-box fashion leaving data about the learner unavailable for macro-adaptation by other systems. By enabling disparate applications to act on relevant information about the learner from other sys- tems, tutors will be able to initiate the correct training content or data that corresponds to the learner needs. Currently a specification, the Experience API, or xAPI, focuses on the capture of interoperable assessment data from learner experiences. Emerging technology around this specification known as the Learning Record Store (LRS) ena- bles the storage and retrieval of xAPI data (known as statements) for intelligent tutor- ing systems to access information surrounding current and previous training perfor- mance. The Army Research Laboratory’s (ARL) is investigating tools, methods and standards for ITSs so that they are used within military training environments. Their efforts have resulted in the design of GIFT, the Generalized Intelligent Framework for Tutoring, which has a modular, service-oriented architecture. GIFT was developed to address authoring, instructional strategies and analysis constraints. Given the im- portance of evolving key ITS features and specifications like xAPI and how they ena- ble additional sources of interoperable data, recent efforts by ARL have resulted in the expansion of GIFT to receive and store xAPI statements for interoperability of data from outside sources. In this paper, we will describe xAPI and the current re- search and development efforts underway to expand GIFT to collect and store xAPI datasets, and how this enhances GIFT’s ability to adapt instruction based on a learn- er’s history. 2 Experience API As we are currently immersed in a technology age, a realization is emerging that learning is continuously happening beyond formal training environments. Learning is also happening through experiential and informal methods. Technologies continue to drive the training community toward new training approaches and the development of systems that create, view, and present content to learners in new ways. A need for common specifications to support these environments has been growing. In 1999, the Advanced Distributed Learning (ADL) Initiative, currently of the Of- fice of the Under Secretary of Defense Personnel and Readiness (OUSD P&R), set out to modernize learning and training in the Department of Defense (DoD). One of ADL’s initial goals was to “establish guidelines on the use of standards and provide a mechanism to assist DoD and other Federal agencies in large-scale development, implementation, and assessment of interoperable and reusable learning systems” [4]. As a result of the creation of ADL, the Sharable Content Object Reference Model (SCORM) emerged [5]. The xAPI is an evolution beyond SCORM that solves many problems that were classically associated with the SCORM guidelines. Unlike SCORM, the xAPI allows tracking outside of an LMS to capture data about digital and non-digital learning ex- periences. The ADL’s Training and Learning Architecture (TLA) capability “encom- passes a set of standardized Web service specifications and Open Source Software (OSS) is designed to create a rich environment for connected training and learning” [6]. Under the TLA capability, the xAPI work is focused on extending the future sup- port of interoperability of learning systems. The xAPI defines a method to capture data about the interaction between a learner and a learning experience. The xAPI is an emerging specification that is evolving the learning technology marketplace and ca- pabilities within organizations. The xAPI specification, 1.0.0 as of April 26, 2013, defines an API to track data about learning experiences [7]. An API defines a protocol intended to be used as an interface by software components to communicate with each other. The xAPI is a specification that describes a common data structure and the storage/retrieval rules that developers can implement to create a learning experience tracking service. The service works by allowing statements of experience (typically learning experiences, but could be any experience) to be delivered to and securely stored in a Learning Record Store (LRS). The format of these statements is based on Activity Streams (<Actor, Verb, Object> or “I did this.”) [8]. In this format, the Actor is the agent the statement is about, like a learner, mentor, teacher, or group. The verb describes the action of the statement, such as read, passed or taught. Finally, the object is what the Actor interacted with, like a book, a test, a video, a class or even a mentor. These are some of the simplest examples, but xAPI also allows complex statements correspond- ing to anything conceivable in natural human language. Figure 1 shows an example of an xAPI statement. }, "id": "http://www.SP2.com/assessed" }, "object": { "objectType": "Agent", "name": "John Bates", { "id": "5d23f4c4-9368-46ad-8390-5a086f45a829", "actor": { "objectType": "Agent", "name": "Smith, Greg", "mbox": "mailto:admin@sp2.com" }, "verb": { "display": { "en-US": "assessed" "mbox": mailto:john.bates@us.army.mil Figure 1. xAPI statement example These statements can be used by themselves to track learning records which will provide the ability to enable richer learning environments. The xAPI specification was also written to be flexible enough to meet the varying use cases of the e-learning community. This flexibility may lead to a degradation of interoperability if communi- ties of practice do not agree to certain conventions. These conventions and rules on how to use the xAPI can be collected into companion specifications and applied to the base xAPI specification as profiles. ADL is currently developing a profile for the SCORM community that addresses topics such as launch of traditional web content, statement verbs and results for reporting success and completion. The purpose of xAPI is to store and provide access to learning experiences including traditional records, such as scores or completion status, as well as assertions of profi- ciency or deficiency for concepts, competencies, knowledge or skills. This intersys- tem communication between Learning Management Systems (LMSs), ITSs, and other systems is allowing domain-independent ITSs and training technology-based solu- tions to leverage a new level of breadth and depth on how users have performed on topics, events and activities across a variety of learning platforms. By understanding the current state and granular historical data of a learner, these systems will ultimately be able to adapt learner pathways at the macro and micro level. Figure 2 shows the data and functional ecosystem of the ITS, LRS and xAPI. Figure 2. ITS, LRS and xAPI Ecosystem  3 Interoperable Performance Assessment (IPA) Currently, there are a number of efforts underway in the DoD related to tracking and assessing performance, an important step to enable adaptive learner-centric envi- ronments. The Army Research Laboratory’s (ARL) Simulation and Technology Training Center (STTC) funded a recent effort to define best practices for defining and encoding good performance measurement using xAPI statements. Additionally, this funded effort supported the development of a technical architecture for interoper- able activity across current Army architectures and a proof of concept called the Sol- dier Performance Planner (SP2). The SP2 allows various systems to connect and share data, to add new individual data, and to track individual and group performance data that will ultimately support adaptive and tailored learning across ITSs and other sys- tems. In its exploration of the xAPI to support IPA, ARL is evaluating the assessment of individuals across multiple systems to establish best practices for the proper extrac- tion of meaningful performance data from training events, as well as visualization of the data [2]. A recent effort, Interoperable Performance Tracking to Support Tailored Learning, has defined approaches to collect data about both learners and content to continually evolve the complexity of potential adaptations at more and more granular levels. The focus to date has been on macro adaptive approaches within GIFT afford- ed by the use xAPI data captured within other systems. The effort focuses on captur- ing xAPI statements, from one or more systems, that define whether a person is at, above, or below a given standard on a given concept. These statements are stored in a central LRS between the systems. The statements are then accessed by a polling com- ponent built with GIFT. The example in Figure 3 is sample xAPI statement in JSON that describes data that was created by an observer using a tablet tool to assess per- formance of an individual using a simulator. The statement generally says that “Greg Smith assessed John Bates in a particular context and he was an expert.” This xAPI statement would be accessed by an ITS (GIFT in this case) from the LRS to provide context for adaptation. { "id": "5d23f4c4-9368-46ad-8390-5a086f45a829", "actor": { "objectType": "Agent", "name": "Smith, Greg", "mbox": "mailto:admin@sp2.com" }, "verb": { "display": { "en-US": "assessed" }, "id": "http://www.SP2Test.com/assessed" }, "object": { "objectType": "Agent", "name": "John Bates", "mbox": "mailto:john.bates@us.army.mil" }, "result": { "extensions": {}, "score": null, "success": false, "response": "Expert" }, "context": { "extensions": {}, "contextActivities": { "parent": [], "grouping": [ { "id": "http://www.SP2test.com/Air_to_Ground_Integration", "definition": { "name": { "en-us": "Air to Ground Integration" }, "description": { "en-us": "Target is destroyed" } } }, { "id": "http://www.SP2test.com/101st_CAB_AATX", "definition": { "name": { "en-us": "101st CAB AATX" }, "description": { "en-us": "7/18/2013 11:20:15 PM" } } } ] }, "platform": null }, "timestamp": "12/10/2013 16:46:11", "authority": { "objectType": "Agent", "name": null, "mbox": null } } The polling component allows GIFT to macro-adapt course recommendations based on a limited number of simple rules: 1) For courses which have an xAPI statement that the learner is at or above com- petency, the user will be informed that the course is no longer recommended. 2) For concepts with an xAPI statement indicating that a user is performing be- low, courses which contain that concept will be presented to the user as rec- ommended. Figure 3. GIFT xAPI statement example. The approach is intended to demonstrate intersystem data value through macro- adaptation of a learning path. If adopted by the community, xAPI can expand infor- mation beyond performance and can incorporate elements of preferences and interac- tion characteristics within ITS environments. Some examples include the capture of common results, behaviors, or preferences that a user exhibits such as higher levels of engagement when involved in informal learning. Based on the results, the US Army Research Laboratory (ARL) is exploring efforts to inform the domain-independent ITS research and development that is being conducted within the community. Efforts to date have shown how the IPA concept can be used to manage/adapt instruction on the individual level. Additionally, the set of standards/heuristics for individualized instruction and how they can be implemented across different domains and systems is underway as a community driven approach. 4 GIFT and ITS Macro-Adaptive Approaches Recently efforts at the ARL have shown significant success in developing and pro- totyping a generalizable architecture for tutoring with an open-source, community development approach. GIFT is currently being utilized across a variety of efforts which are demonstrating practical use cases for military trainers and trainees, academ- ic partners, and commercial sector clients. GIFT currently has over 300 users in 19 different countries. While GIFT and other pedagogical models use collected data on ‘What’ is being instructed and the available ‘Content’ from which to instruct, ‘Who’ is being instruct- ed is often missing from the picture [9]. If ITSs knew information about the learner and the knowledge or skills they possess, current data could inform both micro and macro approaches to training adaptation. IPA can provide a method for future ITSs to make upfront macro-adaptations or inform configurations of micro-adaptations which would be triggered by the learner model, enhancing the learning experience. Current- ly, data that would inform macro approaches is also critical for competency manage- ment or monitoring. In the current IPA efforts, use cases have provided the capability to identify recommended macro strategies for each learner based on their profile, competency data and training outcome assessments. Additional elements are currently being identified for external data sources, such as LMSs, LRSs, or external training events, to pass meaningful data to GIFT and other ITSs to adapt at macro levels. Based on current research, some of the meaningful data or sources of adaptation could be individual differences and task characteristics [9]. Based on research by Goldberg et al. (2012), these may include student performance levels, working memory capacity, knowledge type, task difficulty and complexity, learner’s prior knowledge/expertise, learner’s traits or attributes, learning styles, interactive multi- media instruction (IMI) levels, or tasking learning categories. Ideally IPA influences of GIFT interactions could ultimately adapt the instructional tactics or target of adap- tion to focus on the sequence of instruction, presentation of information, degree of learner control, feedback, problem difficulty or pace of instruction. Considerations for each of these sources and targets are under review and should be examined for future research related to xAPI statements and how that data would be meaningfully passed from ITSs and other systems. IPA can be an important source of adaptation and what the targets would be in the future. With the market adopting common specifications like xAPI, a need exists to shape data capture requirements as a community to allow all potential data-producing sys- tems to be leveraged appropriately. Future adaptive learning ecosystems will be col- lecting data about our learners as they consume content and intersect with experiences across a wide range of delivery modalities to better connect them to the right experi- ence next. Most importantly, this data collection will drive new opportunities for con- necting the learners and experiences together in meaningful ways. Understanding, connecting, and provisioning learners, content, and experiences based upon their past experiences and results is the key to providing learning where it affects performance in positive ways. 5 Future Research While current ITSs are focused on reducing high training development costs, im- proved standards, and adaptability to support tailored needs of the learner, there is still much research and development left to be done to attain this goal. As new domains continue to be explored, interoperable performance assessment best practices docu- mentation and methods will continue to grow and expand based on domain-specific verbs, actors, actor taxonomies, activities, and more domain specific context descrip- tions. Future IPA efforts should continue to explore tools and methods to convert these strategies into specific instructional tactics for implementation. For instance, statements from other systems about competencies, knowledge, and skills could be utilized to select teams, identify group scenarios, and determine individual mission injects that would leverage the strengths of the teams to mitigate team weaknesses. As other systems and ITSs start to build support for the xAPI specification and lever- age intersystem data from the LRS, we as a community can begin to power future ITS with the capability to adapt based on rich learner profiles and aggregations of experi- ence. Additionally, providing content specific to a learner’s needs not only reduces the time spent on unnecessary content, [11] but helps them focus on the areas that need attention or remediation. By conceptualizing instructional management in a do- main-independent ITS, future development efforts should concentrate on developing support for data creation and collection across the entire learning ecosystem. 6 Conclusions While research gaps still exist on retention and deep learning, there is a communi- ty of practice that is shaping the future of personalized and tailored learning. Tech- nology developments continue to grow around those systems that are connecting methods and approaches for tracking the learners experience from both macro and micro levels. With increasing pressure to achieve greater efficiency and effectiveness in training investments, there has been greater interest in adaptive training solutions and systems among researchers and practitioners. Adaptive training represents a method to achieve better results at lower cost. As a result, there exists significant interest in exploring its potential by some of the largest customers for training in the United States and abroad [12]. 7 References 1. L.S. Vygotsky. Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press, 1978. 2. R.A. Sottilare. Considerations in the Development of an Ontology for a Generalized Intel- ligent Framework for Tutoring. In Proceedings of the International Defense and Homeland Security Simulation Workshop 2012, Vienna, Austria, 2012. 3. T.R. Poeppelman, M. Hruska, J. Ayers, R. Long, C. Amburn and M. Bink. Interoperable Performance Assessment using the Experience API. In Proceedings of the Interserv- ice/Industry Training, Simulation and Education Conference, Orlando, Florida, December, 2013. 4. Training Industry, http://www.trainingindustry.com/taxonomy/a/advanced-distributed- learning-(adl)-initiative.aspx 5. AdvancedDistributedLearning,http://www.adlnet.gov/overview 6. AdvancedDistributedLearning,http://www.adlnet.gov/capabilities/tla 7. AdvancedDistributedLearning,https://github.com/adlnet/xAPI-Spec 8. ActivityStreams,http://activitystrea.ms/ 9. R.A. Sottilare, K.W. Brawner, B.S. Goldberg and H.K. Holden. The Generalized Intelli- gent Framework for Tutoring (GIFT). Orlando, FL: U.S. Army Research Laboratory – Human Research & Engineering Directorate (ARL-HRED), 2012. 10. B.S. Goldberg, K.W. Brawner, R.A. Sottilare, R. Tarr, D.R. Billings and N. Malone. Use of Evidence-based Strategies to Enhance the Extensibility of Adaptive Tutoring Technolo- gies. In Proceedings of the Interservice/Industry Training, Simulation and Education Con- ference, Orlando, Florida, December, 2012. 11. P.J. Durlach and R.D. Spain. Framework for instructional technology. In V. G. Duffy (Ed.), Advances in applied human modeling and simulation (pp. 222-231). Boca Raton, FL: CRC Press, 2012. 12. U.S. Army Training and Doctrine Command. The United States Army Learning Concept for 2015. Fort Monroe, VA, 2011.