IFETS - Discussions

Time schedule:
Discussion: April 11-20, 2005
Summing-up: April 21-22, 2005

Moderators and Summarisers:
John Olszewski and Graham Clarke
53Technology, USA

Introduction:

53Technology recently worked with a new medical residency program at Dartmouth Medical School (DMS) to develop a competency-based learning management system. The project sparked some thought provoking and critical questions which we would like to share in this paper. The areas are of central interest to this International Forum of Educational Technology and Society and may even represent an important intersection of a technology enabled transformation of education and society.

Perhaps as you read about our project and its challenges, you will consider, and share your insights about:

How a level of proficiency in a competency domain influences a person’s effectiveness as a ‘self-directed’ learner? i.e., is a specific level of knowledge and skill necessary for a learner to be ‘self-directed’?

If yes, how can we characterize that level?

How do models of competency and models of learning drive the functional requirements for learning application development?

Are there lessons learned and best practices to share?

Are there functionalities that can be designed into a learning management application which will enable the ‘self-directed’ learner to accelerate time to competency?
What can a learning management application do to sustain competency?

Perhaps this project will raise thoughts and issues for you that we have not considered. We look forward to a lively discussion.

Background:

In 2003 the Accreditation Council on Graduate Medical Education (ACGME) called for a shift in residency program accreditation from a focus on structure and process to a focus on competencies and educational outcomes. [1] Between July 2003 and July 2010 all U.S. medical residency programs in all disciplines are required to assess resident capability in six core competencies to maintain accreditation. This represents a significant change in the educational processes used to train and evaluate medical residents.

In response, a group of DMS faculty wanted to support their new residency program [2] with an online competency-based learning management tool and electronic portfolio that would comply with the new ACGME mandate. DMS came to 53Technology for help.

Our Challenge:

Graduate Medical Education takes place in an environment with little tolerance for error. It is fairly easy to get agreement among medical educators about what constitutes competency in hard skills, like surgical knot tying; what constitutes evidence for demonstrating it; and the criteria for evaluating proficiency within the skill. One reason it is easy is because ‘hard’ skills typically fit into the older educational model of a repeatable process.

It is more difficult to get agreement about what constitutes competency in a soft skill such as professionalism [3]; what constitutes evidence for demonstrating it; and what criteria are used to evaluate proficiency within that competency. Skills that are less easily quantified tend to be represented by their ‘outcomes’. They are part of the newly mandated model.

In order to create a new resident teaching program centered on competencies and outcomes rather than structure and repeatable process, the DMS faculty had to go through the painstaking process of rethinking and coming to agreement about what medical residents needed to demonstrate. They then relied on 53Technology to help them determine how this could be demonstrated in a virtual environment.

As consultants, we worked with our clients to fully understand the implications of this new focus for the participants of the new residency program as well as for the broader graduate medical education (GME) community. At the local level this included residents, faculty, clinical staff, non-clinical support staff, program directors, and administrators throughout the institution. Within the broader GME community, each residency program would need to transform their curricula around competencies and educational outcomes.

We gathered the results of this process and organized them into a set of models that provided our customer with the functionality and flexibility they needed to transform resident education. We realized early on that separating the curricular content from the application would give the software the flexibility required to satisfy the diverse needs of the various medical specialties as well as support competency and outcomes based learning in most any professional setting.

We used the Report by the National Postsecondary Education Cooperative (NPEC) on Competency-Based Initiatives in Postsecondary Education [4] as a starting place. NPEC defines a competency as “a combination of skills, abilities, and knowledge needed to perform a specific task.” The report goes on to describe competency-based initiatives as “those purposeful actions undertaken by [institutions] directed at defining, teaching, and assessing competencies across their system.” [5]

It is implicit in competency-based learning that through the learning process a student who is not a ‘self-directed’ learner will become one. Learners may enter this competency-based program dependant on instruction and guidance through a path that is laid out for them. However, to progress successfully through the program, learners are expected to transition into professionals who demonstrate learning self-direction to achieve and demonstrate competency. It is also implicit in NPEC’s definition of competency that competency integrates domain knowledge with critical thinking.

Based in NPEC’s definition of self-directed learning, we reasoned that the learner who grasps the vocabulary of competencies will be more effective in identifying and therefore demonstrating the desired competencies across different educational and non-educational contexts. For learners, it is no longer sufficient to understand a structure and repeat a process; competency is now defined as recognizing what knowledge and skills are called for and anticipating possible outcomes. Competent professionals need to demonstrate that they can think critically as well as thoroughly know their subject. From this understanding, we characterized five steps to differentiate progress toward self-directed learning.

These 5 steps [6] are listed in Table 1.

[Table 1] Steps to Self-Directed Competency-Based Learning

Step 1

Identify and Understand Goals

Step 2

Demonstrate Competency

Step 3

Reflect on Experience

Step 4

Evaluate Progress

Step 5

Bring Learning Forward

Supporting the Steps with Technology:

The result of our development is the REBEL system (Reflective, Evidence-Based Evaluation and Learning). REBEL is a comprehensive learning environment focused on competency-based, reflective learning while encouraging independent learning skills. It features tools related to competency assignment and tracking, learning activity support, threaded discussion, evaluation, journaling, and portfolio building. It was launched July 1, 2003 , is being used by the DMS residency program and is currently being adapted for other residencies and other professional domains.

The following describes the steps to self-directed competency-based learning and how REBEL functionality supports these steps:

Step 1: Identify and Understand Goals

A key to success for the self-directed learner is to

Understand the meaning and purpose of each required competency.

from that understanding, be able to

Identify the potential for demonstrating that competency in any context.

Learners who understand their required competencies and are able to identify the potential for demonstrating those competencies in any context increase their sphere of practice. They are ready to demonstrate the competencies, and thereby move toward becoming more self-directed.

REBEL includes a central learning environment which allows educators to design and deliver highly configurable curriculum to the learner. When a curriculum designer creates a new learning activity such as a course or project, she can specify competencies that must be addressed in that activity (‘applied competency list’). The competencies are displayed prominently to the learner as a visual reminder of the goals of the activity. The curriculum designer may want to ask learners to upload their work and rate themselves on their success in demonstrating the competencies in that activity. This puts the learner in the position of reflecting on the required competencies every time they upload a work product. Assessment tools can be automatically generated and delivered to evaluators who can frame their assessment around the competencies. To further encourage self-directed learning, the curriculum designer can configure the Learning Activity to allow or require learners to create their own ‘applied competency lists’ for a given activity. This creates an environment where the learner must think about the meaning of the required outcomes and embed them into her learning experience. As learners begin to select their own goals, they begin the transition toward becoming a self-directed learner.

Step 2: Demonstrate Competency

By engaging in activities, the learner’s efforts result in work products and applied skills. In the context of a medical residency, work products may include research papers, presentations, reflective writing, and any supporting documents. Applied skills may include appropriate use of medical knowledge and demonstration of clinical expertise. These work products and applied skills are evidence of the learner’s ability to demonstrate desired competencies.

Learners demonstrate their progress towards competencies in REBEL by uploading their work products and tracking applied skills, journaling, and self-reflection and peer/mentor discussion. Learners can also create a ‘learning support team’ for the activity (or an administrator may create one for them). This ‘learning support team’ might consist of the learner’s instructor, mentor, colleagues, peers, administrators, etc. Members of their learning support team can be invited to review their work products, take part in assessments; participate in threaded discussions, and review project proposals. As learners take more responsibility for their learning experience, the learning activity can be configured to allow the learner more control over their environment.

Step 3: Reflect on Experience

Exploring and analyzing one’s own educational experiences in an effort to improve these experiences is an important skill of self-directed learners. Through the process of reflection, learners increase self-awareness and their ability to be constructively critical of themselves. This allows the learner to use her own experiences as evidence of what does and does not work, resulting in the discovery and willingness to try other approaches as well as greater openness for outside feedback and suggestions.

Within REBEL, learners can be asked to write and upload a reflective piece on their learning experience. Workflow documents can be attached to the reflection, allowing a mentor, for example, to review the reflection and make comments. REBEL also features a Journaling Module. Learners are encouraged to keep a journal of their ongoing learning experience and have the ability to share their entries with a system user assigned as a ‘journal coach’. This ‘coach’ can view the learner’s journal entries and comment or suggest strategies to help develop the learner’s reflective skills.

Step 4: Evaluate Progress

Evaluation is a key component to any learning situation. The evaluation process (by instructors, peers and themselves) loops back to the self-directed learner allowing them to analyze their success or failure and re-direct their learning based on areas of continuing need.

REBEL includes a highly-customizable engine for delivering assessment instruments to evaluators inside and outside the system and supports a wide range of evaluation types from within a learning activity including 360° assessments and global surveys, self assessments, multiple choice and open ended exams. Assessment content can also be generated ‘on the fly’ from a learning activity’s applied competency list. REBEL also recognizes the importance of quick feedback to the learner by allowing the learning activity to be configured to deliver an assessment summary report to the learner as soon as minimum returns have been received.

Step 5: Bring Learning Forward

Work products created during the learning process represent a valuable asset for the learner. The ability for a learner to collect and easily access her evidence of learning on an ongoing basis creates a personal record of achievement and should continue to exist as a primary resource for the self-directed learner to use for future review and reflection and for demonstrating accomplishments to others. The ability for the self-directed learner to easily share work with anyone also increases opportunities for feedback beyond the scope of the immediate learning environment.

In REBEL , the learner can ‘bring forward’ her work for future personal review and reflection, to obtain feedback from other sources and as evidence of achievement. REBEL’s User Portfolio allows learners to organize all their uploaded work, evaluation results and other captured documentation into custom ‘collections’ organized by competency, self-ratings, evaluation results or other criteria. The learner can create unlimited collections of work, displaying best work in demonstrating a specific competency or skill. And, within the collection, the learner can include additional annotations of notes, reflections, or comments specific to that collection’s intended audience. These collections can be ‘shared’ with anyone with access to the Internet. This feature empowers learners to go outside their immediate learning environment to request feedback or demonstrate their accomplishments.

REBEL has been successfully deployed for two years. Feedback is positive from each user group, and the DMS residency program is compliant with their accrediting agency, ACGME. The success of this application in supporting the process of becoming and demonstrating the competency of both self-directed learning and domain expertise leads us to raise additional questions about the application of this process to other kinds of learning.

Discussion Questions:

How a level of proficiency in a competency domain influences a person’s effectiveness as a ‘self-directed’ learner? i.e., is a specific level of knowledge and skill necessary for a learner to be ‘self-directed’?

If yes, how can we characterize that level?

How do models of competency and models of learning drive the functional requirements for learning application development?

Are there lessons learned and best practices to share?

Are there functionalities that can be designed into a learning management application which will enable the ‘self-directed’ learner to accelerate time to competency?
What can a learning management application do to sustain competency?
What modifications would be necessary to make this process appropriate to other kinds or levels of education?

References:

ACGME Outcomes Project, http://www.acgme.org/outcome/project/proHome.asp
Dartmouth-Hitchcock Leadership Preventive Medicine Residency, http://www.dhmc.org/webpage.cfm?site_id=2&org_id=120&gsec_id=0&sec_id=0&item_id=3293
ACGME General Competencies, http://www.acgme.org/outcome/comp/compMin.asp, September 28, 1999
U.S. Department of Education, National Center for Education Statistics, Defining and Assessing Learning: Exploring Competency-Based Initiatives,http://nces.ed.gov/pubs2002/2002159.pdf , November 15, 2002
Ibid.
Reference: Gerald Grow, "Teaching Learners to be Self-Directed." Adult Education Quarterly, 41:3 (1991), 125-49. This author identifies 4 steps in contrast to our 5. Our first 4 are comparable, but we, and our faculty both considered it critical for Medical Professionals to sustain and ‘Bring the learning Forward’.