Developing Evidence-Based Formative Assessments

Delivering Efficient, Effective Learning

Today’s medical and health professions students have no time to waste. Demands on their time are only increasing as they strive to become excellent clinicians while balancing the need to achieve the outstanding grades and test scores that are required for the next steps in their education and career paths.

At Aquifer, we wondered: How can we help without compromising our dedication to supporting high-quality, comprehensive clinical learning?

In 2019, we began our journey to create the first nationally-developed comprehensive formative assessments for health professions students. The goal is to give students—and faculty—tools for focused learning and maximum efficiency. Aquifer’s formative assessment strategy is based on the proven concepts of test-enhanced learning combined with focused, repetitive practice.

Test-Enhanced Learning

Test-enhanced learning in health professions education is a pedagogical approach in which students are continuously assessed and re-assessed as a formative process. Test-enhanced learning is the result of the ongoing evolution from assessment of learning to assessment for learning (Schuwirth and Van der Vleuten 2011).

Research has shown that the “retrieval practice” of taking multiple formative assessments over time enhances students’ ability to recall, retain, and apply information over traditional study methodology. The effects are consistent for all health professions, learner levels, formats and learning outcomes (Green et al 2018). Test-enhanced learning is useful for helping students identify relatively weaker areas in need of further study, as well as areas of relative strength on which to build further, allowing for more focused and efficient study methods. Test-enhanced learning has also been shown to improve overall student study behaviors and aid in metacognition.

Focused, Repetitive Practice

Research demonstrates that student study habits and practice over time greatly influences knowledge retention and transfer to new problems. How does test-enhanced learning differ from what currently happens in most schools? Traditionally, students are exposed to new material in a classroom, lab, or assignment–and then move on to learning other new material. Students only go back and review all the material just before a summative mid-term or final exam is looming on the horizon.

On the other hand, research has shown that spaced, or distributed practice, is much more effective. This approach requires recalling information repeatedly over shorter intervals throughout a course of study. Long term retention has been found to be four times greater with spaced learning (Kerfoot). By frequently revisiting information, students can develop purposeful study habits based on evidence.

Aquifer’s Evidence-Based Approach

In our recent survey of Course and Clerkship Directors, more than half said they would require a high-quality formative assessment if one were available. These educators expressed a desire to provide students with immediate, granular feedback, and identified a clear gap: there is no nationally-available assessment designed for formative use in clinical education.

Aquifer is combining the concepts of test-enhanced learning and distributed practice to create shorter, more frequent assessments for learning. Our new formative assessment program will provide students with individualized performance reports and granular feedback that outlines specific learning objectives and teaching points that have been mastered or may require additional study, in addition to direct links to case content for focused self-directed learning.

To make this process more efficient for faculty and administrators, Aquifer will identify which students are performing at an expected level as well as those who may be at risk. A cohort summary report will identify specific learning objectives and teaching points that may require additional attention for all students to facilitate curriculum planning.

Addressing the Competence Gap

In addition to answering the single-best answer multiple-choice questions, students will also rate how certain they feel about the accuracy of their answer for each question. Performance reports will identify specific areas of mastery and uncertainty, as well as potential knowledge gaps and misconceptions in order to support student study plans. Reports will also provide each student with feedback on their current ‘cognitive competence’—an objective calibration of their perceived certainty and the accuracy of their answers. With time and repeated testing, students become armed with a more accurate picture of their current level of understanding and can establish targeted learning goals and study plans to meet their individual needs.

Deeper Dive

Explore how the Master Adaptive Learner Framework applies to clinical learning and how Aquifer’s formative assessment approach supports the development of mastery learning. Here, we demonstrate a coaching approach to facilitate a rich student-faculty dialogue and create personalized plans for mastery learning using real-world examples from Aquifer’s formative assessment.

Presenters:

• David Anthony, MD, M.SC (Brown University, Aquifer Family Medicine Senior Director)
• Jason Chao, MD (Case Western Reserve University, Aquifer Family Medicine Assessment Lead)
• Sherilyn Smith, MD (University of Washington, Aquifer Chief Academic Officer)
• Leslie Fall, MD (Aquifer Chief Executive Officer)

Download Workshop Handout

Learn More

Aquifer Calibrate is actively in development. This innovative new assessment system will accompany our Family Medicine, Internal Medicine, Pediatrics, and Radiology courses. We expect to begin a robust pilot testing program in 2022.