فصل 02

After the teacher stepped out of the room, Agarwal projected a series of slides onto the board at the front of the room and read them to the students. Each slide presented either a multiple choice question or a statement of fact. When the slide contained a question, students used clickers (handheld, cell-phone-like remotes) to indicate their answer choice: A, B, C, or D. When all had responded, the correct answer was revealed, so as to provide feedback and correct errors. (Although teachers were not present for these quizzes, under normal circumstances, with teachers administering quizzes, they would see immediately how well students are tracking the study material and use the results to guide further discussion or study.) Unit exams were the normal pencil-and-paper tests given by the teacher. Exams were also given at the end of the semester and at the end of the year. Students had been exposed to all of the material tested in these exams through the teacher’s normal classroom lessons, homework, worksheets, and so on, but they had also been quizzed three times on one-third of the material, and they had seen another third presented for additional study three times. The balance of the material was neither quizzed nor additionally reviewed in class beyond the initial lesson and whatever reading a student may have done.

The results were compelling: The kids scored a full grade level higher on the material that had been quizzed than on the material that had not been quizzed. Moreover, test results for the material that had been reviewed as statements of fact but not quizzed were no better than those for the nonreviewed material. Again, mere rereading does not much help.

In 2007, the research was extended to eighth grade science classes, covering genetics, evolution, and anatomy. The regimen was the same, and the results equally impressive. At the end of three semesters, the eighth graders averaged 79 percent (C+) on the science material that had not been quizzed, compared to 92 percent (A−) on the material that had been quizzed.

The testing effect persisted eight months later at the end-of-year exams, confirming what many laboratory studies have shown about the long-term benefits of retrieval practice. The effect doubtless would have been greater if the retrieval practice had continued and occurred once a month, say, in the intervening months.8

The lesson from these studies has been taken to heart by many of the teachers at Columbia Middle School. Long after concluding their participation in the research studies, Patrice Bain’s sixth grade social studies classes continue today to follow a schedule of quizzes before lessons, quizzes after lessons, and then a review quiz prior to the chapter test. Jon Wehrenberg, an eighth grade history teacher who was not part of the research, has knitted retrieval practice into his classroom in many different forms, including quizzing, and he provides additional online tools at his website, like flashcards and games. After reading passages on the history of slavery, for example, his students are asked to write down ten facts about slavery they hadn’t known before reading the passages. You don’t need electronic gadgetry to practice retrieval.

Seven sixth and seventh graders needing to improve their reading and comprehension skills sat in Michelle Spivey’s English classroom one period recently with their reading books open to an amusing story. Each student was invited to read a paragraph aloud. Where a student stumbled, Miss Spivey had him try again. When he’d gotten it right, she probed the class to explain the meaning of the passage and what might have been going on in the characters’ minds. Retrieval and elaboration; again, no technology required.

Quizzes at Columbia Middle School are not onerous events. Following completion of the research studies, students’ views were surveyed on this question. Sixty-four percent said the quizzing reduced their anxiety over unit exams, and 89 percent felt it increased learning. The kids expressed disappointment on days when clickers were not used, because the activity broke up the teacher’s lecture and proved enjoyable.

Principal Chamberlain, when asked what he thought the study results indicated, replied simply: “Retrieval practice has a significant impact on kids’ learning. This is telling us that it’s valuable, and that teachers are well advised to incorporate it into their instructional technique.”9

Are similar effects found at a later age?

Andrew Sobel teaches a class in international political economics at Washington University in St. Louis, a lecture course populated by 160–170 students, mostly freshmen and sophomores. Over a period of several years he noticed a growing problem with attendance. On any given day by midsemester, 25–35 percent of the class would be absent, compared to earlier in the semester when maybe 10 percent would be absent. The problem wasn’t unique to his class, he says. A lot of professors give students their PowerPoint slides, so the students just stop coming to class. Sobel fought back by withholding his slides, but by the end of the semester, many students stopped showing up anyway. The class syllabus included two big tests, a midterm and a final. Looking for some way to leverage attendance, Sobel replaced the big tests with nine pop quizzes. Because the quizzes would determine the course grade and would be unannounced, students would be well advised to show up for class.

The results were distressing. Over the semester, a third or more of the students bailed out. “I really got hammered in the teaching reviews,” Sobel told us. “The kids hated it. If they didn’t do well on a quiz they dropped the course rather than get a bad grade in it. Of those who stayed, I got this bifurcation between those who actually showed up and did the work, and those who didn’t. I found myself handing out A-plusses, which I’d never given before, and more Cs than I’d ever given.”10 With so much pushback, he had little choice but to drop the experiment and reinstate the old format, lectures with a midterm and final. A couple of years later, however, after hearing a presentation about the learning benefits of testing, he added a third major test during the semester to see what effect it might have on his students’ learning. They did better, but not by as much as he’d hoped, and the attendance problems persisted.

He scratched his head and changed the syllabus once again. This time he announced that there would be nine quizzes during the semester, and he was explicit about when they would be. No surprises, and no midterm or final exams, because he didn’t want to give up that much of his lecture time.

Despite fears that enrollments would plummet again, they actually increased by a handful. “Unlike the pop quizzes, which kids hate, these were all on the syllabus. If they missed one it was their own fault. It wasn’t because I surprised them or was being pernicious. They were comfortable with that.” Sobel took satisfaction in seeing attendance improve as well. “They would skip some classes on the days they didn’t have a quiz, particularly the spring semester, but they showed up for the quizzes.” Like the course, the quizzes were cumulative, and the questions were similar to those on the exams he used to give, but the quality of the answers he was getting by midsemester was much better than he was accustomed to seeing on the midterms. Five years into this new format, he’s sold on it. “The quality of discussions in class has gone way up. I see that big a difference in their written work, just by going from three exams to nine quizzes.” By the end of the semester he has them writing paragraphs on the concepts covered in class, sometimes a full-page essay, and the quality is comparable to what he’s seeing in his upper division classes.

“Anybody can design this structure. But I also realize that, Oh, god, if I’d done this years ago I would have taught them that much more stuff. The interesting thing about adopting this strategy is I now recognize that as good a teacher as I might think I am, my teaching is only a component of their learning, and how I structure it has a lot to do with it, maybe even more.” Meanwhile, the course enrollment has grown to 185 and counting.

Exploring Nuances

Andy Sobel’s example is anecdotal and likely reflects a variety of beneficial influences, not least being the cumulative learning effects that accrue like compounded interest when course material is carried forward in a regime of quizzes across an entire semester. Nonetheless, his experience squares with empirical research designed to tease apart the effects and nuances of testing.

For example, in one experiment college students studied prose passages on various scientific topics like those taught in college and then either took an immediate recall test after the initial exposure or restudied the material. After a delay of two days, the students who took the initial test recalled more of the material than those who simply restudied it (68 v. 54 percent), and this advantage was sustained a week later (56 v. 42 percent). Another experiment found that after one week a study-only group showed the most forgetting of what they initially had been able to recall, forgetting 52 percent, compared to a repeated-testing group, who forgot only 10 percent.11 How does giving feedback on wrong answers to test questions affect learning? Studies show that giving feedback strengthens retention more than testing alone does, and, interestingly, some evidence shows that delaying the feedback briefly produces better long-term learning than immediate feedback. This finding is counterintuitive but is consistent with researchers’ discoveries about how we learn motor tasks, like making layups or driving a golf ball toward a distant green. In motor learning, trial and error with delayed feedback is a more awkward but effective way of acquiring a skill than trial and correction through immediate feedback; immediate feedback is like the training wheels on a bicycle: the learner quickly comes to depend on the continued presence of the correction.

In the case of learning motor skills, one theory holds that when there’s immediate feedback it comes to be part of the task, so that later, in a real-world setting, its absence becomes a gap in the established pattern that disrupts performance. Another idea holds that frequent interruptions for feedback make the learning sessions too variable, preventing establishment of a stabilized pattern of performance.12

In the classroom, delayed feedback also yields better long-term learning than immediate feedback does. In the case of the students studying prose passages on science topics, some were shown the passage again even while they were asked to answer questions about it, in effect providing them with continuous feedback during the test, analogous to an open-book exam. The other group took the test without the study material at hand and only afterward were given the passage and instructed to look over their responses. Of course, the open-book group performed best on the immediate test, but those who got corrective feedback after completing the test retained the learning better on a later test. Delayed feedback on written tests may help because it gives the student practice that’s spaced out in time; as discussed in the next chapter, spacing practice improves retention.13 Are some kinds of retrieval practice more effective for long-term learning than others? Tests that require the learner to supply the answer, like an essay or short-answer test, or simply practice with flashcards, appear to be more effective than simple recognition tests like multiple choice or true/false tests. However, even multiple choice tests like those used at Columbia Middle School can yield strong benefits. While any kind of retrieval practice generally benefits learning, the implication seems to be that where more cognitive effort is required for retrieval, greater retention results. Retrieval practice has been studied extensively in recent years, and an analysis of these studies shows that even a single test in a class can produce a large improvement in final exam scores, and gains in learning continue to increase as the number of tests increases.14 Whichever theories science eventually tells us are correct about how repeated retrieval strengthens memory, empirical research shows us that the testing effect is real—that the act of retrieving a memory changes the memory, making it easier to retrieve again later.

How widely is retrieval practice used as a study technique? In one survey, college students were largely unaware of its effectiveness. In another survey, only 11 percent of college students said they use this study strategy. Even when they did report testing themselves, they mostly said they did it to discover what they didn’t know, so they could study that material more. That’s a perfectly valid use of testing, but few students realize that retrieval itself creates greater retention.15 Is repeated testing simply a way to expedite rote learning? In fact, research indicates that testing, compared to rereading, can facilitate better transfer of knowledge to new contexts and problems, and that it improves one’s ability to retain and retrieve material that is related but not tested. Further research is needed on this point, but it seems that retrieval practice can make information more accessible when it is needed in various contexts.

Do students resist testing as a tool for learning? Students do generally dislike the idea of tests, and it’s not hard to see why, in particular in the case of high-stakes tests like midterms and finals, where the score comes with significant consequences. Yet in all studies of testing that reported students’ attitudes, the students who were tested frequently rated their classes more favorably at the end of the semester than those tested less frequently. Those who were frequently tested reached the end of the semester on top of the material and did not need to cram for exams.

How does taking a test affect subsequent studying? After a test, students spend more time restudying the material they missed, and they learn more from it than do their peers who restudy the material without having been tested. Students whose study strategies emphasize rereading but not self-testing show overconfidence in their mastery. Students who have been quizzed have a double advantage over those who have not: a more accurate sense of what they know and don’t know, and the strengthening of learning that accrues from retrieval practice.16 Are there any further, indirect benefits of regular, low-stakes classroom testing? Besides strengthening learning and retention, a regime of this kind of testing improves student attendance. It increases studying before class (because students know they’ll be quizzed), increases attentiveness during class if students are tested at the end of class, and enables students to better calibrate what they know and where they need to bone up. It’s an antidote to mistaking fluency with the text, resulting from repeated readings, for mastery of the subject. Frequent low-stakes testing helps dial down test anxiety among students by diversifying the consequences over a much larger sample: no single test is a make-or-break event. And this kind of testing enables instructors to identify gaps in students’ understanding and adapt their instruction to fill them. These benefits of low-stakes testing accrue whether instruction is delivered online or in the classroom.17 The Takeaway

Practice at retrieving new knowledge or skill from memory is a potent tool for learning and durable retention. This is true for anything the brain is asked to remember and call up again in the future—facts, complex concepts, problem-solving techniques, motor skills.

Effortful retrieval makes for stronger learning and retention. We’re easily seduced into believing that learning is better when it’s easier, but the research shows the opposite: when the mind has to work, learning sticks better. The greater the effort to retrieve learning, provided that you succeed, the more that learning is strengthened by retrieval. After an initial test, delaying subsequent retrieval practice is more potent for reinforcing retention than immediate practice, because delayed retrieval requires more effort.

Repeated retrieval not only makes memories more durable but produces knowledge that can be retrieved more readily, in more varied settings, and applied to a wider variety of problems.

While cramming can produce better scores on an immediate exam, the advantage quickly fades because there is much greater forgetting after rereading than after retrieval practice. The benefits of retrieval practice are long-term.

Simply including one test (retrieval practice) in a class yields a large improvement in final exam scores, and gains continue to increase as the frequency of classroom testing increases.

Testing doesn’t need to be initiated by the instructor. Students can practice retrieval anywhere; no quizzes in the classroom are necessary. Think flashcards—the way second graders learn the multiplication tables can work just as well for learners at any age to quiz themselves on anatomy, mathematics, or law. Self-testing may be unappealing because it takes more effort than rereading, but as noted already, the greater the effort at retrieval, the more will be retained.

Students who take practice tests have a better grasp of their progress than those who simply reread the material. Similarly, such testing enables an instructor to spot gaps and misconceptions and adapt instruction to correct them.

Giving students corrective feedback after tests keeps them from incorrectly retaining material they have misunderstood and produces better learning of the correct answers.

Students in classes that incorporate low-stakes quizzing come to embrace the practice. Students who are tested frequently rate their classes more favorably.

What about Principal Roger Chamberlain’s initial concerns about practice quizzing at Columbia Middle School—that it might be nothing more than a glorified path to rote learning?

When we asked him this question after the study was completed, he paused for a moment to gather his thoughts. “What I’ve really gained a comfort level with is this: for kids to be able to evaluate, synthesize, and apply a concept in different settings, they’re going to be much more efficient at getting there when they have the base of knowledge and the retention, so they’re not wasting time trying to go back and figure out what that word might mean or what that concept was about. It allows them to go to a higher level.” 3 Mix Up Your Practice

IT MAY NOT BE INTUITIVE that retrieval practice is a more powerful learning strategy than repeated review and rereading, yet most of us take for granted the importance of testing in sports. It’s what we call “practice-practice-practice.” Well, here’s a study that may surprise you.

A group of eight-year-olds practiced tossing beanbags into buckets in gym class. Half of the kids tossed into a bucket three feet away. The other half mixed it up by tossing into buckets two feet and four feet away. After twelve weeks of this they were all tested on tossing into a three-foot bucket. The kids who did the best by far were those who’d practiced on two- and four-foot buckets but never on three-foot buckets.1

Why is this? We will come back to the beanbags, but first a little insight into a widely held myth about how we learn.

The Myth of Massed Practice

Most of us believe that learning is better when you go at something with single-minded purpose: the practice-practice-practice that’s supposed to burn a skill into memory. Faith in focused, repetitive practice of one thing at a time until we’ve got it nailed is pervasive among classroom teachers, athletes, corporate trainers, and students. Researchers call this kind of practice “massed,” and our faith rests in large part on the simple fact that when we do it, we can see it making a difference. Nevertheless, despite what our eyes tell us, this faith is misplaced.

If learning can be defined as picking up new knowledge or skills and being able to apply them later, then how quickly you pick something up is only part of the story. Is it still there when you need to use it out in the everyday world? While practicing is vital to learning and memory, studies have shown that practice is far more effective when it’s broken into separate periods of training that are spaced out. The rapid gains produced by massed practice are often evident, but the rapid forgetting that follows is not. Practice that’s spaced out, interleaved with other learning, and varied produces better mastery, longer retention, and more versatility. But these benefits come at a price: when practice is spaced, interleaved, and varied, it requires more effort. You feel the increased effort, but not the benefits the effort produces. Learning feels slower from this kind of practice, and you don’t get the rapid improvements and affirmations you’re accustomed to seeing from massed practice. Even in studies where the participants have shown superior results from spaced learning, they don’t perceive the improvement; they believe they learned better on the material where practice was massed.

Almost everywhere you look, you find examples of massed practice: summer language boot camps, colleges that offer concentration in a single subject with the promise of fast learning, continuing education seminars for professionals where training is condensed into a single weekend. Cramming for exams is a form of massed practice. It feels like a productive strategy, and it may get you through the next day’s midterm, but most of the material will be long forgotten by the time you sit down for the final. Spacing out your practice feels less productive for the very reason that some forgetting has set in and you’ve got to work harder to recall the concepts. It doesn’t feel like you’re on top of it. What you don’t sense in the moment is that this added effort is making the learning stronger.2 Spaced Practice

The benefits of spacing out practice sessions are long established, but for a vivid example consider this study of thirty-eight surgical residents. They took a series of four short lessons in microsurgery: how to reattach tiny vessels. Each lesson included some instruction followed by some practice. Half the docs completed all four lessons in a single day, which is the normal in-service schedule. The others completed the same four lessons but with a week’s interval between them.3 In a test given a month after their last session, those whose lessons had been spaced a week apart outperformed their colleagues in all areas—elapsed time to complete a surgery, number of hand movements, and success at reattaching the severed, pulsating aortas of live rats. The difference in performance between the two groups was impressive. The residents who had taken all four sessions in a single day not only scored lower on all measures, but 16 percent of them damaged the rats’ vessels beyond repair and were unable to complete their surgeries.

Why is spaced practice more effective than massed practice? It appears that embedding new learning in long-term memory requires a process of consolidation, in which memory traces (the brain’s representations of the new learning) are strengthened, given meaning, and connected to prior knowledge—a process that unfolds over hours and may take several days. Rapid-fire practice leans on short-term memory. Durable learning, however, requires time for mental rehearsal and the other processes of consolidation. Hence, spaced practice works better. The increased effort required to retrieve the learning after a little forgetting has the effect of retriggering consolidation, further strengthening memory. We explore some of the theories about this process in the next chapter.

Interleaved Practice

Interleaving the practice of two or more subjects or skills is also a more potent alternative to massed practice, and here’s a quick example of that. Two groups of college students were taught how to find the volumes of four obscure geometric solids (wedge, spheroid, spherical cone, and half cone). One group then worked a set of practice problems that were clustered by problem type (practice four problems for computing the volume of a wedge, then four problems for a spheroid, etc.). The other group worked the same practice problems, but the sequence was mixed (interleaved) rather than clustered by type of problem. Given what we’ve already presented, the results may not surprise you. During practice, the students who worked the problems in clusters (that is, massed) averaged 89 percent correct, compared to only 60 percent for those who worked the problems in a mixed sequence. But in the final test a week later, the students who had practiced solving problems clustered by type averaged only 20 percent correct, while the students whose practice was interleaved averaged 63 percent. The mixing of problem types, which boosted final test performance by a remarkable 215 percent, actually impeded performance during initial learning.4 Now, suppose you’re a trainer in a company trying to teach employees a complicated new process that involves ten procedures. The typical way of doing this is to train up in procedure 1, repeating it many times until the trainees really seem to have it down cold. Then you go to procedure 2, you do many repetitions of 2, you get that down, and so on. That appears to produce fast learning. What would interleaved practice look like? You practice procedure 1 just a few times, then switch to procedure 4, then switch to 3, then to 7, and so on. (Chapter 8 tells how Farmers Insurance trains new agents in a spiraling series of exercises that cycle back to key skillsets in a seemingly random sequence that adds layers of context and meaning at each turn.) The learning from interleaved practice feels slower than learning from massed practice. Teachers and students sense the difference. They can see that their grasp of each element is coming more slowly, and the compensating long-term advantage is not apparent to them. As a result, interleaving is unpopular and seldom used. Teachers dislike it because it feels sluggish. Students find it confusing: they’re just starting to get a handle on new material and don’t feel on top of it yet when they are forced to switch. But the research shows unequivocally that mastery and long-term retention are much better if you interleave practice than if you mass it.

Varied Practice

Okay, what about the beanbag study where the kids who did best had never practiced the three-foot toss that the other kids had only practiced?

The beanbag study focused on mastery of motor skills, but much evidence has shown that the underlying principle applies to cognitive learning as well. The basic idea is that varied practice—like tossing your beanbags into baskets at mixed distances—improves your ability to transfer learning from one situation and apply it successfully to another. You develop a broader understanding of the relationships between different conditions and the movements required to succeed in them; you discern context better and develop a more flexible “movement vocabulary”—different movements for different situations. Whether the scope of variable training (e.g., the two- and four-foot tosses) must encompass the particular task (the three-foot toss) is subject for further study.

The evidence favoring variable training has been supported by recent neuroimaging studies that suggest that different kinds of practice engage different parts of the brain. The learning of motor skills from varied practice, which is more cognitively challenging than massed practice, appears to be consolidated in an area of the brain associated with the more difficult process of learning higher-order motor skills. The learning of motor skills from massed practice, on the other hand, appears to be consolidated in a different area of the brain that is used for learning more cognitively simple and less challenging motor skills. The inference is that learning gained through the less challenging, massed form of practice is encoded in a simpler or comparatively impoverished representation than the learning gained from the varied and more challenging practice which demands more brain power and encodes the learning in a more flexible representation that can be applied more broadly.5 Among athletes, massed practice has long been the rule: take your hook shot, knock the twenty-foot putt, work your backhand return, throw the pass while rolling out: again and again and again—to get it right and train your “muscle memory.” Or so the notion holds. The benefits of variable training for motor learning have been gaining broader acceptance, albeit slowly. Consider the one-touch pass in hockey. That’s where you receive the puck and immediately pass it to a teammate who’s moving down the ice, keeping the opposition off balance and unable to put pressure on the puck carrier. Jamie Kompon, when he was assistant coach of the Los Angeles Kings, was in the habit of running team practice on one-touch passes from the same position on the rink. Even if this move is interleaved with a sequence of other moves in practice, if you only do it at the same place on the rink or in the same sequence of moves, you are only, as it were, throwing your beanbags into the three-foot bucket. Kompon is onto the difference now and has changed up his drills. Since we talked, he’s gone over to the Chicago Blackhawks. We would have said “Keep an eye on those Blackhawks” here, but as we revise to go into production, Kompon and team have already won the Stanley Cup. Perhaps no coincidence?

The benefits of variable practice for cognitive as opposed to motor skills learning were shown in a recent experiment that adapted the beanbag test to verbal learning: in this case, the students solved anagrams–that is, they rearranged letters to form words (tmoce becomes comet). Some subjects practiced the same anagram over and over, whereas others practiced multiple anagrams for the word. When they were all tested on the same anagram that the former group had practiced on, the latter group performed better on it! The same benefits will apply whether you are practicing to identify tree species, differentiate the principles of case law, or master a new computer program.6 Developing Discrimination Skills

Compared to massed practice, a significant advantage of interleaving and variation is that they help us learn better how to assess context and discriminate between problems, selecting and applying the correct solution from a range of possibilities. In math education, massing is embedded in the textbook: each chapter is dedicated to a particular kind of problem, which you study in class and then practice by working, say, twenty examples for homework before you move on. The next chapter has a different type of problem, and you dive into the same kind of concentrated learning and practice of that solution. On you march, chapter by chapter, through the semester. But then, on the final exam, lo and behold, the problems are all mixed up: you’re staring at each one in turn, asking yourself Which algorithm do I use? Was it in chapter 5, 6, or 7? When you have learned under conditions of massed or blocked repetition, you have had no practice on that critical sorting process. But this is the way life usually unfolds: problems and opportunities come at us unpredictably, out of sequence. For our learning to have practical value, we must be adept at discerning “What kind of problem is this?” so we can select and apply an appropriate solution.

Several studies have demonstrated the improved powers of discrimination to be gained through interleaved and varied practice. One study involved learning to attribute paintings to the artists who created them, and another focused on learning to identify and classify birds.

Researchers initially predicted that massed practice in identifying painters’ works (that is, studying many examples of one painter’s works before moving on to study many examples of another’s works) would best help students learn the defining characteristics of each artist’s style. Massed practice of each artist’s works, one artist at a time, would better enable students to match artworks to artists later, compared to interleaved exposure to the works of different artists. The idea was that interleaving would be too hard and confusing; students would never be able to sort out the relevant dimensions. The researchers were wrong. The commonalities among one painter’s works that the students learned through massed practice proved less useful than the differences between the works of multiple painters that the students learned through interleaving. Interleaving enabled better discrimination and produced better scores on a later test that required matching the works with their painters. The interleaving group was also better able to match painters’ names correctly to new examples of their work that the group had never viewed during the learning phase. Despite these results, the students who participated in these experiments persisted in preferring massed practice, convinced that it served them better. Even after they took the test and could have realized from their own performance that interleaving was the better strategy for learning, they clung to their belief that the concentrated viewing of paintings by one artist was better. The myths of massed practice are hard to exorcise, even when you’re experiencing the evidence yourself.7 The power of interleaving practice to improve discriminability has been reaffirmed in studies of people learning bird classification. The challenge here is more complex than it might seem. One study addressed twenty different bird families (thrashers, swallows, wrens, finches, and so on). Within each family, students were presented with a dozen species (brown thrasher, curve-billed thrasher, Bendire’s thrasher, etc.). To identify a bird’s family, you consider a wide range of traits like size, plumage, behavior, location, beak shape, iris color, and so on. A problem in bird identification is that members of a family share many traits in common but not all. For instance, many but not all thrashers have a long, slightly hooked beak. There are traits that are typical of a family but none that occur in all members of that family and can serve as unique identifiers. Because rules for classification can only rely on these characteristic traits rather than on defining traits (ones that hold for every member), bird classification is a matter of learning concepts and making judgments, not simply memorizing features. Interleaved and variable practice proved more helpful than massed practice for learning the underlying concepts that unite and differentiate the species and families.

To paraphrase a conclusion from one of these studies, recall and recognition require “factual knowledge,” considered to be a lower level of learning than “conceptual knowledge.” Conceptual knowledge requires an understanding of the interrelationships of the basic elements within a larger structure that enable them to function together. Conceptual knowledge is required for classification. Following this logic, some people argue that practicing retrieval of facts and exemplars would fall short as a strategy for comprehending general characteristics that are required for higher levels of intellectual behavior. The bird classification studies suggest the opposite: strategies of learning that help students identify and discern complex prototypes (family resemblances) can help them grasp the kinds of contextual and functional differences that go beyond the acquisition of simple forms of knowledge and reach into the higher sphere of comprehension.8 Improving Complex Mastery for Medical Students

The distinction between straightforward knowledge of facts and deeper learning that permits flexible use of the knowledge may be a little fuzzy, but it resonates with Douglas Larsen at Washington University Medical School in St. Louis, who says that the skills required for bird classification are similar to those required of a doctor diagnosing what’s wrong with a patient. “The reason variety is important is it helps us see more nuances in the things that we can compare against,” he says. “That comes up a lot in medicine, in the sense that every patient visit is a test. There are many layers of explicit and implicit memory involved in the ability to discriminate between symptoms and their interrelationships.” Implicit memory is your automatic retrieval of past experience in interpreting a new one. For example, the patient comes in and gives you a story. As you listen, you’re consciously thinking through your mental library to see what fits, while also unconsciously polling your past experiences to help interpret what the patient is telling you. “Then you’re left with making a judgment call,” Larsen says.9 Larsen is a pediatric neurologist seeing patients in the university clinic and hospital. He’s a busy guy: in addition to practicing medicine, he supervises the work of physicians in training, he teaches, and as time permits, he conducts research into medical education, working in collaboration with cognitive psychologists. He’s drawing on all of these roles to redesign and strengthen the school’s training curriculum in pediatric neurology.

As you’d expect, the medical school employs a wide spectrum of instructional techniques. Besides classroom lectures and labs, students practice resuscitations and other procedures on high-tech mannequins in three simulation centers the school maintains. Each “patient” is hooked up to monitors, has a heartbeat, blood pressure, pupils that dilate and constrict, and the ability to listen and speak, thanks to a controller who observes and operates the mannequin from a back room. The school also makes use of “standardized patients,” actors who follow scripts and exhibit symptoms the students are required to diagnose. The center is set up like a regular medical clinic, and students must show proficiency in all aspects of a patient encounter, from bedside manner, physical exam skills, and remembering to ask the full spectrum of pertinent questions to arriving at a diagnosis and treatment plan.

From studies of these teaching methods, Larsen has drawn some interesting conclusions. First—and this may seem self-evident: you do better on a test to demonstrate your competency at seeing patients in a clinic if your learning experience has involved seeing patients in a clinic. Simply reading about patients is not enough. However, on written final exams, medical students who have examined patients and those who have learned via written tests do equally well. The reason is that in a written test the student is being given considerable structure and being asked for specific information. When examining the patient, you have to come up on your own with the right mental model and the steps to follow. Having practiced these steps on patients or simulated patients improves performance relative to just reading about how to do it. In other words, the kind of retrieval practice that proves most effective is one that reflects what you’ll be doing with the knowledge later. It’s not just what you know, but how you practice what you know that determines how well the learning serves you later. As the sports adage goes, “practice like you play and you will play like you practice.” This conclusion lines up with other research into learning, and with some of the more sophisticated training practices in science and industry, including the increasingly broad use of simulators—not just for jet pilots and medical students but for cops, towboat pilots, and people in almost any field you can name that requires mastery of complex knowledge and skills and where the stakes for getting it right are high. Book learning is not enough in these cases; actual hands-on practice is needed.

Second, while it is important for a medical student to build breadth by seeing a wide variety of patients manifesting different diseases, placing too much emphasis on variety runs the risk of underemphasizing repeated retrieval practice on the basics—on the typical way the disease presents itself in most patients.

“There’s a certain set of diseases that we want you to know very well,” Larsen says. “So we’re going to have you see these standardized patients again and again, and assess your performance until you really have that down and can show us, ‘I really do that well.’ It’s not either/or, variety versus repetition. We need to make sure we’re appropriately balanced, and also recognize that we sometimes fall into the trap of familiarity. ‘I’ve already seen a bunch of patients with this problem, I don’t need to keep seeing them.’ But really, repeated retrieval practice is crucial to long-term retention, and it’s a critical aspect of training.” A third critical aspect is practical experience. For a doctor, seeing patients provides a natural cycle of spaced retrieval practice, interleaving, and variety. “So much of medicine is based on learning by experience, which is why, after the first two years, we take students out of the classroom and start putting them into clinical settings. A huge question is, what is it about learning and experience that come together? We have lots of experiences we don’t learn from. What differentiates those that teach us something?” One form of practice that helps us learn from experience, as the neurosurgeon Mike Ebersold recounted in Chapter 2, is reflection. Some people are more given to the act of reflection than others, so Doug Larsen has broadened his research to study how you might structure reflection as an integral part of the training, helping students cultivate it as a habit. He is experimenting with requiring students to write daily or weekly summaries of what they did, how it worked, and what they might do differently next time to get better results. He speculates that daily reflection, as a form of spaced retrieval practice, is probably just as critical in the real-world application of medicine as quizzing and testing are in building competencies in medical school.

What about the classroom lecture, or the typical in-service training conference that’s compressed over a couple of days? Larsen figures his school’s interns spend 10 percent of their time sitting in conferences listening to lectures. It may be a talk on metabolic diseases, on different infectious diseases, or on different drugs. The speaker puts the PowerPoint slideshow up and starts going through it. Usually there’s lunch, and the docs eat, listen, and leave.

“In my mind, considering how much forgetting occurs, it’s very discouraging that we’re putting so many resources into an activity that, the way it is currently done, learning research tells us is so ineffective. Medical students and residents go to these conferences and they have no repeated exposure whatsoever to it. It’s just a matter of happenstance whether they end up finally seeing a patient in the future whose problem relates back to the conference topic. Otherwise, they don’t study the material, they are certainly not tested on the material, they just listen then they walk out.” At a minimum, Larsen would like to see something done to interrupt the forgetting: give a quiz at the end of a conference and follow it with spaced retrieval practice. “Make quizzing a standard part of the culture and the curriculum. You just know every week you’re going to get in your email your ten questions that you need to work through.”

He asks, “How are we designing education and training systems that prevent or at least intervene in the amount of forgetting that goes on, and making sure they’re systematic throughout the school in support of what we’re trying to accomplish? As it stands now, medical resident programs are simply dictating: you have to have the curriculum, you have to have the conferences, and it ends there. They present these big conferences, they have all the faculty come through and give their talks. And in the end, what we actually accomplish is really kind of minimal.”10 These Principles Are Broadly Applicable

College football might seem an incongruous place to look for a learning model, but a conversation with Coach Vince Dooley about the University of Georgia’s practice regime provides an intriguing case.

Dooley is authoritative on the subject. As head coach of Bulldogs football from 1964–1988, he piled up an astonishing 201 wins with only 77 losses and 10 tied games, winning six conference titles and a national championship. He went on to serve as the university’s athletic director, where he built one of the most impressive athletics programs in the country.

We asked Coach Dooley how players go about mastering all the complexities of the game. His theories of coaching and training revolve around the weekly cycle of one Saturday game to the next.

In that short period there’s a lot to learn: studying the opposition’s type of game in the classroom, discussing offensive and defensive strategies for opposing it, taking the discussion onto the playing field, breaking the strategies down to the movements of individual positions and trying them out, knitting the parts into a whole, and then repeating the moves until they run like clockwork.

While all this is going on, the players must also keep their fundamental skills in top form: blocking, tackling, catching the ball, bringing the ball in, carrying the ball. Dooley believes that (1) you have to keep practicing the fundamentals from time to time, forever, so you keep them sharp, otherwise you’re cooked, but (2) you need to change it up in practice because too much repetition is boring. The position coaches work with players individually on specific skills and then on how they’re playing their positions during team practice.

What else? There’s practicing the kicking game. There’s the matter of each player’s mastery of the playbook. And there are the special plays from the team’s repertoire that often make the difference between winning and losing. In Dooley’s narrative, the special plays stand as exemplars of spaced learning: they’re practiced only on Thursdays, so there’s always a week between sessions, and the plays are run in a varied sequence.

With all this to be done, it’s not surprising that a critical aspect of the team’s success is a very specific daily and weekly schedule that interleaves the elements of individual and team practice. The start of every day’s practice is strictly focused on the fundamentals of each player’s position. Next, players practice in small groups, working on maneuvers involving several positions. These parts are gradually brought together and run as a team. Play is speeded up and slowed down, rehearsed mentally as well as physically. By midweek the team is running the plays in real time, full speed.

“You’re coming at it fast, and you’ve got to react fast,” Dooley said. “But as you get closer to game time, you slow it down again. Now it’s a kind of rehearsal without physical contact. The play basically starts out the same each time, but then what the opponent does changes it. So you’ve got to be able to adjust to that. You start into the motion and say, ‘If they react like this, then this is what you would do.’ You practice adjustments. If you do it enough times in different situations, then you’re able to do it pretty well in whatever comes up on the field.”11 How does a player get on top of his playbook? He takes it home and goes over the plays in his mind. He may walk through them. Everything in practice can’t be physically strenuous, Dooley said, or you’d wear yourself out, “so if the play calls for you to step this way and then go the other way, you can rehearse that in your mind, maybe just lean your body as if to go that way. And then if something happens where you have to adjust, you can do that mentally. By reading the playbook, rehearsing it in your mind, maybe taking a step or two to walk through it, you simulate something happening. So that kind of rehearsal is added to what you get in the classroom and on the field.” The final quarterback meetings are held on Saturday morning, reviewing the game plan and running through it mentally. The offensive coaches can make all the plans they want to about the hypothetical game, but once play gets under way, the execution rests in the hands of the quarterback.

For Coach Dooley’s team, it’s all there: retrieval, spacing, interleaving, variation, reflection, and elaboration. The seasoned quarterback going into Saturday’s game—mentally running through the plays, the reactions, the adjustments—is doing the same thing as the seasoned neurosurgeon who’s rehearsing what’s about to unfold in the operating room.

The Takeaway

Here’s a quick rundown of what we know today about massed practice and its alternatives. Scientists will continue to deepen our understanding.

We harbor deep convictions that we learn better through single-minded focus and dogged repetition, and these beliefs are validated time and again by the visible improvement that comes during “practice-practice-practice.” But scientists call this heightened performance during the acquisition phase of a skill “momentary strength” and distinguish it from “underlying habit strength.” The very techniques that build habit strength, like spacing, interleaving, and variation, slow visible acquisition and fail to deliver the improvement during practice that helps to motivate and reinforce our efforts.12 Cramming, a form of massed practice, has been likened to binge-and-purge eating. A lot goes in, but most of it comes right back out in short order. The simple act of spacing out study and practice in installments and allowing time to elapse between them makes both the learning and the memory stronger, in effect building habit strength.

How big an interval, you ask? The simple answer: enough so that practice doesn’t become a mindless repetition. At a minimum, enough time so that a little forgetting has set in. A little forgetting between practice sessions can be a good thing, if it leads to more effort in practice, but you do not want so much forgetting that retrieval essentially involves relearning the material. The time periods between sessions of practice let memories consolidate. Sleep seems to play a large role in memory consolidation, so practice with at least a day in between sessions is good.

Something as simple as a deck of flashcards can provide an example of spacing. Between repetitions of any individual card, you work through many others. The German scientist Sebastian Leitner developed his own system for spaced practice of flashcards, known as the Leitner box. Think of it as a series of four file-card boxes. In the first are the study materials (be they musical scores, hockey moves, or Spanish vocabulary flashcards) that must be practiced frequently because you often make mistakes in them. In the second box are the cards you’re pretty good at, and that box gets practiced less often than the first, perhaps by a half. The cards in the third box are practiced less often than those in the second, and so on. If you miss a question, make mistakes in the music, flub the one-touch pass, you move it up a box so you will practice it more often. The underlying idea is simply that the better your mastery, the less frequent the practice, but if it’s important to retain, it will never disappear completely from your set of practice boxes.

Beware of the familiarity trap: the feeling that you know something and no longer need to practice it. This familiarity can hurt you during self-quizzing if you take shortcuts. Doug Larsen says, “You have to be disciplined to say, ‘All right, I’m going to make myself recall all of this and if I don’t, what did I miss, how did I not know that?’ Whereas if you have an instructor-generated test or quiz, suddenly you have to do it, there’s an expectation, you can’t cheat, you can’t take mental shortcuts around it, you simply have to do that.” The nine quizzes Andy Sobel administers over the twenty-six meetings of his political economics course are a simple example of spaced retrieval practice, and of interleaving—because he rolls forward into each successive quiz questions pertaining to work from the beginning of the semester.

Interleaving two or more subjects during practice also provides a form of spacing. Interleaving can also help you develop your ability to discriminate later between different kinds of problems and select the right tool from your growing toolkit of solutions.

In interleaving, you don’t move from a complete practice set of one topic to go to another. You switch before each practice is complete. A friend of ours describes his own experience with this: “I go to a hockey class and we’re learning skating skills, puck handling, shooting, and I notice that I get frustrated because we do a little bit of skating and just when I think I’m getting it, we go to stick handling, and I go home frustrated, saying, ‘Why doesn’t this guy keep letting us do these things until we get it?’ ” This is actually the rare coach who understands that it’s more effective to distribute practice across these different skills than polish each one in turn. The athlete gets frustrated because the learning’s not proceeding quickly, but the next week he will be better at all aspects, the skating, the stick handling, and so on, than if he’d dedicated each session to polishing one skill.

Like interleaving, varied practice helps learners build a broad schema, an ability to assess changing conditions and adjust responses to fit. Arguably, interleaving and variation help learners reach beyond memorization to higher levels of conceptual learning and application, building more rounded, deep, and durable learning, what in motor skills shows up as underlying habit strength.

Something the researchers call “blocked practice” is easily mistaken for varied practice. It’s like the old LP records that could only play their songs in the same sequence. In blocked practice, which is commonly (but not only) found in sports, a drill is run over and over. The player moves from one station to the next, performing a different maneuver at each station. That’s how the LA Kings were practicing their one-touch pass before they got religion and started changing it up. It would be like always practicing flashcards in the same order. You need to shuffle your flashcards. If you always practice the same skill in the same way, from the same place on the ice or field, in the same set of math problems, or during the same sequence in a flight simulator, you’re starving your learning on short rations of variety.

Spacing, interleaving, and variability are natural features of how we conduct our lives. Every patient visit or football game is a test and an exercise in retrieval practice. Every routine traffic stop is a test for a cop. And every traffic stop is different, adding to a cop’s explicit and implicit memory and, if she pays attention, making her more effective in the future. The common term is “learning from experience.” Some people never seem to learn. One difference, perhaps, between those who do and don’t is whether they have cultivated the habit of reflection. Reflection is a form of retrieval practice (What happened? What did I do? How did it work out?), enhanced with elaboration (What would I do differently next time?).

As Doug Larsen reminds us, the connections between the neurons in the brain are very plastic. “Making the brain work is actually what seems to make a difference—bringing in more complex networks, then using those circuits repeatedly, which makes them more robust.”

4 Embrace Difficulties

WHEN MIA BLUNDETTO, age twenty-three, first lieutenant, U.S. Marine Corps, was billeted to logistics in Okinawa, she had to get her ticket punched at jump school. Describing that moment two years later, she said, “I hate falling, that feeling in your chest. There’s not a day in my life I wanted to jump out of an airplane. I wouldn’t even go down a water slide until I was in middle school. But I was in charge of a platoon of Marines who rigged parachutes and jumped out of airplanes and dropped cargo. It’s one of the most sought-out billets as a logistics officer, very hard to get. My commanding officer said, you know, ‘You will be air delivery platoon commander. If you don’t want to do that, I’ll put you somewhere else and we’ll let the next guy have that job.’ There’s no way I could let somebody else have this job that everybody wanted. So I looked him straight in the face and said, ‘Yes, sir, I’ll jump out of planes.’ ”1 Mia is five feet seven inches of blonde ambition. Her father, Frank, ex-marine, is in awe. “She’ll do more pull-ups than most of the guys in her class. She has the Maryland state record in the bench press, she was sixth in the NCAA for powerlifting. Very soft-spoken; you just don’t see it coming.” When we had Mia to ourselves, we asked her if Frank was blowing smoke. She laughed. “He likes to exaggerate.” But when pressed, she admitted to the facts. Until recently, women in the Marines were required to do flex arm hangs instead of pull-ups (where the chin crosses the plane of the pull-up bar), but the newly toughened rules effective in 2014 require a minimum of three pull-ups, the same as the minimum for men. Targets are eight pull-ups for women, twenty for men. Mia does thirteen and is shooting for twenty. As a student at the Naval Academy, she qualified two years in a row for nationals in powerlifting—three sets each of bench press, squats, and dead lifts—setting Maryland state records.

So we know she’s tough. An aversion to falling is an instinctual reflex for self-preservation, but her decision to take the assignment was a foregone conclusion, the kind of grit the Marines and the Blundettos are known for. Mia has a sister and two brothers. They’re all active duty Marines.

As it turned out, the third time Mia threw herself out the jump door of a C130 troop transport at 1,250 feet, she plummeted right onto another soldier’s inflated parachute. But we’re getting ahead of the story.

We’re interested in her jump school training because it’s a great example of how some difficulties that elicit more effort and that slow down learning—spacing, interleaving, mixing up practice, and others—will more than compensate for their inconvenience by making the learning stronger, more precise, and more enduring. Short-term impediments that make for stronger learning have come to be called desirable difficulties, a term coined by the psychologists Elizabeth and Robert Bjork.2 The army’s jump school at Fort Benning, Georgia, is designed to make sure you get it right and get it done, and it’s a model of learning through desirable difficulty. You are not allowed to carry a notebook and write notes. You listen, watch, rehearse, and execute. Jump school is a place where testing is the principal instructional medium, and the test is in the doing. And, like all things military, jump school adheres to a strict protocol. Get it right or get the boot.

The parachute landing fall, or PLF in military parlance, is a technique of hitting the ground and rolling in a way that distributes the impact over the balls of your feet, the side of your calf, the side of your thigh, the side of your hip, and the side of your back. There are six possible directions in which to execute the fall along the length of your body, determined by conditions in the moment such as the direction of your drift, the terrain, wind, and whether you’re oscillating as you approach the ground. In your first exposure to this essential skill of parachuting, you stand in a gravel pit where the PLF is explained and demonstrated. Then you try it: you practice falling along different planes of the body, you get corrective feedback, and you practice it again.

Over the ensuing week the difficulty is notched up. You stand on a platform two feet off the ground. On the command “Ready,” you rock up on the balls of your feet, feet and knees together, arms skyward. On the command “land,” you jump off the wall and execute your PLF.

The test becomes more difficult. You clip yourself onto a zip line a dozen feet off the ground, grab onto an overhead T-bar, and drift down to a landing site, where, on command, you release and execute the PLF. You practice falling to the right and left, forward and backward, mixing it up.

The difficulty is increased again. You climb to a platform twelve feet off the ground, where you practice strapping on your harness, checking gear using the buddy system, and jumping through a mockup of an airplane jump door. The harness has risers like those from a parachute, hooked to a zip line but allowing for the same long arc of suspension, and when you jump, you have the momentary downward sensation of free fall, followed by the broad oscillations of suspension as you move along the cable, getting familiar with the motions of a real jump. But at the bottom it’s the instructor, not you, who pulls the release and drops you the last two or three feet to earth, so now you’re executing your fall randomly, from all directions, simulating what’s to come.

Next, you climb a thirty-four-foot tower to practice all the elements of a jump and the choreography of a mass exit from the aircraft, learning how it feels to fall from a height, how to deal with equipment malfunctions, how to jump with a load of heavy combat equipment.

Through demonstration and simulation, in escalating levels of difficulty that must be mastered in order to progress from one to the next, you learn how to board the aircraft as a part of a jump crew and participate in the command sequence of thirty troops positioning for a mass exit over a drop zone. How to get out the jump door correctly, how to count one-thousand, two-thousand, three-thousand, four-thousand and feel your chute deploy, or if you get to six-thousand, to pull the cord on your reserve chute; how to deal with twisted suspension lines, avoid collisions, hold into the wind, sort out a tangled control line; how to avoid stealing air from another jumper; the contingencies for landing in trees, water, or power lines; how to jump by day or night, in different wind and weather.

The knowledge and skills to be acquired are many, and practice is spaced and interleaved, both by default, as you wait your turn at each of the staging areas, airplane mock-ups, jump platforms, and harness mechanisms, and by necessity, in order to cover all that must be mastered and integrate the disparate components. Finally, if you make it to week 3 without washing out, you jump for real, making five exits from a military transport. With successful completion of the training and five successful jumps, you earn your jump wings and Airborne certificate.

On Mia’s third jump, she was first in line at the port jump door with fourteen jumpers queued behind her and another fourteen queued behind the guy standing at the opposite door. “So what the first person does, in this case me, you hand off your static line to the Sergeant Airborne, and there’s a light and it’s red or green, and you get the one-minute warning, then the thirty-second warning. I’m standing at this door for a few minutes and it’s beautiful. It’s probably one of the prettiest things I’ve ever seen, but I was terrified. There was nothing to get in my way, nothing I had to think about except just waiting, waiting for the ‘Go!’ The guy at the other door went, then I jumped, and I’m counting one-thousand, two-thousand—and suddenly, at four thousand, I had a green parachute wrapped all around me! I’m thinking, There’s no way this can be my parachute! I’d felt my chute open, I’d felt that lift. I realized that I was on top of the first jumper, so I just sort of swam out of his parachute and steered away from him.”