فصل 07کتاب: کاملا بخاطر بسپار - علم یادگیری موفق / فصل 7
- زمان مطالعه 76 دقیقه
- سطح خیلی سخت
دانلود اپلیکیشن «زیبوک»
این فصل را میتوانید به بهترین شکل و با امکانات عالی در اپلیکیشن «زیبوک» بخوانید
متن انگلیسی فصل
Timothy Fellows, Intro Psych Student
Stephen Madigan, a professor at the University of Southern California, was astonished by the performance of a student in his Psych 100 course. “It’s a tough course,” Madigan says. “I use the most difficult, advanced textbook, and there’s just a nonstop barrage of material. Three-quarters of the way through the class, I noticed this student named Timothy Fellows was getting 90 to 95 percent of the points on all the class activities—exams, papers, short-answer questions, multiple-choice questions. Those were just extraordinary grades. Students this good—well he’s definitely an outlier. And so I just took him aside one day and said, ‘Could you tell me about your study habits?’ ”2 The year was 2005. Madigan did not know Fellows outside class but saw him around campus and at football games enough to observe that he had a life beyond his academics. “Psychology wasn’t his major, but it was a subject he cared about, and he just brought all his skills to bear.” Madigan still has the list of study habit Fellows outlined, and he shares it with incoming students to this day.
Among the highlights were these:
• Always does the reading prior to a lecture
• Anticipates test questions and their answers as he reads
• Answers rhetorical questions in his head during lectures to test his retention of the reading
• Reviews study guides, finds terms he can’t recall or doesn’t know, and relearns those terms
• Copies bolded terms and their definitions into a reading notebook, making sure that he understands them
• Takes the practice test that is provided online by his professor; from this he discovers which concepts he doesn’t know and makes a point to learn them
• Reorganizes the course information into a study guide of his design
• Writes out concepts that are detailed or important, posts them above his bed, and tests himself on them from time to time
• Spaces out his review and practice over the duration of the course
Fellows’s study habits are a good example of doing what works and keeping at it, so that practice is spaced and the learning is solidly embedded come exam time.
Tips for Lifelong Learners
The learning strategies we have just outlined for students are effective for anyone at any age. But they are centered around classroom instruction. Lifelong learners are using the same principles in a variety of less-structured settings.
In a sense, of course, we’re all lifelong learners. From the moment we’re born we start learning about the world around us through experimentation, trial and error, and random encounters with challenges that require us to recall what we did the last time we found ourselves in a similar circumstance. In other words, the techniques of generation, spaced practice and the like that we present in this book are organic (even if counterintuitive), and it’s not surprising that many people have already discovered their power in the pursuit of interests and careers that require continuous learning.
Nathaniel Fuller is a professional actor with the Guthrie Theater in Minneapolis. We took an interest in him after a dinner party where the Guthrie’s renowned artistic director, Joe Dowling, on hearing of our work, immediately suggested we interview Fuller. It seems that Fuller has the capacity to so fully learn the lines and movements of a role for which he is understudy that he can go onstage at the last moment with great success, despite not having had the benefit of learning and rehearsing it in the normal way.
Fuller is a consummate professional of the stage, having refined his techniques for learning roles over many years. He is often cast in a leading role; at other times, he may play several lesser characters in a play while also understudying the lead. How does he do it?
When he starts with a new script, Fuller puts it into a binder, goes through it, and highlights all of his lines. “I figure out how much I’ve got to learn. I try to estimate how much I can learn in a day, and then I try to start early enough to get that learned.”3 Highlighting his lines also makes them easy to find and gives him a sense of the construction, so this use of highlighting is rather different from what students do in class when they highlight merely for purposes of rereading. “You get the shape of the line, and how the back-and-forth works.” Fuller uses retrieval practice in various forms. First, he takes a blank sheet of paper and covers a page of the script. He draws it down, silently rendering the lines of the characters he’s playing opposite, because those lines cue his own, and the emotion in them is reflected one way or another by his own character. He keeps his own line covered and attempts to speak it aloud from memory. He checks his accuracy. If he gets the line wrong, he covers it up and speaks it again. When he has spoken it correctly, he reveals the next passage and goes on.
“Half of knowing your part is not just what to say, but knowing when to say it. I don’t have an exceptional brain for memorizing, but one of the keys I’ve found is, I need to try my best to say the line without looking at it. I need to have that struggle in order to make myself remember it.
“I’ll work like crazy. When I get to where it feels like diminishing returns, I’ll quit. Then I’ll come back the next day, and I won’t remember it. That’s where a lot of my friends will panic. I just have faith now that it’s in there, it’s going to come back a little bit better the next time. Then I’ll work on a new chunk, until I get to the end of the play.”
As he progresses through the script, he’s constantly moving from familiar pages and scenes into newer material, the play taking shape like threads added to a growing tapestry, each scene given meaning by those that came before and extending the story in turn. When he reaches the end, he practices in reverse order, moving from the less familiar last scene to practice the more familiar one that precedes it and then continuing on through the last scene again. Then he goes to the part preceding both of those scenes and practices through to the end. His practice continues reaching back in this way until he has come to the beginning of the play. This working backward and forward helps him stitch less familiar material to more familiar, deepening his mastery of the role as a whole.
Learning lines is visual (just as they are laid out in the script), but, he says, it’s also “an act of the body, an act of the muscles, so I’m trying to say the lines in character, get how it feels.” Fuller examines the language of the script, the textures of the words, and the figures of speech for how they reveal meaning. He works to discover the way the character carries himself, the way he moves across the stage, his facial expressions—all facets that reveal the underlying emotions that drive each scene. These forms of elaboration help him develop an emotional approach to the role and a deeper connection to the character.
He also notches up his retrieval practice. In place of the written script, he now speaks every line of the other actors in the play into a palm-sized digital recorder, voiced “in character” as best he can discern it. He tucks the recorder in his hand. His thumb knows where to find the controls. The thumb presses “play,” and Fuller hears the characters’ lines, then his cue; the thumb hits “pause,” and he speaks his line from memory. If in doubt about his accuracy, he checks the script, replays the passage if need be, speaks his lines, and then goes on with the scene.
When he’s understudying a role, before the director and cast have worked out the blocking (how the players move in relation to one another and the set), Fuller practices at home, imagining his living room as the stage and the way the blocking might be laid out. There, as he goes through scenes with his recorder, hearing others’ lines and speaking his own, he is moving through the imagined scene, adding physicality to the part, reacting to imaginary props. When the actor he’s understudying is in rehearsal, Fuller observes from behind the theater seats at the back of the hall, walking through the blocking himself as the actors rehearse on stage. He continues to practice later at home, adapting the imaginary stage within his living room to the now-established blocking.
Fuller’s learning process is a seamless blend of desirable difficulties: retrieval practice, spacing, interleaving, generation (of his character’s soul, carriage, motivations, and idiosyncrasies), and elaboration. Through these techniques, he learns the role and the many levels of meaning that make a performance come alive to himself and to his audience.
In 2013, John McPhee published a piece in the New Yorker about writer’s block. Age eighty-two at the time, McPhee offered his remarks from the vantage of a high perch, atop an illustrious career that has earned him many awards and acknowledgment as a pioneer of the craft of creative nonfiction. Writer’s block is the seemingly insurmountable barrier one must somehow clamber over if he is to have any hope of engaging his subject. Writing, like any art form, is an iterative process of creation and discovery. Many would-be writers fail to find their voices for the simple fact that, until they are clear about what they want to say, they cannot bring themselves to dive in. McPhee’s solution to this problem? He writes a letter to his mother. He tells her how miserable he feels, what hopes he’d had for the subject about which he wants to write (a bear), but that he has no idea how to go about it and, really, it seems that he’s not cut out to be a writer after all. He would like to put across the sheer size of the bear, and how utterly lazy it is, preferring to sleep fifteen hours a day, and so on. “And then you go back and delete the ‘Dear Mother’ and all the whimpering and whining, and just keep the bear.” McPhee’s first draft is an “awful blurting.” “Then you put the thing aside. You get in the car and drive home. On the way, your mind is still knitting at the words. You think of a better way to say something, a good phrase to correct a certain problem. Without the drafted version—if it did not exist—you obviously would not be thinking of ways to improve it. In short, you may actually be writing only two or three hours a day, but your mind, in one way or another, is working on it twenty-four hours a day—yes, while you sleep—but only if some sort of draft or earlier version exists. Until it exists, writing has not really begun.”4 This is the crux: Learning works the same way as McPhee’s “awful blurting.” Your grasp of unfamiliar material often starts out feeling clumsy and approximate. But once you engage the mind in trying to make sense of something new, the mind begins to “knit” at the problem on its own. You don’t engage the mind by reading a text over and over again or by passively watching PowerPoint slides. You engage it by making the effort to explain the material yourself, in your own words—connecting the facts, making it vivid, relating it to what you already know. Learning, like writing, is an act of engagement. Struggling with the puzzle stirs your creative juices, sets the mind to looking for parallels and metaphors from elsewhere in your experience, knowledge that can be transferred and applied here. It makes you hungry for the solution. And the solution, when you arrive at it, becomes more deeply embedded with your prior knowledge and abilities than anything pasted onto the surface of your brain by PowerPoint.
So take a page from McPhee: when you want to master something new, delete the whimpering and go wrestle the bear.
In Chapter 2 we tell how the Mayo Clinic neurosurgeon Mike Ebersold uses the habit of reflection to improve his skills in the operating room. Reflection involves retrieval (What did I do? How did it work?) and generation (How could I do it better next time?), invoking imagery and mental rehearsal as well (What if I take a smaller bite with the needle?). It was this habit of reflection that brought him to devise a surgical solution for the repair of a delicate sinus structure in the back of the skull that cannot be tied off because the structure is somewhat flat and tears when you snug the suture.
Vince Dooley, Georgia Bulldogs football coach (Chapter 3), helped his players use reflection and mental rehearsal to learn their playbooks and their adjustments for next Saturday’s game. The Minneapolis cop David Garman (Chapter 5) uses reflection to improve his undercover strategies. The power of reflection as a learning technique is apparent throughout the personal memoir Highest Duty, by Captain Chesley Sullenberger. “Sully” is the pilot who successfully and miraculously ditched US Airways Flight 1549 on the Hudson River in 2009. Time and again, in reading his autobiography, we see how he refined his understanding of flight and the control of his aircraft through training, personal experience, and the close observation of others. The process started from his earliest days at the stick of a single-engine crop duster, continued to his jet fighter days, his time investigating commercial airline disasters, and his granular analysis of the few available examples of the ditching of commercial aircraft, where he paid particular attention to the lessons for pitch, speed, and level wings. The evolution of Captain Sullenberger shows us that the habit of reflection is more than simply taking stock of a personal experience or the observed experiences of others. At its most powerful this habit involves engagement of the mind through generation, visualization, and mental rehearsal.
When we met the pianist Thelma Hunter, she was learning four new works for an upcoming concert performance: pieces by Mozart, Faure, Rachmaninoff, and William Bolcom. Hunter, who is eighty-eight, won her first prize as a pianist at age five in New York and has been performing ever since. She is not a prodigy, she insists, nor even particularly renowned, but she is accomplished. In addition to a busy life raising six kids with her husband, Sam, a heart surgeon, Hunter has enjoyed a long life of learning, teaching, and performing at the piano, and she is still in the game, sought after and bent to her life’s pleasure at the keyboard.
Giving new learning multiple layers of meaning has been central to Hunter’s methods and illustrates the way elaboration strengthens learning and memory. When she studies a new score, she learns it physically in the fingering, aurally in the sound, visually in the notes on the score, and intellectually in the way she coaches herself through transitions.
Hunter has made some concessions to age. She never used to warm up before playing, but now she does. “My stamina is not as great as it used to be. My reach is not as big. Now, if I memorize something, I have to think about it. I never used to have to do that, I just worked through all the aspects of it and the memorizing came.”5 She visualizes the score and makes mental marginalia. “When I’m practicing, sometimes I say it out loud, ‘Up an octave, at this point,’ but in my mind’s eye I visualize the place on the sheet music, as well.” In comments that resonate with John McPhee’s observations about writing, Hunter says that at the point where a piece is almost memorized, “I’ll be driving, and I can think about the whole piece, which I do. The shape of it, as though I were a conductor, thinking, ‘Oh, that passage makes more sense if I speed it up. I have to practice that to get it faster.’ Those are the large things that I can think about away from the piano.” Hunter’s practice regimen is daily, working through new pieces, slowing down to parse the difficult passages, and then, because she now often performs with a cellist and violinist, the ensemble works through the pieces together to synchronize their individual interpretations.
In Chapter 7 we describe Anders Ericsson’s research into how experts, through thousands of hours of solo, deliberate practice, build libraries of mental models that they can deploy to address a wide universe of situations they encounter in their area of expertise. Hunter describes experiences that would seem to manifest Ericsson’s theory. At times she must sit at the keyboard and devise a fingering plan for playing a difficult passage. Oddly, she says, after having been away from the piece for a week, she will sit down and play it through, using a fingering pattern that she had not planned but feels entirely natural to her and familiar. It’s a paradox, though perhaps not entirely surprising. She credits her subconscious, drawing from her long years of playing, with finding a more fluent solution than what she has devised by puzzling it out at the keyboard. But perhaps it has been the effort at the keys, like McPhee wrestling his bear, that has set her mind to sorting through the closets of her memory for something a little more elegant and natural to fit the occasion.
Tips for Teachers
Here again we are leery of being too prescriptive. Every teacher must find what’s right in his or her classroom. Yet specifics can be helpful. So here are some basic strategies that in our judgment will go a long way toward helping students become stronger learners in the classroom. Brief descriptions follow of what some teachers are already doing along these lines. Between the recommendations and the examples, we hope you will find practical ideas you can adapt and put to work.
Explain to Students How Learning Works
Students labor under many myths and illusions about learning that cause them to make some unfortunate choices about intellectual risk taking and about when and how to study. It’s the proper role of the teacher to explain what empirical studies have discovered about how people learn, so the student can better manage his or her own education.
In particular, students must be helped to understand such fundamental ideas as these:
• Some kinds of difficulties during learning help to make the learning stronger and better remembered.
• When learning is easy, it is often superficial and soon forgotten.
• Not all of our intellectual abilities are hardwired. In fact, when learning is effortful, it changes the brain, making new connections and increasing intellectual ability.
• You learn better when you wrestle with new problems before being shown the solution, rather than the other way around.
• To achieve excellence in any sphere, you must strive to surpass your current level of ability.
• Striving, by its nature, often results in setbacks, and setbacks are often what provide the essential information needed to adjust strategies to achieve mastery.
These topics, woven throughout the book, are discussed in depth in Chapters 4 and 7.
Teach Students How to Study
Students generally are not taught how to study, and when they are, they often get the wrong advice. As a result, they gravitate to activities that are far from optimal, like rereading, massed practice, and cramming.
At the beginning of this chapter we present effective study strategies. Students will benefit from teachers who help them understand these strategies and stick with them long enough to experience their benefits, which may initially appear doubtful.
Create Desirable Difficulties in the Classroom
Where practical, use frequent quizzing to help students consolidate learning and interrupt the process of forgetting. Make the ground rules acceptable to your students and yourself. Students find quizzing more acceptable when it is predictable and the stakes for any individual quiz are low. Teachers find quizzing more acceptable when it is simple, quick, and does not lead to negotiating makeup quizzes. (For one example, consider the way Kathleen McDermott, whose work we describe below, uses daily quizzing in her university class on human learning and memory.) Create study tools that incorporate retrieval practice, generation, and elaboration. These might be exercises that require students to wrestle with trying to solve a new kind of problem before coming to the class where the solution is taught; practice tests that students can download and use to review material and to calibrate their judgments of what they know and don’t know; writing exercises that require students to reflect on past lesson material and relate it to other knowledge or other aspects of their lives; exercises that require students to generate short statements that summarize the key ideas of recent material covered in a text or lecture.
Make quizzing and practice exercises count toward the course grade, even if for very low stakes. Students in classes where practice exercises carry consequences for the course grade learn better than those in classes where the exercises are the same but carry no consequences.
Design quizzing and exercises to reach back to concepts and learning covered earlier in the term, so that retrieval practice continues and the learning is cumulative, helping students to construct more complex mental models, strengthen conceptual learning, and develop deeper understanding of the relationships between ideas or systems. (For an example, read in Chapter 2 how Andy Sobel uses cumulative low-stakes quizzing in his university-level course in political economics.) Space, interleave, and vary topics and problems covered in class so that students are frequently shifting gears as they have to “reload” what they already know about each topic in order to figure out how the new material relates or differs.
Help your students understand the ways you have incorporated desirable difficulties into your lessons, and why. Be up front about some of the frustrations and difficulties this kind of learning entails and explain why it’s worth persisting. Consider having them read the profile earlier in this chapter of the medical student Michael Young, who vividly describes the difficulties and ultimate benefits of using these strategies.
Mary Pat Wenderoth, Biology Professor, University of Washington
Mary Pat Wenderoth introduces desirable difficulties in her classes to help students master their coursework. She also works at helping students learn how to be effective at managing their own learning—to be the capable student within the professional that they envision becoming. Along that path she tackles yet another challenge, helping students learn to judge where their grasp of course material stands on Bloom’s taxonomy of learning, and how to rise to the levels of synthesis and evaluation.
Bloom’s taxonomy classifies cognitive learning on six levels. It was developed in 1956 by a committee of educators chaired by psychologist Benjamin Bloom. The six levels range from gaining knowledge (the most fundamental level) to developing comprehension of the underlying facts and ideas, being able to apply learning to solve problems, being able to analyze ideas and relationships so as to make inferences, being able to synthesize knowledge and ideas in new ways, and, at the most sophisticated level, being able to use learning to evaluate opinions and ideas and make judgments based on evidence and objective criteria.
Here are some of the main techniques Wenderoth uses.
Transparency. At the outset, Wenderoth teaches her students about the testing effect, the principle of desirable difficulties, and the perils of “illusions of knowing.” She promises to make her instructional philosophy transparent and to model these principles in class. As she explained to us recently, “The whole idea of the testing effect is that you learn more by testing yourself than by rereading. Well, it’s very hard to get students to do that because they’ve been trained for so long to keep reading and reading the book.”6 I can’t tell you how many times the students come to me and they show me their textbook and it’s highlighted in four different colors. I say to them, “I can tell you have done a lot of work and that you really want to succeed in this class because you have blue and yellow and orange and green highlighter on your book.” And then I have to try to tell them that any more time spent on this after the first time was a waste. They’re, like, “How is that possible?” I say, “What you have to do is, you read a little bit and then you have to test yourself,” but they don’t quite know how to do that.
So I model it in class for them. Every five minutes or so I throw out a question on the material we just talked about, and I can see them start to look through their notes. I say, “Stop. Do not look at your notes. Just take a minute to think about it yourself.” I tell them our brains are like a forest, and your memory is in there somewhere. You’re here, and the memory is over there. The more times you make a path to that memory, the better the path is, so that the next time you need the memory, it’s going to be easier to find it. But as soon as you get your notes out, you have short-circuited the path. You are not exploring for the path anymore, someone has told you the way.
At other times, Wenderoth will pose a question to the class and ask them to think about it. She has students write three possible answers on the whiteboard up front and then vote on which answer they think is correct by raising the number of fingers that corresponds with the answer on the board. She’ll instruct students to find somebody with fingers “different from yours and talk to them and figure out who has the correct answer.”
Wenderoth gives her students a new way to think about learning, and she gives them a new vocabulary for describing setbacks. When students trip over an exam question, they’ll commonly accuse the test of containing trick questions. When the student blames the test, she says, it’s not a good meeting ground for solving the problem. But now, students come to see her after a disappointing exam and say, “I have the illusion of knowing. How do I get better?” That’s a problem Wenderoth can help with.
Testing groups. Wenderoth has transformed class “study groups” into “testing groups.” In a study group, the person who knows the most talks and the others listen. The emphasis is on memorizing things. However, in a testing group, they all wrestle with a question together, without opening the textbook. “Everybody has bits of information, and you talk with your colleagues and figure it out.” The emphasis is on exploration and understanding.
Wenderoth will ask students in a testing group what ideas they don’t feel really clear on. Then she’ll send one student to the whiteboard to try to explain the concept. As the student struggles, perhaps putting up the pieces of the answer she knows, the rest of the group are instructed to test her by asking questions whose answers will lead her to the larger concept. Throughout, all textbooks remain closed.
Free recall. Wenderoth assigns her students to spend ten minutes at the end of each day sitting with a blank piece of paper on which to write everything they can remember from class. They must sit for ten minutes. She warns that it will be uncomfortable, they will run out of ideas after two minutes, but they must stick it out. At the end of ten minutes, they’re to go to their class notes and find out what they remembered and what they forgot, and to focus on the material they forgot. What they glean from this exercise guides their notes and questions for the next class. Wenderoth finds that the free recall exercise helps students pull learning forward and develop a more complex understanding of how the material interrelates.
Summary sheets. Every Monday, Wenderoth’s students are required to turn in a single sheet of certain dimensions on which they have illustrated the prior week’s material in drawings annotated with key ideas, arrows, and graphs. She’s teaching physiology, which is about how things work, so the summaries take on the form of large cartoons dense with callouts, blowups, directional arrows, and the like. The sheets help her students synthesize a week’s information, thinking through how systems are connected: “This is causing this, which causes this, which feeds back on those. We use a lot of arrows in physiology. The students can work with each other, I don’t care. The sheet they bring in just has to be their own.” Learning paragraphs. From time to time, on a Friday, if she doesn’t feel she’s overburdening them, Wenderoth will assign students to write low-stakes “learning paragraphs” for which she poses a question and asks students to prepare a five- or six-sentence response. A question might be “How is the GI tract like the respiratory system?” Or “You just got your tests back; what would you do differently next time?” The point is to stimulate retrieval and reflection and to capture a week’s learning before it is lost to the countless other concerns and diversions of college life. “What I found over the years is, if I don’t do anything before the test, they don’t do anything until the day before the test.” The learning paragraphs also give her science majors practice in writing a passage of clear prose. She reads through the responses and makes a point to comment on them in class so that students know they’re being read.
Bloom’s taxonomy of learning. To remove some of the abstraction from Bloom’s taxonomy, Wenderoth has translated her class material into the different levels of the taxonomy on an answer key to her tests. That is, for any given question, she provides a different answer for each level of the taxonomy: one that reflects learning at the level of knowledge, a more thorough answer that reflects understanding, a yet more complex answer that reflects analysis, and so on. When students get their tests back, they also receive the answer key and are asked to identify where their answers fell on the taxonomy and to think about what they need to know in order to respond at a higher level of learning.
Closing the achievement gap in the sciences. Wenderoth and her colleagues have experimented with class structure and the principles of active learning to help close the achievement gap in the sciences. Poorly prepared students seldom survive entry-level science courses. As a result, even students whose interests and aptitudes might lead them to successful science careers never get through the door. For whatever reason, these students do not have a history from high school or family life of learning how to succeed in these highly challenging academic settings.
“For most of us who have found our way in the sciences,” Wenderoth says, “any time we fell, there was somebody around to help us up, or to say, ‘This is how you get up.’ You were taught that when things don’t go well, you keep working anyway. You persevere.”
In their experiments, Wenderoth and her colleagues have compared the results of “low-structure” classes (traditional lecturing and high-stakes midterm and final exams) with “high-structure” classes (daily and weekly low-stakes exercises to provide constant practice in the analytical skills necessary to do well on exams). They also teach students the importance of having a “growth mindset” (see the work of Carol Dweck, discussed in Chapter 7)—that is, that learning is hard work and that struggle increases intellectual abilities.
The results? High-structure classes in a gateway biology course significantly reduced student failure rates compared to low-structure classes—narrowing the gap between poorly prepared students and their better prepared peers while at the same time showing exam results at higher levels on Bloom’s taxonomy. Moreover, it’s not just whether the student completes the practice exercises that matters. In the classes where exercises count toward the course grade, even at very low stakes, students achieve higher success over the course of the term compared to students in classes where the exercises are the same but carry no consequences for the grade.
“We talk to the students about how these are the habits of mind,” Wenderoth says. “This is the discipline that you have to have in order to succeed in the sciences. They’ve never thought about that, that every discipline has a culture. We teach them to think like the professionals they want to become. And when they fall, we show them how to get up again.”7
Michael D. Matthews, Psychology Professor, U.S. Military Academy at West Point
The pedagogical philosophy at West Point is founded on an instructional system called the Thayer method, developed almost two hundred years ago by an early superintendent of the academy named Sylvanus Thayer. The method provides very specific learning objectives for every course, puts the responsibility for meeting those objectives on the student, and incorporates quizzing and recitation in every class meeting.
Students’ grades at the academy rest on three pillars of training: academic, military, and physical. Mike Matthews, a professor of engineering psychology at the academy, says the load on students is enormous, greater than the hours available to them. In order to survive at the academy, West Point cadets must develop an ability to zero in on what’s essential and let the rest fall by the wayside. “This is about having very high expectations across multiple dimensions and keeping them real busy,” Matthews, says. In fact, as stunning as it sounds, Matthews will tell a student, “If you’ve read every word of this chapter, you’re not being very efficient.” The point is not to “slide your eyes over the words.” You start with questions, and you read for answers.8 There’s little or no lecturing in Matthews’s courses. Class opens with a quiz on the learning objectives from the assigned reading. From there, on many days, students “take to the boards.” The classrooms have slate on all four walls, and a group of students are sent to each blackboard to collaborate on answering a question given by the professor. These are higher-order questions than are given in the daily quiz, requiring the students to integrate ideas from the reading and apply them at a conceptual level. It’s a form of retrieval practice, generation, and peer instruction. One student is selected from each group to give a recitation to the class explaining how the group has answered the question, and then the group’s work is critiqued. All class meetings focus on constructs, not specific facts, and on the days the students do not take to the boards, they are engaged in other forms of exercise, demonstration, or group work aimed at understanding and articulating the larger concepts underlying the matter at hand.
Clear learning objectives prior to each class, coupled with daily quizzing and active problem solving with feedback, keep students focused, awake, and working hard.
One of the most important skills taught at West Point is something learned outside the classroom: how to shoot an azimuth. It’s a skill used for keeping your bearings in unfamiliar territory. You climb a tree or a height of land and sight a distant landmark in the direction you’re headed. Compass in hand, you note how many degrees your landmark lies off of due north. Then you descend into the bush and keep working your way in that direction. Periodically, you pause to shoot an azimuth and make sure you’re on course. Quizzing is a way of shooting an azimuth in the classroom: are you gaining the mastery you need to get where you’re trying to go?
Matthews has had the privilege of seeing two of his students win Rhodes Scholarships. The most recent was Cadet Kiley Hunkler (now Second Lieutenant Hunkler). Hunkler will be spending the next two years at Oxford University, and then matriculating at Johns Hopkins Medical School. It was Hunkler who spoke to us of shooting an azimuth. “Everything at the academy is about self-responsibility, taking ownership for finding your own way to the objective,” she said.9 The Medical College Admission Test, for example, encompasses four major course blocks: reading, chemistry, physiology, and writing. For each of these blocks, Hunkler created the learning objectives in her head that she deemed most important and then set out to answer them as she studied. “I took a practice test every three days, saw what I got wrong, and adjusted.” Shooting her azimuth. “A lot of students get hung up studying for months, trying to memorize everything, but for me it was more about understanding the concepts. So my azimuth check would be, Okay, what is this question asking, what’s the broader theme here, and does that match up with what I’ve outlined for this section.” One of this book’s authors (Roediger) attended Riverside Military Academy in Gainesville, Georgia, for high school. Riverside used a form of the Thayer method, with students having daily quizzes, problem sets, or assignments to be completed in class. The range of ability of these younger cadets was much more varied than at the elite US Military Academy at West Point, but the Thayer method worked well. In fact, such methods that include daily participation are especially likely to help students who are not prone to work hard on their own outside of class. The Thayer method is a strong encouragement for them to keep at it, and echoes what Mary Pat Wenderoth (above) has found in her empirical studies: that high-structure classes help students who lack a history of using effective learning techniques and habits to develop them and succeed in rigorous settings.
Kathleen McDermott, Psychology Professor, Washington University at St. Louis
Kathleen McDermott administers daily low-stakes quizzes in a university course on human learning and memory. It’s a class of twenty-five students that meets twice a week for fourteen weeks, minus midterms and a final exam. She gives a four-item quiz in the last three to five minutes of every class. The questions hit the high points of the lecture, the readings, or both. If students have understood the material, they will get all four answers right, but they’ll have to think in order to do it. Anything covered in the course to date is fair game for a quiz, and she will sometimes draw from past material that she feels the students haven’t fully grasped and need to review.
McDermott sets the ground rules very clearly at the start of the term. She lays out the research on learning and the testing effect and explains why the quizzes are helpful, even if they don’t feel helpful. Students are allowed to drop four quizzes across the semester. In exchange, absences need not be justified, and no missed quizzes will be made up.
Students initially are not happy about the quiz regime, and in the first few weeks of the term McDermott will get email from students explaining why they had a legitimate excuse for an absence and should be allowed to make up a missed quiz. She reiterates the terms: four free absences, no makeups.
McDermott says the quizzes provide an incentive for students to attend class and give students a way to contribute to their grade on a daily basis if they answer four out of four questions correctly. By the end of the semester, her students say that the quizzes have helped them keep up with the course and discover when they are getting off track and need to bone up.
“The key with quizzes is to establish very clear ground rules for the student, and make them manageable for the professor,” McDermott says. “As a student, you’re either there and you take it, or you’re not. For the professor, no hassling over makeup tests.”10
The quizzes in totality count for 20 percent of a student’s grade in the course. In addition, McDermott gives two midterm exams and a final. The last two exams are cumulative. Having cumulative exams reinforces learning by requiring students to engage in spaced review.
Columbia, Illinois, Public School District
As recounted in Chapter 2, we have worked with teachers in a middle school in Columbia, Illinois, to test the effects of integrating low-stakes quizzing into the curriculum. Regular quizzing and other forms of retrieval practice have been adopted by teachers in the school who were a part of the research study and by others who were not but who observed the beneficial results. The initial research project has since been extended into history and science classes in the district’s high school, where frequent retrieval practice is being used both to bolster learning and to help teachers focus instruction on areas where student understanding and performance need to be improved.
The Illinois State Board of Education has adopted new math and English language arts standards for K–12 education in line with the Common Core State Standards Initiative led by the National Governors Association and endorsed by the nation’s secretary of education. Common Core establishes standards for college and career readiness that students should be able to meet on graduation from high school. The Columbia School District, like others, is redesigning its curriculum and its tests to be more rigorous and to engage students in more writing and analysis work, with the aim of promoting the higher-level skills of conceptual understanding, reasoning, and problem solving that will enable students to meet the standards established by the state. As one example of this overhaul, the sciences curriculum is being vertically aligned so that students are reexposed to a subject at various stages of their school careers. The result is more spaced and interleaved instruction. In physical sciences, for instance, middle school students may learn to identify the six basic machines (inclined plane, wedge, screw, lever, wheel and axle, and pulley) and how they work, and then may return to these concepts in subsequent grades, delving into the underlying physics and how these basic tools can be combined and applied to solve different problems.
Tips for Trainers
Here are some ways trainers are using the same principles as those who teach in schools, in a variety of less structured and nonclassroom settings.
Licensed professionals in many fields must earn continuing education credits to keep their skills current and maintain their licenses. As the pediatric neurologist Doug Larsen describes in Chapter 3, this kind of training for doctors is typically compressed into a weekend symposium, out of respect for participants’ busy schedules, set at a hotel or resort, and structured around meals and PowerPoint lectures. In other words, the strategies of retrieval practice, spacing, and interleaving are nowhere to be seen. Participants are lucky to retain much of what they learn.
If you see yourself in this scenario, there are a few things you might consider doing. One, get a copy of the presentation materials and use them to quiz yourself on the key ideas, much as Nathaniel Fuller quizzes himself on the arc of a play, his lines, the many layers of character. Two, schedule follow-up emails to appear in your inbox every month or so with questions that require you to retrieve the critical learning you gained from the seminar. Three, contact your professional association and ask them to consider revamping their approach to training along the lines outlined in this book.
The testing effect forms the basis of a new commercial training platform called Qstream that helps trainers send learners periodic quizzes via their mobile devices to strengthen learning through spaced retrieval practice. Similarly, an emerging platform called Osmosis uses mobile and Web based software to provide learners access to thousands of crowdsourced practice questions and explanations. Osmosis combines the testing effect, spacing, and social networking to facilitate what its developers call “student-driven social learning.” Qstream (qstream.com) and Osmosis (osmose-it.com) suggest interesting possibilities for redesigning in-service training for professionals. Many other companies are developing similar programs.
Kathy Maixner, Business Coach
The Maixner Group is a consulting shop based in Portland, Oregon, that helps companies identify growth strategies and improve their sales tactics. Kathy Maixner fries big fish and little. One of the big fish added $21 million to its annual revenue as a result of hooking up with Maixner. One of the small ones, Inner Gate Acupuncture (profiled at the close of this chapter), learned how to establish a solid business management footing under a clinical practice whose growth was outpacing its control systems.
We’re interested in Maixner because the coaching techniques she has developed over her career line up so well with the learning principles described in this book. In short, Maixner sees her role as helping the client dig past the symptoms of a problem to discover its root causes, and then to generate possible solutions and play out the implications of different strategies before committing to them.
Maixner told us: “If you hand people the solution, they don’t need to explore how you got to that solution. If they generate the solution, then they’re the ones who are traveling down that road. Should they go left or right? We discuss the options.”11
Maixner’s years of experience working with clients in many different fields helps her see around corners, where the hazards lie. She often uses role-playing to simulate problems, getting her clients to generate solutions, try them out, get feedback, and practice what works. In other words, she introduces the difficulties that make the learning stronger and more accurately reflect what the client will encounter out in the marketplace.
Corporate sales training can be complicated. Typically, it’s about corporate culture, beliefs and behavior, and learning to promote and protect the brand. It’s also technical, learning the features and advantages of the products. And it’s partly strategic, learning about the target market and how to generate prospects and make sales. At Farmers Insurance, whose principal sales force is a cadre of about fourteen thousand exclusive independent agents, training must also equip the company’s reps to become successful as entrepreneurs, building and managing their own agency.
Farmers sells property and casualty policies and investment products like annuities and mutual funds to the tune of about $20 billion a year. Describing the full scope of their training could fill volumes, but we’ll focus on the way Farmers brings new agents on board, training them in the four areas of sales, marketing systems, business planning, and advocacy of the brand. The company’s new-agent training is an excellent example of interleaving the learning and practice of different but related topics so that each adds meaning to the other, broadening and deepening competency.
The company recruits upward of two thousand new agents annually. Many leave traditional jobs elsewhere, drawn to the rewards of running their own business and the opportunity to represent an established product line. Newly appointed agents arrive at one of two training campuses for an intensive weeklong program of learning exercises that spiral upward in sophistication.
At the start, participants are given a pile of magazines, scissors, and marking pens with which to illustrate on posterboard what being a successful Farmers agent would look like to them personally, five years down the road. For some, the poster shows fancy houses and cars. For others, kids are being sent to college and aging parents are being cared for. The point is simple: if your definition of success requires, say, $250,000 a year in revenues and twenty-five hundred policies in force, we can help you work backward to set the metrics for where you need to be in four years, in three years, and even three months from now. The image on the poster shows where you’re headed, the metrics are your road map, and the skills that are learned over the coming days and months are the tools that will enable you to make the journey.
From here, the week is not so much about teaching from the top down—there are no PowerPoint lectures as such—but about learning from the bottom up, as in: “What knowledge and skills do I need in order to succeed?”
The learning unfolds through a series of exercises that cycle through the principal topics of sales, marketing systems, business planning, and advocacy of the company’s values and its brands—returning time and again to each, requiring that participants recall what they have learned earlier and apply it in a new, enlarged context.
For example, when participants first arrive, they’re assigned to a red, blue, or green group. The red group is instructed to go meet people in the room. The blue group is instructed to go learn three things about somebody in the room. The green group is instructed to ask another member of the class about his or her family, prior occupation, favorite forms of recreation, and what he or she enjoys most. When the class reconvenes, they share what they have learned about others, and it is quickly evident that the green group, which had a structure for talking to others, learned much more than did their peers.
When talking about sales later in the week the question comes up, what’s an effective way to learn about a prospective customer? Somebody will recall the initial get-acquainted exercise that proved so fruitful: asking about one’s family, occupation, recreation, and enjoyment. That icebreaker now morphs into a handy tool for getting to know a prospective client and it gets an acronym: FORE.
Throughout the week the four principal training topics are repeatedly touched on, a point is made, and the exercises shift to related questions. In one session, participants brainstorm what kinds of marketing and development strategies might generate the flow of leads they need in order to meet sales targets. An effective sales and marketing system has a structure called 5-4-3-2-1. Five new business marketing initiatives every month, four cross-marketing and four retention programs in place, three appointments scheduled every day, two appointments kept (prospects often have to reschedule), one new customer sold on average two policies per sale. At twenty-two working days a month, that’s about five hundred new policies in a year, making twenty-five hundred over the five-year horizon of the agent’s vision.
Practice is a central learning strategy. For example, they practice how to respond to a sales lead. Trying to sell the company’s products is how they learn about selling, but it’s also how they learn about the products they’re selling—not by sitting in front of PowerPoint slides gazing at long lists of product features. You be the agent, I’ll be the customer. Then we’ll switch.
Interwoven with these exercises are others that help the new agents learn about the company’s history, what it stands for, and the value of its products in people’s lives, for instance through stories of how it has helped people recover from catastrophes like Hurricane Katrina.
Given the emphasis on marketing and the limited resources new agents have to invest, how does an agent determine which strategies will pay? The question goes out: What’s a reasonable return to expect from a direct mail campaign? The agents mull it over and hazard guesses. Usually, one or more of the agents will have had direct-mail marketing experience and offer the sobering answer: returns are closer to 1 percent than the 50 percent many had guessed.
Once you turn up a lead, how do you discover needs he or she has that the company’s products can meet? They return to the handy acronym FORE. Now, the habit of asking about one’s family, occupation, recreation, and enjoyment becomes something even more potent than a tool for getting acquainted. It provides an opening into four of the most important realms of a prospect’s life where insurance and financial products can help that person protect his or her assets and achieve his or her financial goals. At each pivot from one subject back to another, understanding deepens, and new skills take form.
In this way, through generation, spaced practice, and interleaving of the essential core curriculum, with an eye always to the five-year vision and road map, new agents learn what they need to do, and how, in order to thrive as a part of the Farmers Insurance family.
If you don’t expect innovations in training to spring from your local service garage, Jiffy Lube may surprise you. An integrated suite of educational courses under the felicitous name Jiffy Lube University is helping the company’s franchisees win customers, reduce employee turnover, broaden their service offerings, and boost sales.
Jiffy Lube is a network of more than two thousand service centers in the United States and Canada that provide oil changes, tire rotation, and other automotive services. Although the company is a subsidiary of Shell Oil Company, every outlet is owned and operated by an independent franchisee, who hires employees to serve customers.
The rapid-oil-change business, like most others, has had to adjust to changes in the marketplace and advances in technology. Synthetic lubricants have made oil changes less frequent, and because cars have become more complicated, garage employees need higher levels of training to understand diagnostic codes and provide appropriate services.
No employee may work on a customer’s car until he or she has been certified as proficient. For this, they enter Jiffy Lube University, a Web-based learning platform. Certification starts with interactive e-learning, with frequent quizzing and feedback to learn what a particular job entails and how it’s to be performed. When employees score 80 percent or better on an exam, they are eligible to begin training on the job, practicing new skills by following a written guide that breaks each service activity into its component steps. The steps may number as many as thirty and are performed as a part of a team, often involving call and response (for example, between a technician working from the top side of an engine and another underneath). A supervisor coaches the employee and rates his or her performance on each step. When the technician demonstrates mastery, certification is recorded in his or her permanent file, signed by the supervisor. Technicians must recertify every two years to keep their mastery up to snuff and adapt to operational and technical changes. Higher-level jobs for advanced services like brake repair or running engine diagnostics are trained in the same manner.
The e-learning and on-the-job training are active learning strategies that incorporate various forms of quizzing, feedback, and spaced and interleaved practice. All progress is displayed by computer on a virtual “dashboard” that provides an individualized learning plan, enabling an employee to track his or her performance, focus on skills that need to be raised, and monitor his or her progress against the company’s completion schedule. Jiffy Lube employees are typically eighteen to twenty-five years old and filling their first job. As a technician is certified in one job, he or she begins training in another, until he or she has trained in all store positions, including management.
Ken Barber, Jiffy Lube International’s manager of learning and development, says training has to be engaging in order to hold employees’ attention. At the time we spoke, Barber was putting the finishing touches on a computer-based simulation game for company managers called “A Day in the Life of a Store Manager.” The service center manager is confronted with various challenges and is required to select among a range of possible strategies for resolving them. The manager’s choices determine how the game unfolds, providing feedback and the opportunity to strive for better outcomes, sharpening decision-making skill.
In the six years since Jiffy Lube University was launched, it has received many accolades from the training profession and earned accreditation by the American Council on Education. Employees who progress through training in all job certifications can enroll at a postsecondary institution with seven hours of college credit under their belts. Since the program’s beginning, employee turnover has dropped and customer satisfaction has increased.
“For most employees of a Jiffy Lube franchisee, this is a way into the workforce, and the training curriculum helps them to continue to grow and expand their knowledge,” Barber says. “It helps them find a path to success.”12
Andersen Windows and Doors
At Andersen Windows and Doors, a culture of continuous improvement turns learning on its head: the production workers teach the managers how to make the plant more efficient.
This story is a little different from the others in this chapter in two respects. It’s partly about creating a learning culture in the workplace, and partly about empowering employees to use what they learn to change the workplace. By encouraging employees to identify problems on the job and propose improvements, the company is supporting one of the most powerful learning techniques we have discussed, wrestling to solve a problem.
A good place to focus is on the company’s division called Renewal by Andersen, which produces replacement windows of all types and sizes: double-hung, casement, gliding, picture windows, and specialty windows in nontraditional shapes.
At Renewal by Andersen’s facility in Cottage Grove, Minnesota, their double-hung production line employs thirty-six people during an eight-hour shift that is divided into three work cells, one for sash fabrication, another for frame fabrication, and one for final assembly. Each work cell has four work stations and is led by a crew leader who is responsible for safety, quality, cost, and delivery within that cell. Workers change jobs every two hours to minimize repetitive stress injuries and broaden cross-training. Like interleaving the practice of two or more different but related topics, frequent switching between jobs builds an understanding of the integrated process for which their unit is responsible and equips workers to respond more broadly to unexpected events that arise.
It probably won’t surprise you to learn that every job is performed to a written standard that describes each step and the way it is to be taken. The written standard is essential for uniformity of product and quality. Without it, plant manager Rick Wynveen says, four different people will perform the job in four different ways, and produce four different versions of the product.
When a new employee comes on board, he or she is trained following an instructional sequence of practice and feedback that Wynveen calls “tell—show—do—review.” The new worker is paired with an experienced worker, practice is on-the-job, and feedback brings learning and performance in line with the written standard.
How do the workers train the managers? When a worker has an idea for improving productivity and management endorses it, for instance revamping the way parts arrive at a work station to make life easier for the worker and assembly faster, the worker who offered it takes leave from production to help implement the new standard. “Everyone’s idea is valuable,” Wynveen told us, “whether you’re an engineer, a maintenance technician, or a production worker.”13 Likewise, when one of the production line teams comes up short in meeting its targets, it’s the workers who are asked to identify the problem and redesign the production process to solve it.
The instructional role of employees is most dramatically illustrated in what Wynveen calls a Kaizen event. Kaizen is a Japanese term for improvement. It has been central to Toyota Motor Company’s success and has been adopted by many other companies to help create a culture of continuous improvement.
When Wynveen wanted to effect a major increase in the productivity of the plant’s double-hung window line, he recruited a design team to engage in a Kaizen event. The team consisted of an engineer, a maintenance technician, a crew leader from the production line, and five production workers. They were given the stretch goals of reducing the line’s space requirement by 40 percent and doubling production. (Stretch goals are ones that cannot be reached through incremental improvement but require significant restructuring of methods.) The team met in a conference room eight hours a day for a week, in effect teaching each other the elements, capacities, and constraints of the production process and asking themselves how to make it smaller and better. The following week they came back to Wynveen saying “Here’s what we think we can do.” Wynveen took their plan to each of the twelve work stations on the line with a simple question: What changes are needed to make this plan work? Production workers and their crew leaders put their heads together and redesigned the components to fit the new plan. The line was disassembled and rebuilt in two halves, over two weekends, restarted, and fine-tuned over subsequent months, a process that generated yet an additional two hundred improvements suggested by production workers: a learning process of testing, feedback, and correction.
The result? After five months, the plant had met Wynveen’s stretch goals and cut costs in half. During the conversion and shakedown, the production teams never missed a shipment and never had a quality issue. The principle of engagement—actively seeking the ideas of employees from all levels of the plant—is central to the company’s culture of continuous improvement. “Engagement is a management style of trust and a willingness to talk,” Wynveen says. The production employees learned how to refine the design as they worked, and the company provided a way for suggestions to be heard and for employees to participate in their implementation.
A learning culture places the responsibility for learning with the employees and empowers them to change the system. Problems become information rather than failures. And learning by solving the problems (generation) and by teaching others (elaboration) becomes an engine for continuous improvement of performance by individuals and by the production line that they compose.
Inner Gate Acupuncture
There are times when getting learning and teaching right can shape the trajectory of an entire life. Consider Erik Isaacman, a thirty-something husband, father of two, and passionate practitioner of traditional Chinese medicine: acupuncture, massage, and herbal therapy. We close this chapter with the story of a turning point in Erik’s fledgling practice, Inner Gate Acupuncture in Portland, Oregon. It’s the story of a clinic that was succeeding in its therapeutic mission but struggling as a business.
Erik and his business partner, Oliver Leonetti, opened Inner Gate in 2005, after earning graduate degrees in traditional Chinese medicine. Through networking and creative marketing, they began to build a stream of clients. Portland is fertile territory for alternative therapies. The business grew, and so did expenses: They leased larger space, hired an assistant to schedule appointments and manage the office, brought in a third clinician, and hired a back-office employee. “We were growing 35 to 50 percent every year,” Erik recalled when we spoke. “The growth covered up a lot that was missing: We didn’t have the systems in place to manage costs. We didn’t have clear goals or a management hierarchy. It was fast becoming clear that we had no idea how to run a business.”14 One of Erik’s patients is the Oregon business coach Kathy Maixner. Maixner offered to help. “Unmanaged growth is scary,” she told us. “You jump ahead, then you flounder.” She asked a lot of questions that quickly focused Erik’s and Oliver’s thinking on critical gaps in their systems. The three then set out a schedule of frequent coaching sessions, between which Erik and Oliver generated elements of the missing infrastructure: operating manual, job descriptions, financial goals, metrics for measuring the performance of their clinicians.
Every business serves two masters, its customer and its bottom line. “Our clinicians need to understand more than how to practice traditional Chinese medicine,” Erik said, as he reflected on his and Oliver’s learning curve. “They need to understand how to turn a patient visit into a relationship, and how to help the patient understand his insurance coverage. Satisfying our customers is our highest priority. But we have to pay the bills, too.”
Maixner used generation, reflection, elaboration, and rehearsal in her coaching sessions, asking questions that exposed gaps in thinking or that invited the partners to strengthen their understanding of the behavior and tools they needed to adopt in order to be effective managers who delegate and empower their employees.
They developed a system to track clinic metrics, like the number of patient visits, patient disappearance rates, and referral sources. They learned how to ensure they were paid appropriately by insurance companies, raising reimbursements from as little as 30 cents on the dollar. They drafted a uniform protocol, or template, for clinicians to follow in seeing a new patient. They role-played conversations between themselves and their employees.
Central to putting the clinic on sound footing has been Erik’s becoming an effective coach and teacher of his coworkers. “We’re not just letting it be intuitive,” he said. For example, the new protocol for clinicians to follow in a patient’s initial session helps to clarify what brought the patient in, the therapies that might be useful, how to describe these therapies in terms the patient would be likely to understand, how to discuss fees and insurance reimbursement options, and how to recommend a treatment plan.
“If you’re the clinician, we’ll role-play: You are now the patient, and I’m the clinician. We raise questions, objections, and we practice how to respond and end up at the right place for the patient and for the clinic. Then we’ll switch roles. We record the role playing, and we listen to the differences: how you have responded to the patient, and how I have responded.”
In other words, learning through simulation, generation, testing, feedback, and practice.
As we write this, Inner Gate is in its eighth year, supporting four clinicians and two and a half administrative staff. A fifth clinician is coming up to speed, and the partners are looking to open a second location. By dedicating themselves to being learners as well as teachers, Erik and Oliver have turned their passion into a solid enterprise, and a top-rated acupuncture clinic in Portland.
We have talked throughout this book about learning, not about education. The responsibility for learning rests with every individual, whereas the responsibility for education (and training, too) rests with the institutions of society. Education embraces a world of difficult questions. Are we teaching the right things? Do we reach children young enough? How should we measure outcomes? Are our young people mortgaging their futures to pay for a college degree?
These are urgent issues, and we need to wrestle through them. But while we’re doing that, the techniques for highly effective learning that are outlined in this book can be put to use right now everywhere learners, teachers, and trainers are at work. They come at no cost, they require no structural reform, and the benefits they promise are both real and long-lasting.
مشارکت کنندگان در این صفحه
تا کنون فردی در بازسازی این صفحه مشارکت نداشته است.
🖊 شما نیز میتوانید برای مشارکت در ترجمهی این صفحه یا اصلاح متن انگلیسی، به این لینک مراجعه بفرمایید.