A Randomized Controlled Trial of Interleaved Mathematics Practice [pdf](uweb.cas.usf.edu)
uweb.cas.usf.edu
A Randomized Controlled Trial of Interleaved Mathematics Practice [pdf]
http://uweb.cas.usf.edu/~drohrer/pdfs/Rohrer_et_al_InPressJEdPsych.pdf
12 comments
Yeah, I think breadth-first is also better for student engagement. No more "what do I even need this for"-"two months pass"-"oh, that's what I need this for" cycles.
On this linear schedule the most annoying thing for me is when the thing you actually need down the road kept getting bumped from teaching schedules along the way.
Basically me in AP Calc the whole year long.
I too am a huge believer in breadth-first learning rather than depth-first, and my reasoning is the same: in my experience, ideas need to stay with you for a while before they take root in your mind and you get used to them.
The big ideas in any Mathematics subject are usually very few. They can be covered at a high-level in a few classes before going into them in detail. This also gives you a broader context for you to place the details as you come across them.
The big ideas in any Mathematics subject are usually very few. They can be covered at a high-level in a few classes before going into them in detail. This also gives you a broader context for you to place the details as you come across them.
I think the best way to practice math has already been discovered. Get a book, pen and paper, and practice math problems that are very challenging to hard for you. Do this day out and day in.
Problem? ... It's not very fun. You always feel frustrated and dumb because you don't understand it. So you'll eventually stop practicing at all because you feel like you don't make any progress.
Same thing with chess; play only with Grand masters (if you can find them) and after a few years of getting your ass kicked, you'll eventually get good. (Disclaimer: I only skimmed the article)
Problem? ... It's not very fun. You always feel frustrated and dumb because you don't understand it. So you'll eventually stop practicing at all because you feel like you don't make any progress.
Same thing with chess; play only with Grand masters (if you can find them) and after a few years of getting your ass kicked, you'll eventually get good. (Disclaimer: I only skimmed the article)
> Same thing with chess; play only with Grand masters (if you can find them) and after a few years of getting your ass kicked, you'll eventually get good. (Disclaimer: I only skimmed the article)
That sounds suspiciously like one of those surface leval optimal solutions which don't survive reality. I imagine the best way to improve is like the best way to diet or exercise. The flawed system you can sustain is better than the optimal system cannot.
Playing against the pinnacle of a sport as an amateur is likely sustainable for neither the amateur nor the master. The amateur will quickly tire of never feeling like they have a chance, and not seeing a real improvement when measured against the vast gulf of ability compared to the master. The master will be bored and unchallenged.
I think a close to optimal path to improvement for real people would be a club of people all dedicated to improving, with enough competition between them all that none can be complacent, and a mentor with enough skill to provide useful advice at all skill levels. Martial arts schools in the Eastern tradition, for example.
Now I'm wondering what the world would be like if every GM ran their own chess dojo. :)
That sounds suspiciously like one of those surface leval optimal solutions which don't survive reality. I imagine the best way to improve is like the best way to diet or exercise. The flawed system you can sustain is better than the optimal system cannot.
Playing against the pinnacle of a sport as an amateur is likely sustainable for neither the amateur nor the master. The amateur will quickly tire of never feeling like they have a chance, and not seeing a real improvement when measured against the vast gulf of ability compared to the master. The master will be bored and unchallenged.
I think a close to optimal path to improvement for real people would be a club of people all dedicated to improving, with enough competition between them all that none can be complacent, and a mentor with enough skill to provide useful advice at all skill levels. Martial arts schools in the Eastern tradition, for example.
Now I'm wondering what the world would be like if every GM ran their own chess dojo. :)
Keeping motivation to practice and learn is also part of a good study plan. The best way isn't some horror plan that can't be followed by mortal man, it's one that can be and yields the best results.
>Same thing with chess; play only with Grand masters (if you can find them) and after a few years of getting your ass kicked, you'll eventually get good.
Wouldn't the grand masters not enjoy beating you over and over again? That's why Go has a handicap system and why online games have matchmaking systems. I think it would be more courteous to try and play with people that could potentially learn something from you, instead of you only learning from them.
Wouldn't the grand masters not enjoy beating you over and over again? That's why Go has a handicap system and why online games have matchmaking systems. I think it would be more courteous to try and play with people that could potentially learn something from you, instead of you only learning from them.
> Wouldn't the grand masters not enjoy beating you over and over again?
I mean, a couple times maybe? At some point I feel like they’d think this is a waste of time.
I mean, a couple times maybe? At some point I feel like they’d think this is a waste of time.
Sure... but also while you're sitting there doing challenging problems, mix them up... rather than just doing trig problems, alternate a bit of trig with bit of square equations, etc. That way you'll tire less quickly and retain more. That's what this article is about, not that there's a better method than sitting and doing challenging exercises.
> Same thing with chess; play only with Grand masters (if you can find them) and after a few years of getting your ass kicked, you'll eventually get good. (Disclaimer: I only skimmed the article)
I doubt that this would work. It would be like trying to learn math just by reading cutting edge research papers and trying to solve major open conjectures. The gaps between your current level of skill and knowledge and what is needed to comprehend anything from the material you are trying to learn from would be too big.
Yes, you learn best with challenging material, but it needs to be sufficiently close to your current skill level that you can at least partly understand it.
There used to be a chess club that met weekly in the '80s in Sunnyvale, CA, at the Lockheed Employees Recreation Association facilities. Not many people actually went to it to play chess, because a regular attendant was Richard Shoreman, a well known chess teacher and coach. If you brought the score from one of your recent tournament games, Shoreman would play over it and annotate it for everyone, and give you suggestions on how to improve.
His specialty was getting people whose progress had gotten stuck, typically in the 1800-2300 USCF rating range, unstuck, although he was also good at getting lower rated players to rapidly improve.
The #1 reason people got stuck, especially around the 2000 rating neighborhood, was they were spending too much time studying GM games and trying to play like GMs. This doesn't work because, to paraphrase Shoreman (because I don't remember the exact statement), GMs play good chess, and before you can play good chess, you have to get good at playing bad chess.
Good chess is what GMs do--deep subtle positional play and a thorough opening understanding. Underpinning all of that good chess, though, is tactics. Every positional thing ultimately is either to enable or prevent some tactical thing from happening, and to really play positional chess well you have to have a good tactical understanding.
So what Shoreman did with these stuck 1800-2300s was get them to set aside their opening repertoires that they had copied from their favorite GMs, and get them play aggressive tactical openings. It is important to note that these openings were NOT required to sound--the important thing was to get into a tactical game. You are playing for a knockout here, not for a slightly superior pawn endgame. If you can get a good attack from a sacrifice, but can't calculate it out for a win--go for it! I think I recall Shoreman saying to always play the most aggressive move that you cannot see an outright refutation for, or something to that effect.
That kind of chess is what he meant by bad chess. Once you really understand that bad chess, you can then start to understand good chess.
I doubt that this would work. It would be like trying to learn math just by reading cutting edge research papers and trying to solve major open conjectures. The gaps between your current level of skill and knowledge and what is needed to comprehend anything from the material you are trying to learn from would be too big.
Yes, you learn best with challenging material, but it needs to be sufficiently close to your current skill level that you can at least partly understand it.
There used to be a chess club that met weekly in the '80s in Sunnyvale, CA, at the Lockheed Employees Recreation Association facilities. Not many people actually went to it to play chess, because a regular attendant was Richard Shoreman, a well known chess teacher and coach. If you brought the score from one of your recent tournament games, Shoreman would play over it and annotate it for everyone, and give you suggestions on how to improve.
His specialty was getting people whose progress had gotten stuck, typically in the 1800-2300 USCF rating range, unstuck, although he was also good at getting lower rated players to rapidly improve.
The #1 reason people got stuck, especially around the 2000 rating neighborhood, was they were spending too much time studying GM games and trying to play like GMs. This doesn't work because, to paraphrase Shoreman (because I don't remember the exact statement), GMs play good chess, and before you can play good chess, you have to get good at playing bad chess.
Good chess is what GMs do--deep subtle positional play and a thorough opening understanding. Underpinning all of that good chess, though, is tactics. Every positional thing ultimately is either to enable or prevent some tactical thing from happening, and to really play positional chess well you have to have a good tactical understanding.
So what Shoreman did with these stuck 1800-2300s was get them to set aside their opening repertoires that they had copied from their favorite GMs, and get them play aggressive tactical openings. It is important to note that these openings were NOT required to sound--the important thing was to get into a tactical game. You are playing for a knockout here, not for a slightly superior pawn endgame. If you can get a good attack from a sacrifice, but can't calculate it out for a win--go for it! I think I recall Shoreman saying to always play the most aggressive move that you cannot see an outright refutation for, or something to that effect.
That kind of chess is what he meant by bad chess. Once you really understand that bad chess, you can then start to understand good chess.
This is aligned with everything I learned in Dr.'s Oakley and Sejnowski class entitled "Learning How to Learn" on Coursera. Conceptually, the class is ridiculously simple,but with some of the most powerful mental tools available for learning subjects. This interleaving tool is included, and it should be noted that this is part of a self-testing phase of learning, which is something that they want you to leap into as quickly as possible.
I would strongly recommend the course to anyone interested in learning and who doesn't have a good learning process, because that's all that it ultimately is: a set of steps that you use to train your brain to remember a body of knowledge.
I would strongly recommend the course to anyone interested in learning and who doesn't have a good learning process, because that's all that it ultimately is: a set of steps that you use to train your brain to remember a body of knowledge.
I've seen interleaving research like this before and it's my understanding interleaving trains not just the ability to work a particular problem but the ability to better recognize the class of problem.
Anecdotally, after reading a paper on this, I made a Python script to drill guitar pieces, chords, progressions, etc. in an interleaved fashion and I believe it has helped me become a better player. Before doing this, when I would practice, e.g., scales, I might start in A then B, C, etc. but what I noticed was after doing the first scale, the rest were just minor variations so my training went from deliberate to auto-pilot. Now, with interleaving, I have to give some brief thought not just to how to mechanically perform the piece but what exactly the piece being asked for is. Consequently, during a performance, my ability to go from an intention of doing something on the guitar to actually plucking the first string is much more comfortable and faster.
Anecdotally, after reading a paper on this, I made a Python script to drill guitar pieces, chords, progressions, etc. in an interleaved fashion and I believe it has helped me become a better player. Before doing this, when I would practice, e.g., scales, I might start in A then B, C, etc. but what I noticed was after doing the first scale, the rest were just minor variations so my training went from deliberate to auto-pilot. Now, with interleaving, I have to give some brief thought not just to how to mechanically perform the piece but what exactly the piece being asked for is. Consequently, during a performance, my ability to go from an intention of doing something on the guitar to actually plucking the first string is much more comfortable and faster.
In the mid-1980s, I used my trusty Commodore 64 to build "test banks" for my math, chem, physics, and bio courses in high school and early college.
My rationale was the same as in this paper: I wanted to handicap myself from knowing the "type" of question I was trying to answer and force myself to think, "Ok. Where is this question 'located' on the map of everything I've been taught? And once I know its location, where do I need to go to get to its answer?"
So, I hand-entered every question I could get my hands on and I had the computer throw random problems/questions at me before an exam.
The problem w/ my approach was that I had to hand-enter problems from my textbooks, so there was a decent chance that I would still remember the type of question simply from the act of typing it into my 'database.' But it was the best I could do.
I teach some college chemistry now and I think interleaving would be helpful to my students. I say this because one of the biggest issues on exams is that the students will have no idea what _type_ of problem they are looking at. If they correctly determine the type of problem, the remaining work is relatively easy. And I think the problem stems from students practicing the work by answering multiple problems that belong to the same category.
My rationale was the same as in this paper: I wanted to handicap myself from knowing the "type" of question I was trying to answer and force myself to think, "Ok. Where is this question 'located' on the map of everything I've been taught? And once I know its location, where do I need to go to get to its answer?"
So, I hand-entered every question I could get my hands on and I had the computer throw random problems/questions at me before an exam.
The problem w/ my approach was that I had to hand-enter problems from my textbooks, so there was a decent chance that I would still remember the type of question simply from the act of typing it into my 'database.' But it was the best I could do.
I teach some college chemistry now and I think interleaving would be helpful to my students. I say this because one of the biggest issues on exams is that the students will have no idea what _type_ of problem they are looking at. If they correctly determine the type of problem, the remaining work is relatively easy. And I think the problem stems from students practicing the work by answering multiple problems that belong to the same category.
Interleaving different types of problems is one of several "evidence based" study techniques described in the excellent book "Make it stick" by Peter Brown. Highly recommended; this book is based on the latest and most solid research on learning and dispels a lot of long-time misconceptions.
I am affraid this might be https://en.wikipedia.org/wiki/Hawthorne_effect
I don't see anything controlling for that in the setup:
"The present study examined interleaved practice in a large number of classes at multiple schools over a period of five months, and all instruction was delivered solely by teachers who had no prior association with the intervention or the authors."
I don't see anything controlling for that in the setup:
"The present study examined interleaved practice in a large number of classes at multiple schools over a period of five months, and all instruction was delivered solely by teachers who had no prior association with the intervention or the authors."
There's a control group. Both groups were watched by the experimenters. Not only that, the same teachers had students in both groups in different classes.
But the control group would know that nothing has changed. The group testing the new method would feel the excitement - the control group would not.
I believe that interleaving is a principle to applied to life in general
Say you want to learn something - let's just say math or a new language - how do you go about interleaving the learning materials?
Spaced repetition aids are fairly common. If you can store your curriculum as flash cards, then apps like Memrise and Anki are designed specifically for this.
Spaced repetition is not the same as interleaving, but the apps will force you to interleave knowledge anwyay.
Spaced repetition is not the same as interleaving, but the apps will force you to interleave knowledge anwyay.
I think this is why practicing on the actual test (SAT, GMAT, CFA etc) is so much more effective than exercises on particular sections.
> so much more effective than exercises on particular sections
Effective at what though. Just passing the test?
Do the students actually understand the material? Can they apply it to novel (to them) situations?
When I was studying physics ('74-'77) a significant number of students who all had just as good test results as I did in senior high school dropped out after the first term because they just couldn't manage the material. They had learnt enough to pass the exams (British A-levels) but had not really understood and had not practised enough. I suspect that rise of modular courses might make this situation even worse but that might be prejudice on my part.
Effective at what though. Just passing the test?
Do the students actually understand the material? Can they apply it to novel (to them) situations?
When I was studying physics ('74-'77) a significant number of students who all had just as good test results as I did in senior high school dropped out after the first term because they just couldn't manage the material. They had learnt enough to pass the exams (British A-levels) but had not really understood and had not practised enough. I suspect that rise of modular courses might make this situation even worse but that might be prejudice on my part.
Specifically wrt to tests like SAT, GMAT and other type of admissions testing, I believe the idea of practicing on the test as a whole is to allow you to get better at "strategizing" your time thereby maximizing the number of questions you could attempt. This paper seems to talk about a different concept.
I keep finding most of those papers obvious on their face and quite simple to derive from first principles and some observation. What happened to teaching that so many patently obvious concepts have to be "rediscovered" so often?
Let's say I'm teaching 50 min periods. 15 mins for questions. Half hour to introduce new material, with a couple of examples. Five mins left. Interleave now?
Everyone has sat through 12 or 16 years of school. That can lead them to think they know all about it. I've been teaching since 79 and I'm still learning a lot,all the time. It ain't obvious.
Everyone has sat through 12 or 16 years of school. That can lead them to think they know all about it. I've been teaching since 79 and I'm still learning a lot,all the time. It ain't obvious.
I've only been actively teaching (at home, no less) for a couple of years, but I can definitely identify with this. I make index cards with each kind of question we encounter in the curriculum, then I draw some random cards from the deck in order to pick questions going all the way back to the start. We do a quiz every week for every subject that's made up of those questions.
It always comes down to time though, it seems like you can use all these ideas and teach for mastery, but it costs about 30% in terms of forward motion through the material. I'm lucky in that that I can just decide to pay that price, but it would be a real stretch to put all this into practice in a place where you have a coverage deadline that's already hard to hit.
It always comes down to time though, it seems like you can use all these ideas and teach for mastery, but it costs about 30% in terms of forward motion through the material. I'm lucky in that that I can just decide to pay that price, but it would be a real stretch to put all this into practice in a place where you have a coverage deadline that's already hard to hit.
'Obvious' != True
Now we have some evidence that it is probably true; we still need a bunch more studies to make sure this is reproducible and robust for a variety of circumstances. Yay science !
Now we have some evidence that it is probably true; we still need a bunch more studies to make sure this is reproducible and robust for a variety of circumstances. Yay science !
It wasn't so obvious to plenty of smart and competent teachers and textbook writers, so if nothing else it's worth "rediscovering" to bring it to their attention. Not everyone has the time or energy to derive their everyday decisions from first principles.
In the same way that muscle need to rest to grow, knowledge need to rest to stay with you. Even if subjectively seems that nothing has been learned the first time that you are in contact with a subject, that is hardly ever truth. Even if only the name of the concepts are familiar that's some progress from where it's possible to build solid knowledge.