What's Wrong with What Works?

Sexual intercourse began

In nineteen sixty-three

(which was rather late for me) -

Between the end of the Chatterley ban

And the Beatles’ first LP.

(from Annus Mirabilis by Philip Larkin)

            Evidence-based practice has become all but a cliché in educational discourse. Perhaps finally tiring of talking about ‘learnings’, ‘privileging’ and verbing any other noun they can get their hands on, educationists have decided to "sing from the same songsheet” of evidence-based practice. That’s got to be a good thing, right? Well, yes, it would be if they were singing the same tune and the same words. Unfortunately, evidence-based practice means different things to different people. This is why I personally prefer the term scientific evidence-based practice. But how are we to know what constitutes (scientific) evidence-based practice?

The Education Minister for New South Wales has recently (August 2012) launched the Centre for Education Statistics and Evaluation which “undertakes in depth analysis of education programs and outcomes across early childhood, school, training and higher education to inform whole-of-government, evidence based decision making.” (See http://tinyurl.com/d53f2y2 and http://tinyurl.com/c6uh3y4). Moreover, we are told,

“The Centre turns data into knowledge, providing information about the effectiveness of different programs and strategies in different contexts – put simply, it seeks to find out what works”.

Ah, ‘what works’, that rings a bell. It is too early to tell whether this new centre will deliver on its promises but what about the original ‘What Works Clearinghouse’ (WWC), the US based repository of reports on educational program efficacy that originally promised so much?

As Daniel Willingham has pointed out:

“The U.S. Department of Education has, in the past, tried to bring some scientific rigor to teaching. The What Works Clearinghouse, created in 2002 by the DOE's Institute of Education Sciences, evaluates classroom curricula, programs and materials, but its standards of evidence are overly stringent, and teachers play no role in the vetting process.” (See http://tinyurl.com/bn8mvdt)

My colleagues and I have also been critical of WWC. And not just for being too stringent. Far from being too rigorous, the WWC boffins frequently make, to us, egregious mistakes; mistakes that, far too often for comfort, seem to support a particular approach to teaching and learning.

I first became a little wary of WWC when I found that our own truly experimental study on the efficacy of Reading Recovery (RR) had been omitted from their analyses underlying their report on RR. Too bad, you might think, that’s just sour grapes. But, according to Google Scholar, the article has been cited 160 times since publication in 1995 and was described by eminent American reading researchers Shanahan and Barr as one of the “more sophisticated studies”. Interestingly enough, it is frequently cited by proponents of RR (we did find it to be effective) as well as by its critics (but effective only for one in three children who received it). So why was it not included by WWC? It was considered for inclusion but was rejected on the following grounds:

“Incomparable groups: this study was a quasi-experimental design that used achievement pre-tests but it did not establish that the comparison group was comparable to the treatment group prior to the start of the intervention.”

You can read the details of why this is just plain wrong, as well as other criticisms of WWC, in Carter and Wheldall (2008) (http://tinyurl.com/c6jcknl). Suffice to say that participants were randomly allocated to treatment groups and that we did establish that the control group (as well as the comparison group) was comparable to the (experimental) treatment group who received RR prior to the start of the intervention. This example also highlights another problem with WWC’s approach. Because they are supposedly so ‘rigorous’, they discard the vast majority of studies from the research literature on any given topic as not meeting their criteria for inclusion or ‘evidence standards’. In the case of RR, 78 studies of RR were considered and all but five were excluded from further consideration. Our many other criticisms of what we regard as a seriously flawed WWC evaluation report on RR are detailed in Reynolds, Wheldall, and Madelaine (2009) (http://tinyurl.com/cuj8sqm).

Advocates of Direct Instruction (DI) seem to have been particularly ill-served by the methodological ‘rigour’ of WWC, for not only are most more recent studies of the efficacy of DI programs excluded because they do not meet the WWC evidence standards but they also impose a blanket ban on including any study (regardless of technical adequacy) published before 1985; an interesting if somewhat idiosyncratic approach to science. Philip Larkin told us that sex only began in 1963 but who would have thought that there was no educational research worth considering before 1985?  (Insert your own favourite examples here of important scientific research in other areas that would fall foul of this criterion. Relativity anyone? Gravity?) Zig Engelmann, the godfather of DI, has written scathingly about the practices of WWC (http://tinyurl.com/c5pjm9d and http://tinyurl.com/85t2vpt), concluding:

“I consider WWC a very dangerous organization. It is not fulfilling its role of providing the field with honest information about what works, but rather seems bent on finding evidence for programs it would like to believe are effective (like Reading Recovery and Everyday Mathematics).”

Engelmann can be forgiven for having his doubts given that for the 2008 WWC evaluation report on the DI program Reading Mastery (RM) (http://tinyurl.com/d8kawf7), WWC could not find a single study that met their evidence standards out of the 61 studies they were able to retrieve. (Engelmann claims that there were over 90 such studies, mostly peer reviewed.)

The most recent WWC report on RM in 2012 (http://tinyurl.com/7bdobxv), specifically concerned with its efficacy for students with learning disabilities, determined that only two of the 17 studies it identified as relevant met evidence standards and concluded:

“Reading Mastery was found to have no discernible effects on reading comprehension and potentially negative effects on alphabetics, reading fluency, and writing for students with learning disabilities.”

In response to this judgement, the Institute for Direct Instruction pointed out, not unreasonably, that, of the two studies considered:

“One actually showed that students studying with RM had significantly greater gains than students in national and state norming populations. Because the gains were equal to students in Horizons (another DI program), the WWC concluded that RM had no effect. The other study involved giving an extra 45 minutes of phonics related instruction to students studying RM. The WWC interpreted the better results of the students with the extra time as indicating potentially negative effects of RM.” (http://tinyurl.com/9oewdlo)

 In other words when Reading Mastery was compared with another very similar DI program (in each case), and the results were no different from or slightly better than the standard Reading Mastery program, it was concluded that Reading Mastery was therefore ineffective for students with learning disabilities and possibly even detrimental to their progress. It is conclusions such as these that have led some experts in the field to wonder whether this is the result of incompetence or bias: cock up or conspiracy.

If we needed any further proof of the unreliability of WWC reports, we now have their August 2012 report on whether Open Court Reading^© improves adolescent literacy (http://tinyurl.com/9nzv5wj). True to form, they discarded 57 out of 58 studies as not meeting evidence standards. On the basis of this one study they concluded that Open Court “was found to have potentially positive effects on comprehension for adolescent readers”. There are at least three problems with this conclusion. First, this is a bold claim based on the results for just one study, the large sample size and their ‘potentially positive’ caveat notwithstanding. Second, the effect size was trivial at 0.16, not even ‘small’, and well below WWC’s own usual threshold of 0.25. Third, and most important of all, this study was not even carried out with adolescents! The study sample comprised “more than 900 first-grade though fifth-grade who attended five schools across the United States”. As Private Eye magazine would have it “shorely shome mishtake” …

There is, then, good reason for serious concern regarding the reliability of the judgments offered by WWC. The egregious errors noted above apart, there is the more pressing problem that truly experimental trials are still relatively rare in educational research and those that have been carried out may often be methodologically flawed. In its early years, What Works was renamed ‘Nothing Works’ by some because there was little or no acceptable evidence available on many programs. Clearly, teachers cannot just stop using almost all programs and interventions until there are sufficient RCTs testifying to their efficacy to warrant adopting them. Hattie, for example, in his seminal 2009 work ‘Visible Learning’ has synthesized over 800 meta-analyses relating to achievement in order to be able to offer evidence-based advice to teachers (http://tinyurl.com/3h9jssl). (Very few of the studies on which the meta-analyses were based were randomized control trials, however, as Hattie makes clear.)

Until we have a large evidence base of methodologically sound randomized control trials on a wide variety of educational programs, methods and procedures, we need a more sophisticated and pragmatic analysis of the evidence we currently have available. It is not a question of accepting any evidence in the absence of good evidence, but rather of assessing the existing research findings and carefully explaining the limitations and caveats.

As I have attempted to show, the spurious rigour of WWC whereby the vast majority of studies on any topic are simply discarded as being too old or too weak methodologically, coupled with their unfortunate habit of making alarming mistakes, makes it hard to trust their judgments. If the suggestions of bias regarding their pedagogical preferences have any substance, we have even more cause for concern. As it stands, What Works simply won’t wash.


Postscript November 15, 2013 

Further to my original blog post ‘What’s wrong with What Works?’, WWC have released new reports on reading interventions that confirm that at WWC it is business as usual. My colleague, Mark Carter, alerted me to problems associated with the WWC evaluation (March 2013) of the efficacy of FastForWord® (FFW):

“I just can't believe WWC. They found "positive" effects of FFW for alphabetics but the ES was 0.15 – trivial, below their own 0.25 (low) standard for educational significance and moving the average child a whole 6 percentile ranks. They got exactly the same results for fluency and comprehension and reached different conclusions for each. Three identical effect sizes - three different conclusions. 

They say in the text that the effects are below 0.25 and are "indeterminate" but then give it a positive rating. They seem to be vote counting the number of significant outcomes in a given area. This is conceptually antithetical to the whole idea of meta-analysis. The problem with examining significance is that it is substantially a function of sample size - that is why we use effect sizes to aggregate findings across studies. And, to boot, they are ignoring their own criteria for educational significance. I really can't believe it.” 

It is also worth noting that WWC based their efficacy evaluation on just nine studies out of the 342 studies they originally identified that looked at the efficacy of FFW on early reading skills; seven studies that met their evidence standards without reservations and two studies that met their standards with reservations.

Taking care of Matthew

Some 25 years ago now, Keith Stanovich popularised the concept of Matthew effects in reading. The Matthew Effect takes its name from a passage in the St Matthew Gospel:

“For unto every one that hath shall be given, and he shall have abundance: but from him that hath not shall be taken away even that which he hath” (Matthew 25:29)

Often translated as “the rich get richer and the poor get poorer”, this neat analogy clearly struck a chord with reading specialists. Because if you start off by learning to read well, you will indeed get richer and richer; certainly metaphorically, but probably quite literally too.

You will read far more than your less fortunate peers, you will have greater exposure to a wider variety of books, you will learn far more new vocabulary words, you will pass from learning to read to reading to learn much more quickly, and so on. If, on the other hand, you come to school ill-prepared for learning to read, with poor language skills as a result of social disadvantage for example, and you do not latch on to what reading is all about quickly, in that first year of schooling, you are likely to experience continuing failure throughout your educational trajectory. Because literacy underpins everything. In other words the gap between the reading haves and the reading have nots, the literacy rich and the literacy poor, widens over the years creating social division and perpetuating social disadvantage from one generation to the next. Teaching children to read, then, is not just an educational issue; it is a social justice issue.

And it is this concern with social justice that has motivated our work in MultiLit from the outset. Our aim has always been to help those who struggle to learn to read for whatever reason, but particularly those students from socially disadvantaged, culturally diverse and Indigenous backgrounds, as well as those with other special learning needs. Our non-categorical approach to teaching asserts that what all of these children need is effective scientific evidence-based instruction.

It is a source of continuing bewilderment to me, that so many professionals in education continue to resist the idea of direct, explicit and systematic instruction in phonics, while at the same time shedding crocodile tears about the plight of the disadvantaged children struggling in our schools. But they are often the cause of the problem. To use a medical term, the problem is iatrogenic; it is the ‘treatment’ the students are receiving, or rather the lack of effective instruction in schools, that is often the cause of the reading problem. Low-progress readers are frequently simply ‘instructional casualties’.

It is a concern with helping all children to learn to read, regardless of the apparent reason for their difficulties, that has been the driving force behind our work. This is why we launched the Making Up Lost Time In Literacy or MultiLit Initiative in 1995. We were initially concerned with trying to help older low-progress readers in the final years of primary school so that they could stand some chance of making a success of high school. The MultiLit Reading Tutor Program was our solution, a one to one program of direct, explicit and systematic instruction in the basics of reading comprising a focus on teaching word attack skills, teaching a bank of frequently occurring sightwords and providing a supportive context for book reading in which to generalise these newly acquired skills.

We worked with older low-progress readers because this is what we knew; this is what we had been researching for a number of years. But we were always aware that we were the ambulance picking up the instructional casualties while also lobbying for more effective initial reading instruction in schools. As we often said, our aim was to make ourselves redundant! We naively thought that the effort we put into pressing for the National Inquiry into the Teaching of Literacy might bring this about. But we all know what happened as a result of the Inquiry … very little indeed.

Whatever the case with initial reading instruction, and even if scientific evidence-based teaching of reading were to be introduced, there will always be a minority of children who will struggle to learn to read. We currently estimate that we should target the bottom 25% of students as being in need of additional remedial support. Consequently, it makes good sense to intervene early with children who are struggling to learn to read in their first year of schooling, before the Matthew effect becomes entrenched. Whatever reservations one might have about aspects of Reading Recovery (and we have certainly not been bashful about expressing our own reservations), the late Dame Professor Marie Clay deserves enduring credit for her pioneering work on the importance of intervening early with children who are seen still to be struggling in the key skills of learning to read after their first year of schooling.

While we can admire Marie Clay’s pioneering work, it has to be remembered that Reading Recovery has remained virtually unchanged in over thirty years now. An awful lot has happened in that time. We now know far more about how reading works and how it should best be taught. But we always thought that someone else would come up with an alternative program that would ideally be at least as effective but more cost effective than Reading Recovery, predicated as it is on daily one to one tutoring by a highly trained Reading Recovery Teacher. We waited but nothing happened and so, over five years ago now, we started to think about how MultiLit might be adapted to meet the needs of this much younger target group.

The MiniLit Early Literacy Intervention Program is the product of an ongoing program of research and development carried out by our specialist team of academic researchers and special educators. A continuing process of refinement by trial and revision was employed until the program met the stringent efficacy criteria to justify its release to the wider community. We do not just write and hope! This final published version of MiniLit entailed a major revision of previous experimental versions and represents what we believe to be scientific evidence-based best practice.

MiniLit is informed by the findings of scientific research, carried out over the past 40 years, into how reading works and how it may best be taught. It is also completely in accord with the recommendations of the various national reports into effective reading instruction that have emphasised the five key pillars of reading instruction (sometimes known as the ‘five big ideas’), namely: phonemic awareness, phonics, fluency, vocabulary and comprehension.

It consists of 80 carefully structured lessons (sufficient for at least two terms of instruction) in an easy to hard sequence and is divided into two levels, Levels 1 and 2, with 40 lessons at each level. There are three main components of each lesson: Sounds and Words Activities; Text Reading; and Story Book Reading. It is very important that all three of these components are taught daily, as specified in the manual.

But does it work? A recent analysis of the progress of 90 struggling young readers who had attended MiniLit programs for four days per week for 15 weeks showed that they had made substantial and statistically significant gains on all of the measures of reading and related skills with large effect sizes evident.  ‘Gold Standard‘ randomised control trials have also been completed supporting the efficacy of the new MiniLit program.

Who's for coffee?

Several years ago now, my wife, Robyn, and I found to our amazement that we had actually won something in a prize draw. Apparently our name had been pulled out of the hat at the local vet clinic and we had won a brand new, all singing, all dancing, espresso coffee machine. (Considering the vast amount of money we had invested over the years with the vet to keep our dogs in tip top health, it was not a huge return, but decidedly better than nothing at all.)

            With mean-spirited glee, and to the angst of our trendy kids then still living at home, we decided to take it away with us to our weekender in the country (‘Harefield’), so that the ‘young adults’ would not wreck it while we were away.

            It sat on the kitchen bench, still in its box, for several country weekends before I finally decided that I really did have to learn how to drive the thing. To this old fogie, it looked more like something that belonged in a science lab, or in Dr Who’s Tardis, than in a kitchen.

            I examined each part carefully and gratefully located the instructions booklet. This is when the trouble started…  I read the instruction booklet thoroughly. Then I read it again. I was still none the wiser. As a literate adult, I could read and understand every word in the booklet but I still had no idea what to do with the coffee machine.

            The booklet continually referred to ‘the coffee maker’ that, not unreasonably, I took to mean the gadget, the espresso machine. It was only on the third reading that the light went on and it dawned on me that it meant me. I was the coffee maker!

            Now what lessons can we draw from this little tale of domestic confusion, other than that it is never safe to trust a baby boomer with a Generation Y affectation …?

            First, we can reflect on the simple model of reading that posits that reading comprehension is the product of reading accuracy and listening comprehension. It is no use being able to read the words if you do not know what they mean. I thought I knew what was meant by coffee maker in this context but I was wrong. Hence, I could read the booklet but I could not understand what I had read.

            Second, what I really needed was tutoring in coffee making from a barista or at least someone who was more accomplished in the dark arts of coffee preparation than I am. And this sort of tutoring is, of course, one of the things that low-progress readers need to achieve success: effective tutoring by a more accomplished reader.

            Third, I should have stuck with the coffee plunger that works really well for me. (Incidentally, my wife bought about ten coffee plungers on sale, for next to nothing, but we could not give them away to our sophisticated kids!) Now the analogy is not perfect because I like good espresso coffee as much as the next poseur but new and modern, let alone cool and trendy, is not necessarily a good thing. And so it is with reading instruction.

            We have learned to our cost that cool and trendy Whole Language teaching made for far too many low-progress readers. We once knew how to teach reading; we did it quite well and most kids learned how to read reasonably quickly. And then we wilfully adopted an unproven method that proved to be not nearly as effective, but we carried on using it regardless. And we called it progressive education! As my kids would say: “how weird is that?”

            Fourth, it wasn’t my fault! The espresso machine booklet was a lousy instructor: it did not operationally define its terms and present the information in a logical order. Nor did my pathetic attempts at discovery learning serve me any better. I was completely unable to construct my own personal approach to coffee making with this machine.

            With basic skills, like coffee making and reading, we all need initial direct instruction in the early stages. I can construct my own version of a decaf triple shot macchiatto later, in the same way that I’ll subsequently be able to detect the inherent, privileged phallocentic hegemonies in the books I read. But I do actually need to learn how to read first.

            I could go on … I’m on a roll here with the lessons to be learned about reading from my attempts to make espresso coffee. Suffice to say that the espresso machine languished in a kitchen cupboard for quite a time while I continued to use the plunger. Some years later I finally learned how to use it. Now, who’s for coffee …?

How do we ensure that all children learn to read?

            If all children are to learn to read to a good level of proficiency in the first few years of schooling, we need a clear plan to ensure that no child falls through the net. Such a plan must be both effective and cost-effective. It has become increasingly accepted in recent years that a three tier, phased model of reading instruction, known as Response to Intervention (or RtI), is the best means of achieving this.

            The three tier RtI model is predicated upon a first tier of exemplary initial instruction in reading for all students during their first year of schooling (Kindergarten in New South Wales). This first tier of instruction should essentially comprise the best scientific evidence-based instruction. To the layman, this sounds obvious but in many Australian schools a less effective implicit model of reading instruction has held sway for the last few decades. Much of this implicit approach to reading instruction is highly desirable as a bedrock upon which to build effective reading instruction, and it may be enough for a minority of children, but most will need direct, explicit and systematic instruction in the five pillars or ‘five big ideas’ of teaching reading : phonemic awareness, phonics, fluency, vocabulary and comprehension. What is often lacking in initial reading instruction, in particular, is effective, specific instruction in what is known as synthetic phonics; how to relate letters to sounds and to blend letter sounds into words.

            Even when afforded exemplary reading instruction, there will always be some children who take longer than others to catch on. It is important to identify these low-progress readers as early as possible so that they do not fall too far behind their peers as their difficulties compound. Children who do not learn to read in the first few years of schooling are typically destined to a school career of educational failure because reading underpins almost all subsequent learning. A safe strategy is to target students who fall into the bottom twenty five per cent of the population for remedial reading intervention, as soon as their difficulties become apparent. Students’ progress should be checked regularly, in order to provide remedial intervention for those who need it from the beginning of Year One, at the very latest.   

            The RtI model recommends that struggling readers, the low-progress readers who comprise the bottom quartile, should be offered more intensive Tier 2 intervention in small groups of three to four students. Again the instruction provided to these students should be based on what the scientific research evidence has shown to be most effective— essentially the same five big ideas of reading instruction but more intensive and more individualised. In small groups, teachers are able to be more responsive to the specific idiosyncratic needs of the students with whom they are working. Small group instruction can be just as effective as one-to-one instruction for children without severe reading difficulties.

            Even with a solid Tier 2 small group reading program in place, there will still be a very small number of students who ‘fail to thrive’, perhaps about 3-5% of all Year 1 students. These are the students for whom we should reserve Tier 3 one-to-one intensive reading instruction, preferably with a specialist reading teacher with a sound background in special education. The same five big ideas are still critical. What is different, of course, is the intensity of instruction. Having successfully taught the vast majority of Year 1 students the basics of learning to read by Tier 1 and, where necessary, Tier 2 small group teaching, it is a far more manageable proposition to provide these few remaining students with the individual reading support that they will need, for as long as they need it.

            With this three tier Response to Intervention model in place, predicated upon scientific evidence based reading instruction, almost all, if not all, children will become proficient readers. Of course, the RtI model does not stop at the end of Year 1, it is important to monitor reading progress closely for all students, especially for the first three years of schooling. But by employing these procedures rigorously and teaching scientifically, it is not too much to expect all of our children to learn to read.

Note: A longer version of this post is available at: http://tinyurl.com/7cr8ddo