Saturday, July 21, 2018

Hattie Haters and Lovers: Both are Still Missing the Effective Implementation Steps that Practitioners Need


Critical Questions to Ask your “Hattie Consultant” Before You Sign the Contract

 [For the Entire Blog Message:  CLICK HERE]


Dear Colleagues,

Introduction

   Over the past five years especially, John Hattie has become internationally-known for his meta-analytic research into the variables that most-predict students’ academic achievement (as well as other school-related outcomes). 

   Actually, his different Visible Learning books (which have now generated a “Hattie-explosion” of presentations, workshops, institutes, and “certified” Hattie consultants) report the results of his “meta-meta-analyses.”  These are analyses that average the effect sizes from many other meta-analyses that themselves have pooled research that investigated the effect of one psychoeducational variable, strategy, intervention, or approach on student achievement.

   Let me be clear from the start.  This Blog is not to criticize or denigrate, in any way, Hattie on a personal or professional level.  He is a prolific researcher and writer, and his work is quite impressive.

   However, this Blog does critique the statistical and methodological underpinnings of meta- and meta-meta-analytic research, discuss its strengths and limitations, but most essentially, delineate its research-to-practice implications and implementation necessities.

   It also reviews a recent critique by Dr. Robert Slavin, the primary developer of Success for All, an evidence-based literacy (and math) program that is one of the longest-standing, best-researched, and most-effective instructional approaches in recent history. 

   On June 21, 2018, Slavin published a Blog, John Hattie is Wrong, where he reported his analyses of and concerns about Hattie’s research and conclusions. 

   Peter DeWitt, a Hattie colleague and trainer, then responded to Slavin in a June 26, 2018 Education Week article, John Hattie Isn’t Wrong. You Are Misusing His Research.  This Blog included quotes from Hattie in response to Slavin.

   And then, Slavin responded back in an Education Week Letter to the Editor published on July 17, 2018.
_ _ _ _ _

   My current Blog quotes from all three of these pieces so that the issues surrounding meta-analyses and meta-meta-analyses are clearly delineated.

   In this end, it is important that educators understand:

·      The strengths and limitations of meta-analytic research—as well as meta-meta-analytic research;

·       What conclusions can be drawn from the results of sound meta-analytic research;

·       How to transfer sound meta-analytic research into actual school- and classroom-based instruction or practice; and

·       How to decide if an effective practice in one school, classroom, or teacher is “right” for your school, classrooms, and teachers.

   But there is more. . .
_ _ _ _ _ _ _ _ _

My Addition:  The Implementation Methods are Missing

   The second half of this Blog identifies Hattie’s current (June, 2018) “Top Twenty” approaches showing the strongest meta-meta-analytic effects on student learning and achievement.

   Significantly, many of these “Top 20” approaches are different than when I wrote a previous Blog message regarding Hattie’s work on September 25, 2017 Blog—less than 10 months ago.  Second, there are new approaches on the list that have never previously been cited. . . and other approaches have new labels. 

   Finally, and most importantly, many of the approaches have such generic or global names that it is virtually impossible to determine, at a functional school, classroom, teacher, or student level, what methods and implementation steps were used in the research, and what methods and steps should be used, right now, by a district or school interested in implementation.

   This science-to-practice implementation issue is what none of the researchers are really talking about. 

   And, with the ever-changing list of Top Twenty effects appearing to be a moving target, districts and schools have an additional dilemma of trying to keep up with and accurately interpreting the research.
_ _ _ _ _

   The remainder of the Blog discusses a step-by-step approach that districts and schools need to take to translate Hattie’s research into effective and meaningful practice.

   The critical premise here is that—just because we know from meta-analytic research that a program, strategy, or intervention significantly impacts student learning—we do not necessarily know the implementation steps that were in the research studies used to calculate the significant effect . . . and we cannot assume that all or most of the studies used the same implementation steps, or that these steps are the “right” ones for a specific district or school.

   Two “Top Twenty” examples (“Response to Intervention” and “Interventions for Students with Learning Needs”) are used to demonstrate how these steps could be used.
_ _ _ _ _ _ _ _ _ _

The Questions to Ask the Outside “Hattie Consultants”

   The Blog closes by identifying Five Questions that districts and schools need to ask outside “Hattie consultants”—who are now making themselves available to help “implement Hattie’s research and work”—BEFORE they sign the contract.

   As Hattie’s work has become more and more popular, we now have a “cottage industry” of “official and unofficial” Hattie consultants who are available to assist.

   With no disrespect intended, just because someone has been trained by Hattie, has heard Hattie, or has read Hattie—that does not give them the expertise, across all of the 138 (or more) rank-ordered influences on student learning and achievement, to analyze and implement any of the approaches identified through Hattie’s research.

   And so, the questions that districts and schools need to ask when consultants say that their consultation is guided by Hattie’s research are detailed.
_ _ _ _ _ _ _ _ _ _

Summary

   Once again, none of the points expressed in this Blog are about John Hattie.  Hattie has made many astounding contributions to our understanding of the research in areas that impact student learning and the school and schooling process.

   However, many of my points relate to the strengths, limitations, and effective use of research reports using meta-analysis and meta-meta-analyses.  If we are going to translate this research to sound practices that impact student outcomes, educational leaders need to objectively and successfully understand, analyze, and apply the research so that they make sound system, school, staff, and student-level decisions.

   And if the educational leaders are going to use other staff or outside consultants to guide the process, they must ask the questions and get the answers to ensure that these professionals have the knowledge, skills, and experience to accomplish the work.

   In the end, schools and districts should not invest time, money, professional development, supervision, or other resources in programs that have not been fully validated for use with their students and/or staff. 

   Such investments are not fair to anyone—especially when they become (unintentionally) counterproductive by (a) not delivering the needed results, (b) leaving students further behind, and/or (c) creating staff resistance to “the next program”—which might, parenthetically, be the “right” program.
_ _ _ _ _

   I hope that this discussion has been useful to you.

   As always, I look forward to your comments. . . whether on-line or via e-mail.

   For those of you still on vacation, I hope that you have been enjoying the time off.

   If I can help you in any of the areas discussed in this Blog, I am always happy to provide a free one-hour consultation conference call to help you clarify your needs and directions on behalf of your students, staff, school(s), and district.

Best,

Howie


[For the Entire Blog Message:  CLICK HERE]