Hattie Haters and Lovers: Both are Still Missing the Effective Implementation Steps that Practitioners Need

Critical Questions to Ask your “Hattie Consultant” Before You Sign the Contract

 [For the Entire Blog Message:  CLICK HERE]

Dear Colleagues,

Introduction

   Over the past five years especially, John Hattie has become internationally-known for his meta-analytic research into the variables that most-predict students’ academic achievement (as well as other school-related outcomes). 

   Actually, his different Visible Learning books (which have now generated a “Hattie-explosion” of presentations, workshops, institutes, and “certified” Hattie consultants) report the results of his “meta-meta-analyses.”  These are analyses that average the effect sizes from many other meta-analyses that themselves have pooled research that investigated the effect of one psychoeducational variable, strategy, intervention, or approach on student achievement.

   Let me be clear from the start.  This Blog is not to criticize or denigrate, in any way, Hattie on a personal or professional level.  He is a prolific researcher and writer, and his work is quite impressive.

   However, this Blog does critique the statistical and methodological underpinnings of meta- and meta-meta-analytic research, discuss its strengths and limitations, but most essentially, delineate its research-to-practice implications and implementation necessities.

   It also reviews a recent critique by Dr. Robert Slavin, the primary developer of Success for All, an evidence-based literacy (and math) program that is one of the longest-standing, best-researched, and most-effective instructional approaches in recent history. 

   On June 21, 2018, Slavin published a Blog, John Hattie is Wrong, where he reported his analyses of and concerns about Hattie’s research and conclusions. 

   Peter DeWitt, a Hattie colleague and trainer, then responded to Slavin in a June 26, 2018 Education Week article, John Hattie Isn’t Wrong. You Are Misusing His Research.  This Blog included quotes from Hattie in response to Slavin.

   And then, Slavin responded back in an Education Week Letter to the Editor published on July 17, 2018.

_ _ _ _ _

   My current Blog quotes from all three of these pieces so that the issues surrounding meta-analyses and meta-meta-analyses are clearly delineated.

   In this end, it is important that educators understand:

·      The strengths and limitations of meta-analytic research—as well as meta-meta-analytic research;

·       What conclusions can be drawn from the results of sound meta-analytic research;

·       How to transfer sound meta-analytic research into actual school- and classroom-based instruction or practice; and

·       How to decide if an effective practice in one school, classroom, or teacher is “right” for your school, classrooms, and teachers.

   But there is more. . .

_ _ _ _ _ _ _ _ _

My Addition:  The Implementation Methods are Missing

   The second half of this Blog identifies Hattie’s current (June, 2018) “Top Twenty” approaches showing the strongest meta-meta-analytic effects on student learning and achievement.

   Significantly, many of these “Top 20” approaches are different than when I wrote a previous Blog message regarding Hattie’s work on September 25, 2017 Blog—less than 10 months ago.  Second, there are new approaches on the list that have never previously been cited. . . and other approaches have new labels. 

   Finally, and most importantly, many of the approaches have such generic or global names that it is virtually impossible to determine, at a functional school, classroom, teacher, or student level, what methods and implementation steps were used in the research, and what methods and steps should be used, right now, by a district or school interested in implementation.

   This science-to-practice implementation issue is what none of the researchers are really talking about. 

   And, with the ever-changing list of Top Twenty effects appearing to be a moving target, districts and schools have an additional dilemma of trying to keep up with and accurately interpreting the research.

_ _ _ _ _

   The remainder of the Blog discusses a step-by-step approach that districts and schools need to take to translate Hattie’s research into effective and meaningful practice.

   The critical premise here is that—just because we know from meta-analytic research that a program, strategy, or intervention significantly impacts student learning—we do not necessarily know the implementation steps that were in the research studies used to calculate the significant effect . . . and we cannot assume that all or most of the studies used the same implementation steps, or that these steps are the “right” ones for a specific district or school.

   Two “Top Twenty” examples (“Response to Intervention” and “Interventions for Students with Learning Needs”) are used to demonstrate how these steps could be used.

_ _ _ _ _ _ _ _ _ _

The Questions to Ask the Outside “Hattie Consultants”

   The Blog closes by identifying Five Questions that districts and schools need to ask outside “Hattie consultants”—who are now making themselves available to help “implement Hattie’s research and work”—BEFORE they sign the contract.

   As Hattie’s work has become more and more popular, we now have a “cottage industry” of “official and unofficial” Hattie consultants who are available to assist.

   With no disrespect intended, just because someone has been trained by Hattie, has heard Hattie, or has read Hattie—that does not give them the expertise, across all of the 138 (or more) rank-ordered influences on student learning and achievement, to analyze and implement any of the approaches identified through Hattie’s research.

   And so, the questions that districts and schools need to ask when consultants say that their consultation is guided by Hattie’s research are detailed.

_ _ _ _ _ _ _ _ _ _

Summary

   Once again, none of the points expressed in this Blog are about John Hattie.  Hattie has made many astounding contributions to our understanding of the research in areas that impact student learning and the school and schooling process.

   However, many of my points relate to the strengths, limitations, and effective use of research reports using meta-analysis and meta-meta-analyses.  If we are going to translate this research to sound practices that impact student outcomes, educational leaders need to objectively and successfully understand, analyze, and apply the research so that they make sound system, school, staff, and student-level decisions.

   And if the educational leaders are going to use other staff or outside consultants to guide the process, they must ask the questions and get the answers to ensure that these professionals have the knowledge, skills, and experience to accomplish the work.

   In the end, schools and districts should not invest time, money, professional development, supervision, or other resources in programs that have not been fully validated for use with their students and/or staff. 

   Such investments are not fair to anyone—especially when they become (unintentionally) counterproductive by (a) not delivering the needed results, (b) leaving students further behind, and/or (c) creating staff resistance to “the next program”—which might, parenthetically, be the “right” program.

_ _ _ _ _

   I hope that this discussion has been useful to you.

   As always, I look forward to your comments. . . whether on-line or via e-mail.

   For those of you still on vacation, I hope that you have been enjoying the time off.

   If I can help you in any of the areas discussed in this Blog, I am always happy to provide a free one-hour consultation conference call to help you clarify your needs and directions on behalf of your students, staff, school(s), and district.

Best,

Howie

[For the Entire Blog Message:  CLICK HERE]

Elementary School Principals’ Biggest Concern: Addressing Students’ Behavior and Emotional Problems

The Solution? Project ACHIEVE’s Multi-Tiered, Evidence-Based Roadmap to Success

[For the entire Blog Message, CLICK HERE]

Introduction

   While most students and staff are off on vacation and not thinking about homework and teaching, many administrators are still on the job.  And without the daily focus on immediate tasks, to-do’s, and trouble spots, these administrators often use their summer “simmer” time thinking about ways to resolve their most pressing problems.

   In order to gain some insight into these “problems,” the National Association of Elementary School Principals (NAESP) recently conducted its once-every-ten-year survey of a representative sample of elementary school principals across the country.

   [CLICK HERE for a Link to the Report]

  The most-relevant survey question here asked respondents to separately rate their concerns—along a five-point scale from “Extreme” to “None”—with over 20 different student-related areas.

   The Table below identifies those areas where 50% or more of the principals rated their concerns at the top of the scale—as either “Extreme” or “High.”  The NAESP synthesized a number of these items, as well as some additional highly-rated items from this question, and concluded that:

Students’ social, emotional, and behavioral status was “the top-ranked concern for 2018 responding principals. . . (including) addressing the increase of students with emotional problems. Among those issues identified were the management of student behavior, student mental health issues, absenteeism, lack of effective adult supervision at home, and student poverty. In contrast, none of the student-related issues were identified as a major concern in 2008.”

_ _ _ _ _ _ _ _ _ _

Addressing Principals’ Most Pressing Concern: 

Project ACHIEVE’s Evidence-Based, Multi-Tiered Social-Emotional Learning/Positive Behavioral Support System

   In order to directly address the most-pressing concerns voiced in the NAESP survey, this Blog describes Project ACHIEVE’s Social-Emotional Learning/Positive Behavioral Support System (SEL/PBSS).  Project ACHIEVE’s SEL/PBSS is an evidence-based model (through the U.S. Office of the Substance Abuse and Mental Health Services Administration) that has been implemented in thousands of schools across the country with consistent and clearly documented student, staff, and school success.

An Overview of Project ACHIEVE

   Project ACHIEVE is a comprehensive preschool through high school continuous improvement and school effectiveness program that has been implemented in urban, suburban, and rural districts across the country since 1990.  Project ACHIEVE was recognized by the U.S. Department of Health & Human Service’s Substance Abuse and Mental Health Services Administration (SAMHSA) as an evidence-based model prevention program in 2000.  Its effectiveness has also been recognized by the U.S. Department of Justice’s Office of Juvenile Justice and Delinquency Prevention (OJJDP, 2003); the Collaborative for Academic, Social, Emotional Learning (CASEL, 2002); and other national, regional, and state groups. 

   Project ACHIEVE is now listed on SAMHSA’s National Registry of Evidence-based Programs and Practices (NREPP), and its implementation blueprints, procedures, and strategies are embedded into its well-defined components that maximize school and district success across an assortment of ESEA/ESSA and IDEA-required outcomes.

   Between 2003 and 2015, Project ACHIEVE was implemented on a statewide basis in Arkansas through the Arkansas Department of Education’s (ADE) State Improvement and State Personnel Development grants (SIG and SPDG, respectively).  These grants were awarded to the ADE by the U.S. Office of Special Education Programs (OSEP). 

   Directed by the author, these grants resulted, during that time, in Project ACHIEVE being designated as the ADE’s primary school improvement model for all of the Focus schools in the state—through its successful ESEA Flexibility application with the U.S. Department of Education.  Concurrently, Project ACHIEVE was used as the Department and state’s official PBIS and MTSS approaches.

_ _ _ _ _

Project ACHIEVE’s SEL/PBSS System

   Because the same psychologically-based science of individual, group, and systems-level behavior, that targets and results in students’ social, emotional, behavioral, and mental health self-management, is at the foundation of its practices, Project ACHIEVE’s SEL/PBSS system addresses a host of proactive goals and problematic conditions in a unified and integrated way.

   This means that districts and schools implement a single evidence-based model, rather than (a) multiple, potentially-competing (as well as time-consuming, exhausting, and expensive) unaligned programs, or (b) frameworks that lack sound science-to-practice implementation guidance, or that encourage people to “choose-your-own-strategies”—a recipe that often results in chasing symptoms and not real problems, or choosing activities by convenience and not causality.

   As such, some of the most important areas addressed by Project ACHIEVE’s integrated, science-to-practice SEL/PBSS system are:

·       School safety and prevention,

·       Positive school culture and classroom climate,

·       Classroom discipline and management,

·       Student engagement and self-management,

·       Social Skills training and teaching 21st Century SEL/Soft Skills

·       Productive student interactions in cooperative and project-based groups,

·       Student trauma and trauma-sensitive practices,

·       Teasing and bullying,

·       Harassment and physical aggression,

·       Chronic student absences and school/class tardiness,

·       Office discipline referrals and suspensions/expulsions,

·       Disproportionality and retiring zero tolerance policies, and

·       Preventing and responding to students’ mental health status and needs.

   To accomplish this integration, Project ACHIEVE’s multi-tiered SEL/PBSS approach starts by helping teachers with classroom management, student engagement, and the development of an explicit student motivation and behavioral accountability matrix.  Added to this is a social skills curriculum where teachers teach students, from preschool to high school, the interpersonal, social problem-solving, conflict prevention and resolution, and emotional control and coping skills that they need for individual, peer, classroom, and school success.

   When students do not respond, need additional help, or require more strategic or intensive attention, the multi-tiered system uses data-based functional assessment approaches to determine the underlying causes of students’ challenges, and links the results to the implementation of relevant services, supports, strategies, and interventions.  These are implemented along a continuum involving classroom-based consultation by related services staff (e.g., counselors, school psychologists, special education teachers, social workers) through individual cognitive-behavior therapy by  school-based mental health professions.

   This entire system involves a whole school approach that focuses on positive, safe, proactive, supportive, and consistent school climates and settings; and building school and district capacity such that the entire process is embedded in everyone’s day-to-day interactions and the continuous improvement process of the school and district. 

   In these ways, virtually all of the NAESP principals’ highest concerns are simultaneously addressed in an effective and efficient way—without years of training and waiting for the most-needy students to receive services.

_ _ _ _ _

Project ACHIEVE’s SEL/PBSS Goals

   The ultimate SEL/PBSS goal is to maximize all students’ social, emotional, and behavioral competence and self-management.  Simultaneously, there are a number of complementary student, staff, and school goals. 

   These goals are guided by a scaffolded preschool through high school Social-Emotional Competence, and Physical-Mental Health-and-Wellness scope and sequence that is created and individualized by each district and its schools at the beginning of the SEL/PBSS initiative.

   Ultimately, success depends on seamlessly coordinated whole-district and whole-school approaches—embedded in their respective and ongoing continuous improvement processes—that include students, staff, administration, and parents working together to build and reinforce (a) positive, safe, supportive, proactive, and consistent school climates and settings; and (b) school and district capacity, competence, and independence. 

   As noted above, the SEL/PBSS system involves the following broader comprehensive goals:

Student Goals:

   Student social, emotional, and behavioral competency and self-management as demonstrated by:

·       High levels of effective interpersonal, social problem solving, conflict prevention and resolution, and emotional coping skills and behaviors by all students;

·       High levels of critical thinking, reasoning, and social-emotional application skills and behaviors by all students; and

·       High levels of academic engagement and academic achievement for all students.

Staff Goals:

·       High levels of effective instruction and classroom management across all teachers and instructional support staff; and

·       High levels of teacher knowledge, skill, and confidence relative to analyzing why students are academically and behaviorally underachieving, unresponsive, or unsuccessful, and to implementing strategic or intensive academic or behavioral instruction or intervention to address their needs.

School Goals:

·       High levels of the consultative resources and capacity needed to provide functional assessment leading to strategic and intensive instructional and intervention services, supports, strategies, and programs to academically and behaviorally underachieving, unresponsive, or unsuccessful students;

·       High levels of parent and community outreach and involvement in areas and activities that support students’ academic and social, emotional, and behavioral learning, mastery, and proficiency;

·       High levels of positive school and classroom climate, and low levels of school and classroom discipline problems that disrupt the classroom and/or require office discipline referrals, school suspensions or expulsions, or placements in alternative schools or settings; and

·       High levels of student success that result in high school graduation and post-secondary school success.

_ _ _ _ _

The SEL/PBSS Blueprint, Science-to-Practice Components, and Comprehensive Outcomes

   Project ACHIEVE’s SEL/PBSS strategies are implemented in a series of carefully-sequenced steps that typically occur over a four-year period, and are embedded in a district’s and schools’ strategic planning processes and ongoing school improvement activities. 

   Critically, the time and the activities in the four-year blueprint are only guideposts.  Some districts or schools are able to accomplish their SEL/PBSS activities with a different sequence of steps and activities and in less time.  Others take more time.  The four-year timeframe, however, is geared more to building the capacity of the district and its schools so that they can continue without outside consultation or support.

   Critically, students who need immediate social, emotional, behavioral, or mental health services or supports are addressed immediately in the needs assessment process, and then the initial implementation activities.

   The remainder of the Blog describes, in detail, the SEL/PBSS Implementation Blueprint, its five interdependent Science-to-Practice Components, a number of the related implementation activities, and the comprehensive outcomes that districts and schools have achieved when the process is implemented with integrity and strategic intensity.

[For the entire Blog Message, CLICK HERE]

Summary

   The National Association of Elementary School Principals’ survey gives us some important insight into the most-pressing concerns in our schools today.  Critically, my interactions, consultations, and visits with schools across the nation at the secondary level tell me that they have the same concerns about students’ social, emotional, behavioral, and mental health status.

   The bottom line to all of this, however, is that districts and schools need to use evidence-based science-to-practice systems that have been field-tested and flexibly applied in diverse settings and situations across the country. . . 

. . . that are implemented on-site with consultants (not just district-employed school staff who have been trained by “the experts” for a handful of days) who are experienced in the multi-tiered services, supports, and interventions needed by their current students.

   We hope that you read the entire Blog message describing this model—one that effectively addresses the most significant concerns of principals across the country.

   Many districts and schools are funding Project ACHIEVE’s SEL/PBSS implementation with a blend of ESEA/ESSA (Chapter IV and I, especially), IDEA, and state funds.  We are happy to discuss your district or school needs at any time by phone or videoconference.

   What do you think?

[For the entire Blog Message, CLICK HERE]