Mark Rickinson

Image by Adobe Stock

It’s not just identifying quality evidence, it’s quality use of it that makes the difference

evidence-based practices, , , , , , , ,

Medical experts around the world are channeling rapidly evolving, and sometimes contradictory, research evidence to inform politicians and the public on the best way forward during this pandemic. A high level of expertise is required to navigate this torrent of information, determine the most appropriate evidence, communicate it, and help work out ways to apply it across diverse populations.

School teachers and leaders, like medical experts, demonstrate a similar expertise to determine appropriate evidence from a range of sources, including student data and research evidence, then adapt and apply it to inform decision-making, planning and implementation in diverse education settings.

Yet, how school educators access and use research evidence is still far from well understood. So at Monash University we have embarked on a large scale, five year project to investigate how teachers in Australia use research evidence to inform their practice, and to help educators who are interested in improving the quality of their use of evidence in their classrooms and schools .

Our project

Understanding how educators use research evidence is an emerging field of research in education and is at the heart of the Monash Q Project. Our research is a first for Australia.

We began by searching more than 10,000 scholarly records from databases across education, health, social work and policy, as well as over 100 documents and 65 organizational websites to understand what it means to use evidence well. We reviewed and synthesised this global knowledge and, coupled with regular feedback from project partners and stakeholders, used it as the basis for defining what quality use of evidence might be in education, and to develop a best practice framework for use of evidence in classrooms and schools.

We defined the quality use of research evidence in education as: the thoughtful engagement with and implementation of appropriate research evidence, supported by a blend of individual and organisational enabling components within a complex system.

Our framework describes the key characteristics of quality use of research evidence that are salient to education. It focuses on the quality of use of evidence as well as the quality of evidence.

Quality use of research evidence framework

Our framework is a resource for anyone interested in improving the use of research evidence within and across all levels of schools and school systems.

At the centre of our framework are two core components we believe are needed to use evidence well. The first is the ability to find and understand appropriate research evidence, and the second is to be able to thoughtfully engage with and implement the evidence.

The ability to identify appropriate research evidence

Our research indicated that being able to identify appropriate research evidence well involves, among other things, understanding the strengths and weaknesses of different forms of research evidence, as well as their potential and practicality to inform teaching and learning. 

According to a principal in a P-12 school in Queensland who was involved in our study, using appropriate research evidence well means

“considering the context of the research, and working out the extent to which the research applies to our local context and students”.

The ability to thoughtfully engage with and implement the evidence

Alongside the ability to identify appropriate research evidence is the ability to thoughtfully engage with and implement evidence. This involves engagement with the evidence, shared deliberation about its meaning and effective integration of aspects of evidence within practice. Our research indicated that to do this well includes questioning assumptions about the evidence within the context of practice, working collaboratively in professional learning communities, and working to adapt strategies over time. 

In our study a middle school leader in a Victorian High School emphasised,

“all teachers involved in implementing a program or practice that purports to be informed by research evidence would have sufficient professional learning time ALLOCATED to read, review, and critically analyse that evidence”.

The inter-dependencies of these two components of our framework are more nuanced than simply applying research to practice, particularly in highly variable contexts such as schools and classrooms. 

Cross sector insights

Our cross-sector research in health, social care and policy provided insights around evidence use, highlighting a central role for practitioner expertise in using evidence well. Practitioner expertise was characterised as the ability to apply external and practical knowledge in context – referred to as tacit and explicit knowledge. Far from just following the evidence, all sectors emphasised the need for such expert interaction with theevidence.

In one of the first and most enduring definitions of evidence-based medicine, American-Canadian physician and a pioneer in evidence-based medicine, David Lawrence Sackett, emphasised the need to balance external evidence with expertise:

“Without clinical expertise, practice risks becoming tyrannized by external evidence, for even excellent external evidence may be inapplicable to or inappropriate for an individual patient. Without current best external evidence, practice risks becoming rapidly out of date, to the detriment of patients.”

Practitioner expertise and professionalism

These ideas are relevant in the education sector, where educators have to navigate education research, some more fashionable than others, to determine what is most appropriate for their own students. With the growing expectation for Australian schools to use research evidence to underpin their improvement efforts, the challenge of building expertise (and confidence) to use evidence becomes more salient.

According to British educationalist, Professor Dylan Wiliam, “Evidence is important, of course, but what is more important is that we need to build teacher expertise and professionalism so that teachers can make better judgments about when, and how, to use research.”

The need to understand and support such expertise raises the question: How can practising ‘thoughtful engagement with and implementation of appropriate research evidence’ become part of educational professionalism?

As our experiences of the COVID-19 crisis have shown, evidence does not speak for itself but depends on careful decisions about whether, when and how to use and act on it in specific contexts, raising the question: How can quality use of research help us understand the potential and limitations of research evidence in responding to educational challenges?

Responding to the current pandemic requires growing our collective understanding of and respect for the level of expertise needed to negotiate an ever-changing body of knowledge and then apply it effectively in a rapidly unfurling context. In education, the Monash Q Project is working to better understand what this kind of expertise looks like among teachers and leaders in our schools.

Now, more than ever, a concerted effort is needed to understand, develop and support educators and schools to make better evidence-informed decisions to improve the quality of teaching and learning.

Connie Cirkony is a research fellow with the Q Project in the Faculty of Education at Monash University, investigating how educators use evidence in their practice. Connie’s background is in science and environmental education, and in educational practice and policy. Her research is focused on improving students learning experiences. Connie is on Twitter @ConnieCirkony

Lucas Walsh is Professor of Education Policy and Practice, Youth Studies, in the Faculty of Education at Monash University. He is currently a chief investigator on The Q Project (Quality Use of Evidence Driving Quality Education) funded by The Paul Ramsay Foundation. Recent books include: Imagining Youth Futures: University Students in Post-Truth Times (Springer, with Rosalyn Black), and Young People in Digital Society: Control Shift (Palgrave Macmillan with Amanda Third, Philippa Collin, and Rosalyn Black,.

Mark Rickinson is an Associate Professor in the Faculty of Education at Monash University. His work is focused on understanding and improving the use of research in education. He is currently leading the Monash Q Project, a five-year initiative with the Paul Ramsay Foundation to improve the use of research evidence in Australian schools.

Joanne Gleeson is a Research Fellow with the Q Project in the Faculty of Education at Monash University. Joanne draws from cross-sectoral professional experience in executive human resource management, business consulting, careers counselling, education and education research. Her research is focused on improving adolescents’ career identity, employability and education-work transitions. Joanne is on Twitter @dr_gleeson

Mandy Salisbury is a Research Assistant in the Faculty of Education at Monash University. She has worked in the early years and primary sectors in teaching and leadership roles, and also has commercial experience. Mandy has a passion for supporting teachers and pursuing equitable educational opportunities and outcomes.

Access the Monash Q Project’s Quality Use of Research Education Framework

Access the Discussion Paper “Towards Quality Use of Research Evidence In Education.”

Learn more about the Monash Q Project

Join the Twitter Conversation @MonashQProject

Readers are encouraged to connect with the Q Project and be part of strategic dialogue and system-level change around research evidence use in Australian education.

Image by Adobe Stock

National Evidence Institute: let’s push for quality of USE, not just quality of evidence

National Evidence Institute, , ,

The use of evidence in education has never had a higher profile. In Australia much interest in school evidence use has been sparked by the promise to develop an ‘independent national evidence institute’ as part of the recent National School Reform Agreement. It is how this institute will work that interests me.

The aim of the institute, as the agreement puts it, will be to make sense of what works in improving school outcomes and – most importantly – then translate this research into practical resources that can be used by classroom teachers and school leaders

As we are still in the planning stages, based on research on evidence use elsewhere in the world, I believe there are qualities and characteristics that educators could be pushing to help ensure the institute will work well for them.

1. We need a national evidence institute that focuses on use as well as evidence

Any evidence institute faces the risk of being drawn into expectations of just providing evidence rather than supporting how teachers and educational leaders will use that evidence. Arguably this latter process is far more important and difficult to establish.

The experiences of the What Works Centres in the UK are illustrative. A recent analysis of their work against their three main aims (generating evidence, translating evidence, and supporting evidence adoption), found that there was far less activity around the evidence adoption relative to evidence generation and translation.

 It seems, then, that it is all too easy for evidence centres to slip into ‘a research production (push) approach to the use of research, rather than problem-solving, demand-led (pull) approach’.

I believe in Australia there is an exciting opportunity for the Australian evidence institute to take a different approach, to learn from experiences in the UK and elsewhere, and articulate the explicit aspiration to be a national evidence use institute. 

2. We need a national evidence institute that supports quality of use as well as quality of evidence

A Monash University project designed specifically to improve the use of research evidence in Australian schools, the Q Project, has argued that discussions about quality in relation to evidence use have focused almost exclusively on the quality of the evidence, but not the quality of its use. While there has been long-standing debate about what counts as quality evidence, deliberation about what counts as quality use has been much more limited.

This situation is changing as awareness and understanding of ‘quality use’ starts to grow. Work within and beyond education is beginning to suggest that quality use involves not only appropriate, rigorous evidence but also thoughtful, critical use of that evidence within decision-making processes that are transparent and accountable.

In addition, it requires not only relevant skills and understandings but also inquiry mindsets and relationships of respect and challenge. Fostering the development of these kinds of characteristics therefore offers a distinctive opportunity for the Australian national evidence institute to be part of supporting not only high-quality evidence but also high-quality use.

As head of one of the What Works Centres in the UK, Sir Kevan Collins, reasoned recently: ‘Used intelligently, evidence is the teacher’s friend’.

Helping to work out what it means to use evidence intelligently in Australian schools and school systems needs to be a key priority for any new national evidence institute.

3. We need a national evidence institute that frames everything around improvement

Another key mission for a new national evidence institute should be establishing clarity about the relationship between using evidence and improving education. As Harvard professor, Carol Weiss, argued thirty years ago, educators would do well to stop thinking about ‘How can we increase the use of research in decision making?’ and focus instead on ‘How can we make wiser decisions, and to what extent, in what ways, and under what conditions, can social research help?’. 

These two questions are subtly (but significantly) different because they shift the focus from increasing the impact of research to supporting the improvement of practice, reminding us that evidence use is a means to an end, not an end in itself.

To me this suggests that the underlying purpose of a national evidence institute should be to support educational improvement and to make clear the distinctive contribution that using evidence can make to realising this aspiration.  Work in the US on improving the use of research evidence talks about ‘advancing the use of research evidence in ways that benefit youth’. The final five words of this statement are the most important ones, as they make clear the ‘So what?’ of evidence use. 

4. We need a national evidence institute that follows an ethos of ‘less is more’

In a world of information abundance it is increasingly being argued that it is most effective to focus ‘on a small number of carefully selected and optimised activities that strongly support things you value, and then happily miss out on everything else’ (as professor of computer science at Georgetown University in the US, Cal Newport, reminds us). This is potentially good advice when it comes to educational evidence use.

Indeed, there are a number of ways in which a national evidence institute would benefit from being intentionally minimalist or ‘less is more’ in its approach.

First, is the benefit of being very clear about the limits as well as the potential of evidence –as emeritus professor of educational assessment at University College London, Dylan Wiliam, argues, ‘Evidence is important, but what is more important is […] teacher expertise and professionalism [to] make better judgments about when, and how, to use research’.

Second, is the benefit of being clear which educational challenges to focus on – such as by developing processes to identify practice in need of evidence or by developing evidence-based guidance on high-priority issues with strong evidence but inconsistent practice.

Finally, is the benefit of using evidence to help people and organisations to know what to stop doing. In this respect, a national evidence institute needs to show leadership not only in focusing on ‘a small number of carefully selected and optimised activities’ but also in ‘happily missing out on everything else’.

Against the backdrop of calls for a research-rich teaching profession in Australia, the establishment of an independent national evidence institute represents a unique opportunity to better understand and support evidence-informed improvement in our schools. Let’s work together to realise this potential.

Mark Rickinson is an Associate Professor in the Faculty of Education at Monash University in Melbourne. Mark’s work is focused on improving the use and usefulness of educational research in policy and practice. He is currently leading a new 5-year initiative (The Q Project) to improve the use of research evidence in Australian schools. Mark is on Twitter @mark_rickinson

The above blog post is based on a presentation Mark gave as part of a panel discussion event about the National Evidence Institute on 26 July 2019 organised by the ‘Schools and Education Systems’ Special Interest Group (SIG) of the Australian Association of Research in Education (AARE).