Te Ara Whakamana: Mana Enhancement MoE Evaluation

AATEA Solutions presents this report on the evaluation of Te Ara Whakamana – Mana Enhancement Programme (Te Ara Whakamana) developed and delivered by Ako Solutionz (the developers). In December 2019, this evaluation was commissioned by the Ministry of Education (the Ministry). This was subsequent to earlier agreements and discussions between the Ministry, AATEA Solutions (the evaluators), and the developers that an appropriate evaluative methodology be developed to underpin this evaluation. The evaluation included the development of a methodology plan, kaupapa Māori research scan, evaluation methodology and evaluation of Te Ara Whakamana in schools. These activities were undertaken from December 2019 to October 2020.

Author(s): Kiwa Hammond, AATEA Solutions Ltd. Report for the Ministry of Education

Date Published: July 2021


Executive Summary

Background

The evaluation of Te Ara Whakamana was commissioned by the Ministry following a pilot evaluation conducted in 2017.1 The Ministry noted significant uptake of the model with 23 schools fully trained (whole staff), 13 schools partially trained (some staff), and Resource Teachers: Learning and Behaviour (RTLB) clusters and individuals have received training in Te Ara Whakamana.2 The Ministry also observed that ‘support for schools and RTLB clusters seeking to make informed decisions around the implementation of the model rests on the availability of a robust, independently commissioned evaluation’.3 This report seeks to fulfil that objective.

The report from the pilot evaluation advised the Ministry on the overall effectiveness of Te Ara Whakamana and included investigation of the potential for behaviour change and improved achievement for students with behavioural issues, improved teacher pedagogical practice with respect to managing behavioural issues, and teacher confidence in managing behaviour. The programme was ‘achieving overall effectiveness as a kaupapa Māori behaviour management programme which enhances tamariki Māori, improves student achievement, improves pedagogical practice, and builds teacher confidence’.4


Methodology

This evaluation used predominantly qualitative research methods with some quantitative data analysis provided by the Ministry. The Ministry, the evaluators, and the developers recognised that there were several complexities to developing an appropriate evaluative methodology, including:
  • Aligning the evaluation with Government priorities and the confirmed outcomes of the Child and Youth Wellbeing Strategy (released 29 August 2019).
  • The need for quality of evaluations undertaken and adhering to established Ministry guidelines, as outlined through the Ministry of Education Evaluation Hub.
  • A need to ensure the availability of school data pertinent to the evaluation could be drawn from a timeframe aligned with the school year.5
  • Reconciling the above with kaupapa Māori methodologies.

Developing Theory

Te Hōkai Rangi – Te Hōkai Nuku is the kaupapa Māori methodology created to underpin this evaluation. The methodology is informed by in-depth understanding of Māori worldviews and philosophies and the practical application of indigenous frameworks. The key evaluation questions were developed through a series of discussions between the evaluators, the Ministry, and the developers. The overarching evaluative question was:

  • To what extent does Te Ara Whakamana increase schools’ and the education system’s ability to enhance wellbeing for students, teachers, schools, and whānau and communities?
This question was contextualised with the following lines of enquiry:

  • What conditions enable Te Ara Whakamana to be implemented successfully?
  • How effective is Te Ara Whakamana as a teaching and learning tool? How has it enhanced teacher praxis and relationships?
  • How does Te Ara Whakamana integrate with other school and Ministry interventions within the school?
A Māori worldview-based Theory of Change was also developed for this evaluation. The Te Ara Whakamana: Theory of Change draws from the Te Hōkai Rangi - Te Hōkai Nuku methodology, previous research conducted by the evaluators and initial observations of Te Ara Whakamana implementation in schools. The elements and progressions acknowledge the widelyrecognised Poutama — a multi-levelled pathway to enlightenment, self-realisation, and levels of achievement sought and attained.6

The Theory of Change advanced into Poutama: Praxis of Change. The Poutama demonstrates three key elements that progress change and set conditions of success for Te Ara Whakamana.

Evaluation Participants

Initially, a non-random sample of five schools agreed to participate in the evaluation. These schools were representative of the wider network of Te Ara Whakamana schools. Due to circumstances beyond the control of the evaluators, and impacts of Covid-19 restrictions, two schools withdrew from the evaluation. To address this, two more principals (School 4 and 5) were engaged with the evaluation, though in-depth focus groups were not conducted due to time constraints and Covid-19 restrictions.

The participants were either full primary schools or contributing schools. The evaluation did not include secondary schools or kura kaupapa. The demographic descriptions of schools are shown in Table 1.

Table 1: School Demographics
Note:
  1. Source: Education Review Office.
SchoolRollMāoriPacificPākehāIndianChineseSchool Type
114084%-9%--Years 1 to 6
231474%19%7%--Years 1 to 8
378110%13%28%24%9%Years 1 to 6
47248%-20%--Years 1 to 8
54372%23%5%--Years 1 to 6

The evaluation involved in-depth focus groups and interviews with principals, teachers, students, Boards of Trustees, whānau at participating schools, and discussions with training workshop participants. In total, the evaluation team conducted 16 focus groups, six interviews, and nine pre-meetings. Only two in-person interview/focus groups were permitted due to Covid-19. The remaining interviews and focus groups were conducted online.

The evaluation activities also included:
  • Attendance, observation, and analysis of presentations and workshops at two 1-day Te Ara Whakamana UnConferences in March 2019 and March 2020.
  • Observations of four 1-day Te Ara Whakamana training sessions at Communities of Learning (CoL) meetings and customised events at various schools.
These sources combined have laid the foundation of the evaluation methodology, the development of a culturally appropriate theory of change, and compilation, analysis, and articulation of the data gathered. Other relevant research and literature specifically about Te Ara Whakamana, and Māori Wellbeing, was sourced, and this is referred to in the body of evidence presented in this report.7

Quantitative Data

The Ministry was asked to provide an analysis of trends emerging from schools that have engaged with Te Ara Whakamana. This information summarised data on school outcomes for primary schools participating in Te Ara Whakamana, in comparison to similar non-participating primary schools. The information was based primarily on the only measurements available at that time, that is, data provided by schools themselves regarding recorded incidents, stand-downs, suspensions, exclusions, attendance, and teacher turnover.

The Ministry’s analysis recognised that such measurements are inadequate for truly measuring the impact of Te Ara Whakamana but it is still necessary to understand the current baseline data available to identify emerging trends and measure the impact of Te Ara Whakamana. On review of the analysis of trends provided, the evaluators determined that this information added limited value to the overall evaluation.

However, this did highlight the importance and need for the design of quantitative and qualitative tools to measure the impact of Te Ara Whakamana on specific indicators of success like:

  • Student engagement and achievement.
  • Whānau engagement, and
  • Teacher and school effectiveness.

Summary of Findings

A thematic content analysis, informed by grounded theory, was undertaken to analyse the data. This method of analysis was used to identify themes, categories, and examples from the data to inform the findings. Four key themes emerged about the influence and effectiveness of Te Ara Whakamana on:

  • Wellbeing.
  • Cultural capability.
  • Education pedagogy.
  • Change and Transformation.

Wellbeing

Te Ara Whakamana positively impacts on four significant areas covered by the Wellbeing theme, in categories of emotional regulator, social regulator, relationship tool, and wellbeing model. Te Ara Whakamana supports students, teachers, schools, and whānau and communities to:

  • Develop understanding and language to communicate about themselves and their emotions which supports emotional regulation.
  • Establish meaningful relationships that enhance effective teaching philosophies and practices to improve educational outcomes.
  • Create a school environment that encourages student and whānau engagement and whole school community partnership in education and learning.
  • Develop a whole school wellbeing framework extending beyond a behavioural model that is culturally responsive and inclusive of the whole school community

Cultural Capability

Te Ara Whakamana has a significant positive cultural impact on whole school communities. The Cultural Capability theme includes categories of Māori wellbeing model and effective relationships. Te Ara Whakamana supports students, teachers, schools, and whānau and communities to:

  • Increase in knowledge, understanding and application of Māori knowledge, concepts, frameworks and theory to enhance teaching and learning.
  • Create a strong cultural context for all teaching and learning that is strengths based.
  • Enhance a schools’ cultural capability and responsiveness, and increase the use of te reo Māori, tikanga Māori and Māori knowledge in teaching and learning.
  • Engage with whānau, hapū and iwi and wider community to build positive relationships that support teaching and learning.

Education Pedagogy

Te Ara Whakamana has a significant positive impact on Education Pedagogy. The Education Pedagogy theme includes categories of learning tool, teaching practice, student empowerment, and transition. Te Ara Whakamana supports students, teachers, schools, and whānau and communities to:

  • Develop a whole school approach to improve wellbeing, teaching, learning and achievement.
  • Develop student-centred, personal, strength-based approaches to teaching and learning across the curriculum.
  • Encourage student self-efficacy and student empowerment in decision making about their learning.
  • Support student transition on three levels: transitional plan of learning, transition between classes and schools, and the transition from children to young learners.

Change and Transformation

Te Ara Whakamana can lead to positive change and transformation. The Change and Transformation theme, in categories of professional learning and development, funding and resources, and key conditions for success. Te Ara Whakamana supports teachers, schools, and whānau and communities to:

  • Engage in professional learning and development to build confidence and competence in implementing and localising Te Ara Whakamana in their schools.
  • Reflect on and challenge assumptions, attitudes and teaching practices that are not contributing to student wellbeing and positive educational achievement.
  • Recognize the potential role of Te Ara Whakamana in catalyzing transformational culture change and that financial investment is critical.
  • Establish and engage in wider Te Ara Whakamana networks of support across schools, including early education providers, education services, social services, health services and the wider community.

Recommendations

The recommendations are presented as opportunities for the Ministry to consider increased investment in Te Ara Whakamana to support:

  • Aotearoa New Zealand culturally responsive education.
  • Professional learning and development.
  • Resource development and on-going support.
  • Evidencing the positive impact and outcomes for all students.
  1. Investing in Te Ara Whakamana and supporting its implementation in schools will support Aotearoa New Zealand culturally responsive education.
  2. Investing in Professional Learning and Development will enable greater access and support for all teachers and schools in Aotearoa New Zealand.
  3. Investing in resource development and on-going support will increase the effectiveness and impact of Te Ara Whakamana which will lead to improved outcomes for all students.
  4. Investing in the design of quantitative and qualitative tools for measuring the impact of Te Ara Whakamana will support schools to gather data and report to the Ministry on programme effectiveness.

Full Evaluation

The full evaluation can be downloaded from the Education Counts website here: https://www.educationcounts.govt.nz/publications/learning-support/te-ara-whakamana-mana-enhancement-evaluation



Footnotes:

  1. AATEA Solutions (2017) Mana Enhancement Programme Evaluation: Final Report. Prepared for and submitted to the Ministry of Education. Please note that in the report named here Te Ara Whakamana was referred to as the ‘Mana Enhancement Programme’ or ‘MEP’. Since then, there has been a noted increase in the use of the name ‘Te Ara Whakamana’ regarding this model and so for the purposes of this report the authors have chosen to use that name in its entirety when referring to it
  2. Ministry in Education Services Statement of Work MOE06064_A.
  3. Ibid.
  4. AATEA/Hammond, 2017, p29; also, as cited by the Ministry in Education Services Statement of Work MOE06064_A.
  5. Ibid.
  6. AATEA/Hammond, 2017.
  7. Lovelock, M., (2020) Te Ara Whakamana: Mana Enhancement Framework in the mahi (work) of New Zealand Psychologists’. A thesis presented in partial fulfilment of the requirements for the degree of Master of Arts in Psychology at Massey University, Manawatu, New Zealand; also Smith, Tākirirangi author., Tinirau, Rāwiri, 1980- editor., Smith, Cherryl (Editor), editor. Whanganui, Aotearoa New Zealand: Te Atawhai o te Ao, [2019].