Te Ara Whakamana: Mana Enhancement MoE Evaluation AATEA Solutions presents this report on the evaluation of Te Ara Whakamana – Mana Enhancement Programme (Te Ara Whakamana) developed and delivered by Ako Solutionz (the developers). In December 2019, this evaluation was commissioned by the Ministry of Education (the Ministry). This was subsequent to earlier agreements and discussions between the Ministry, AATEA Solutions (the evaluators), and the developers that an appropriate evaluative methodology be developed to underpin this evaluation. The evaluation included the development of a methodology plan, kaupapa Māori research scan, evaluation methodology and evaluation of Te Ara Whakamana in schools. These activities were undertaken from December 2019 to October 2020. Author(s): Kiwa Hammond, AATEA Solutions Ltd. Report for the Ministry of Education Date Published: July 2021 Executive Summary Background The evaluation of Te Ara Whakamana was commissioned by the Ministry following a pilot evaluation conducted in 2017.1 The Ministry noted significant uptake of the model with 23 schools fully trained (whole staff), 13 schools partially trained (some staff), and Resource Teachers: Learning and Behaviour (RTLB) clusters and individuals have received training in Te Ara Whakamana.2 The Ministry also observed that ‘support for schools and RTLB clusters seeking to make informed decisions around the implementation of the model rests on the availability of a robust, independently commissioned evaluation’.3 This report seeks to fulfil that objective. The report from the pilot evaluation advised the Ministry on the overall effectiveness of Te Ara Whakamana and included investigation of the potential for behaviour change and improved achievement for students with behavioural issues, improved teacher pedagogical practice with respect to managing behavioural issues, and teacher confidence in managing behaviour. The programme was ‘achieving overall effectiveness as a kaupapa Māori behaviour management programme which enhances tamariki Māori, improves student achievement, improves pedagogical practice, and builds teacher confidence’.4 Methodology This evaluation used predominantly qualitative research methods with some quantitative data analysis provided by the Ministry. The Ministry, the evaluators, and the developers recognised that there were several complexities to developing an appropriate evaluative methodology, including:
Developing Theory Te Hōkai Rangi – Te Hōkai Nuku is the kaupapa Māori methodology created to underpin this evaluation. The methodology is informed by in-depth understanding of Māori worldviews and philosophies and the practical application of indigenous frameworks. The key evaluation questions were developed through a series of discussions between the evaluators, the Ministry, and the developers. The overarching evaluative question was:
This question was contextualised with the following lines of enquiry:
A Māori worldview-based Theory of Change was also developed for this evaluation. The Te Ara Whakamana: Theory of Change draws from the Te Hōkai Rangi - Te Hōkai Nuku methodology, previous research conducted by the evaluators and initial observations of Te Ara Whakamana implementation in schools. The elements and progressions acknowledge the widelyrecognised Poutama — a multi-levelled pathway to enlightenment, self-realisation, and levels of achievement sought and attained.6 The Theory of Change advanced into Poutama: Praxis of Change. The Poutama demonstrates three key elements that progress change and set conditions of success for Te Ara Whakamana. Evaluation Participants Initially, a non-random sample of five schools agreed to participate in the evaluation. These schools were representative of the wider network of Te Ara Whakamana schools. Due to circumstances beyond the control of the evaluators, and impacts of Covid-19 restrictions, two schools withdrew from the evaluation. To address this, two more principals (School 4 and 5) were engaged with the evaluation, though in-depth focus groups were not conducted due to time constraints and Covid-19 restrictions. The participants were either full primary schools or contributing schools. The evaluation did not include secondary schools or kura kaupapa. The demographic descriptions of schools are shown in Table 1.
The evaluation involved in-depth focus groups and interviews with principals, teachers, students, Boards of Trustees, whānau at participating schools, and discussions with training workshop participants. In total, the evaluation team conducted 16 focus groups, six interviews, and nine pre-meetings. Only two in-person interview/focus groups were permitted due to Covid-19. The remaining interviews and focus groups were conducted online. The evaluation activities also included:
These sources combined have laid the foundation of the evaluation methodology, the development of a culturally appropriate theory of change, and compilation, analysis, and articulation of the data gathered. Other relevant research and literature specifically about Te Ara Whakamana, and Māori Wellbeing, was sourced, and this is referred to in the body of evidence presented in this report.7 Quantitative Data The Ministry was asked to provide an analysis of trends emerging from schools that have engaged with Te Ara Whakamana. This information summarised data on school outcomes for primary schools participating in Te Ara Whakamana, in comparison to similar non-participating primary schools. The information was based primarily on the only measurements available at that time, that is, data provided by schools themselves regarding recorded incidents, stand-downs, suspensions, exclusions, attendance, and teacher turnover. The Ministry’s analysis recognised that such measurements are inadequate for truly measuring the impact of Te Ara Whakamana but it is still necessary to understand the current baseline data available to identify emerging trends and measure the impact of Te Ara Whakamana. On review of the analysis of trends provided, the evaluators determined that this information added limited value to the overall evaluation. However, this did highlight the importance and need for the design of quantitative and qualitative tools to measure the impact of Te Ara Whakamana on specific indicators of success like:
Summary of Findings A thematic content analysis, informed by grounded theory, was undertaken to analyse the data. This method of analysis was used to identify themes, categories, and examples from the data to inform the findings. Four key themes emerged about the influence and effectiveness of Te Ara Whakamana on:
Wellbeing Te Ara Whakamana positively impacts on four significant areas covered by the Wellbeing theme, in categories of emotional regulator, social regulator, relationship tool, and wellbeing model. Te Ara Whakamana supports students, teachers, schools, and whānau and communities to:
Cultural Capability Te Ara Whakamana has a significant positive cultural impact on whole school communities. The Cultural Capability theme includes categories of Māori wellbeing model and effective relationships. Te Ara Whakamana supports students, teachers, schools, and whānau and communities to:
Education Pedagogy Te Ara Whakamana has a significant positive impact on Education Pedagogy. The Education Pedagogy theme includes categories of learning tool, teaching practice, student empowerment, and transition. Te Ara Whakamana supports students, teachers, schools, and whānau and communities to:
Change and Transformation Te Ara Whakamana can lead to positive change and transformation. The Change and Transformation theme, in categories of professional learning and development, funding and resources, and key conditions for success. Te Ara Whakamana supports teachers, schools, and whānau and communities to:
Recommendations The recommendations are presented as opportunities for the Ministry to consider increased investment in Te Ara Whakamana to support:
Full Evaluation The full evaluation can be downloaded from the Education Counts website here: https://www.educationcounts.govt.nz/publications/learning-support/te-ara-whakamana-mana-enhancement-evaluation Footnotes:
|