Australia’s Indigenous evaluation strategy

The Indigenous Evaluation Strategy embraces the overarching principle that First Nations peoples, perspectives, priorities and knowledges need to be front and centre in evaluations that concern them and the Strategy Guide provides useful information on how to do this.

But, having worked for more than 10 years with the Central Land Council (CLC) to help them monitor and evaluate their Community Development Program (CDP), we think the documents don’t go far enough in recognising monitoring and evaluation (M&E) as a challenging, non-linear, time-consuming and deeply political process. It would also benefit from more practical guidance. The strategy guide includes only one reference to checking whether Indigenous people can participate in evaluations on an equal footing and makes no reference to evaluation politics. The literature increasingly points to engaging with worldviews and power differences to co-create Indigenous evaluations that acknowledge merit in multiple perspectives (see here and here). This is hard to do and organisations thinking of embarking on evaluating Indigenous programs should be prepared for the challenge.

Take the following program administered under the CLC’s CDP. Set up in 2005, the Warlpiri Education and Training Trust (WETT) is governed by Warlpiri landowners and educators (including co-author Napaljarri). WETT has so far invested over $30 million of gold mining royalties in education and training programs that benefit four remote Warlpiri communities in the Tanami Desert, plus extensive community engagement and governance processes. WETT M&E has been facilitated by the CLC since 2009, informed by the staff (including co-author Danielle) belief that it would support improved process, outcomes and accountability to Indigenous people and help demonstrate that supporting Indigenous locally-led development is more effective than centralised, top-down programming.

Over the last decade the CLC, WETT and the Institute for Human Security and Social Change (where all co-authors now work) have experimented with many different evaluative approaches and methods in trying to assess WETT process and impact. Three things we’ve learnt stand out.

First, because Indigenous and non-Indigenous stakeholders operate from fundamentally different worldviews, we don’t necessarily value the same outcomes. Non-Indigenous stakeholders are generally interested in education, training, employment and Indigenous capacity development outcomes. Warlpiri aspirations include these objectives, but they are underpinned by a broader set, including maintenance of Warlpiri culture, language and connection to country, respectful relationships and self-determination.

Second, views on the purpose of M&E, the type of information that constitutes quality evidence and the best methods to generate it differ. Everyone agreed that accountability to Indigenous people was the primary intention but CLC staff were also focused on a secondary accountability to the Australian Government. This was due to their ultimate control of resourcing through the Commonwealth legislation that provides for royalties and the payment of CLC operational costs. On top of regular opportunities for Warlpiri feedback and reflection through WETT’s CD process, formal M&E methods and processes were adopted. This included interviewing Warlpiri decision-makers and participants, focus group discussions with CLC staff, and analysing project reports and expenditure. But CLC staff worried that this may not be rigorous or ‘objective’ enough to convince external stakeholders so explored other approaches including an ‘evidence-based’ participatory capacity development framework. The WETT committee enjoyed the first workshop but saw it as too time consuming and disconnected from their WETT work to repeat and it was shelved. In another example, the CLC looked into doing a cost-benefit analysis but did not proceed (the required data wasn’t available and the methodology limited local decision-making and control).

These attempts took the M&E away from the Warlpiri focus on mutual accountability between WETT decision makers, landowners, communities, project partners and the CLC. Warlpiri prefer methods that support relationality and the co-creation of WETT knowledge through sharing stories, interviews and collaborative paintings.

We have our own Yapa (Warlpiri person) ways of watching closely to assess how things are going, giving encouragement and feedback so things can improve, and sharing information through stories and paintings. But Kartiya (non-Indigenous people) don’t see this or maybe they don’t understand it.

(Marlkirdi Napaljarri)

Third, you can’t simply ‘mix’ evaluation processes that work well for Indigenous and non-Indigenous stakeholders. This ignores power dynamics that can skew evaluation and privilege certain ways of assessing programs. The earlier focus on government priorities, using forms of evidence valued by governments was a pragmatic approach – and accountability to Warlpiri was happening in other ways – but it was at the expense of setting up a co-created evaluation approach designed to meet everyone’s needs. On reflection we made the mistake of approaching evaluation as a technical process and an add-on to WETT, which was otherwise Warlpiri-led.

A new 18-month WETT funded project brings the Institute, CLC and WETT together to co-create an evaluation framework that will try to weave together different priorities and knowledge systems. The Ground Up Monitoring and Evaluation Approach being developed in Northern Australia and the Indigenous Canadian concept of “two-eyed seeing” – bringing together the strengths of Indigenous and non-Indigenous ways of knowing – are proving helpful. We are paying close attention to differences in power and perspectives, including that of the Institute’s non-Indigenous staff, to deliberately build a shared approach.

This is slow and complex work. It builds on 15 years of M&E learning about what does and doesn’t work. It draws on long-term relationships grown advocating for WETT in meetings from Lajamanu to Canberra and working together on the ground in the Tanami. It involves experienced Warlpiri researchers and WETT champions with a deep understanding of the local context and non-Indigenous researchers with knowledge of the strengths and limitations of different evaluative forms, who are committed to valuing Indigenous knowledge and shifting power. Valuing and navigating different knowledge systems is not something that can be workshopped, written into an evaluation plan and delivered. In our experience, it’s facilitated by a shared commitment to ‘two-eyed seeing’, trusting relationships which allow for frank cross-cultural conversations, and endless learning and adaptation. The fact that Warlpiri are funding the evaluation design work for the first time has also sharpened the focus on Warlpiri priorities.

We are still in the talking, trialling and refining phase but we all agree that approaching evaluation as a complex, political process that needs careful navigation and co-creation seems more likely to generate a rich picture of WETT’s ongoing journey and achievements for all stakeholders. The Indigenous Evaluation Strategy is right to call for centring Indigenous people in evaluation, but it presents it as a pretty technical and apolitical process. It would benefit from more examples of the challenging reality of bringing together different knowledge systems and more detail on what it takes to support good Indigenous evaluation in practice.