ANZTSR blog post – Matthew Hall

Beyond frameworks – key questions to address in measuring social impact For third sector organisations (TSOs), when it comes to measuring social impact, there is certainly no shortage of ready-made frameworks.1 This is not surprising as frameworks offer structure and guidance, and come with the promise of glossy reports, benchmarked data and the assurance stemming from compliance with a set of standards and procedures. And having a ready-made framework can be very helpful in cash strapped TSOs without the resources for customised approaches. But frameworks can have their downsides. They tend to focus on producing reports rather than the more mundane but fundamentally important task of rigorous data collection and measurement. They also seek to standardise, as this brings with it the promise of comparable data and the ability to benchmark. Ready-made frameworks are also often an add-on to pre-existing measurement and reporting processes, and so are typically not well integrated with ongoing strategic and operational decision making.
And most importantly, frameworks tend to obscure reflection on fundamental questions at the core of any social impact measurement process. These questions relate to what social data to collect and track, what forms of data will count as knowledge about social impacts, and which stakeholders will be involved in the process and how. Any social impact measurement process raises questions about what social data to track, how, and how often. Answers to these questions provide a structure for a TSO’s social book-keeping system. Unfortunately, frameworks do not often begin with social book-keeping. Rather, data tracking tends to be shaped to fit the needs of the reporting format of specific approaches. This means less attention focused on ensuring high quality and cost-effective data collection systems, including the use of innovative ways to capture a wide variety of data about social impacts on stakeholders, such as selfreporting through custom built apps, online surveys and platforms, or virtual meetings and focus groups.
Developing any social bookkeeping system means deciding on what will count as knowledge about social impacts. This includes decisions about what types of data will be permissible. For example, can social impact data be collected in words and stories, or only in numbers and dollars? It also raises a question about the required level of rigour and certainty of the data. For example, what standards will be used to assess the quality of the data, and what will be the threshold levels for data to meet in order to be included in the social bookkeeping system?
For many TSOs, social impacts are wide ranging and often unpredictable, and so are not easily reducible to a set of fixed, standardised, quantified metrics only. There is often a need for flexibility in forms of representation, recognising that some social impacts may best be represented in words and stories, others in quantified metrics, and perhaps others in dollars where appropriate. We should be wary of frameworks advocating an increasing quantification of social impacts, recognising that not all social impacts are equally amenable to being captured and expressed in numbers. Instead of focusing on quantification, a social impact measurement system could prioritise inclusivity or representativeness, potentially accepting a wide variety of information to ensure important social impacts are not excluded. This can sometimes generate criticisms over a claimed lack of reliability associated with narratives and stories to the extent they should not be included in any (so-called) rigourous social impact measurement process.
But there is always a flipside to this focus on rigour, which can be thought of as the potential for errors of exclusion. These errors relate to the costs associated with excluding important social impact information simply because it cannot (yet) be quantified.
Any social impact measurement process also raises questions about whose criteria of effectiveness are to be employed in measuring and reporting on social impact. For example, will the focus be on the providers of funds or a broader set of stakeholders and constituents? There is also the question of which stakeholders are to be involved and how. If the social impacts of TSOs are distributed among different stakeholders, and are consequential, then the measurement and reporting of social impacts should involve and focus upon the perspectives of those stakeholder groups. This is more democratic in the sense that it involves those stakeholders affected by the organisation. But it is also likely to produce a social impact measurement process that is more informative.
This is because if a TSO has consequential social impacts on a variety of stakeholders, then a full accounting of and thus understanding of its social impact is severely limited if such stakeholders are excluded from the measurement and reporting process. But involvement can be a slippery word. On the one hand, stakeholders can be ‘consulted’ yet have no real power over the choice of what social impacts to account for and how – a process that is more akin to a sham ritual. On the other hand, a genuinely inclusive process might involve stakeholders having veto rights over social impact reports, or even be involved directly in setting up data collection systems and reporting frameworks used by the TSO.
Whilst the use of ready-made frameworks for measuring and reporting on social impacts is likely to continue, there is reason to pause to consider how any framework addresses fundamental questions arising in social impact measurement. These questions might include: How is data on social impact to be captured in a social bookkeeping system? What types of data are permissible and how might these requirements exclude important social impacts from view? What is the precise role of different stakeholder groups and who exactly gets to decide the measures and procedures to be used in the social impact measurement process?

1 Ebrahim, A, and Rangan, V.S. (2014). What impact? A framework for measuring the scale and scope of social performance. California Management Review, 56(3), 118-141.