Introduction

• Focus on how to improve the measurement and evaluation of non-profits through multi-dimensional frameworks (Bagnoli & Megali, 2010)

  • Balanced scorecards (Kaplan, 2001)

  • Linkages to strategic decision making (LeRoux & Wright, 2010)

• Field is often cluttered with new ideas, novel approaches and the latest toolkits (Jacobs et al., 2010)

• Analysing the ideals of different evaluation approaches can advance understanding of performance measurement and evaluation practice and theory in the nonprofit sector in 3 ways:

  • Directs attention to the normative properties of performance measurement

    • Can guide decision making (Friedland & Alford, 1991)

  • Focuses attention on how different evaluation logics can privilege different kinds of knowledge and methods of knowledge generation

    • Stakeholders are increasingly demanding that evaluation information be quantitative in nature and directed towards the impact o f non profit organizations (McCarthy, 2007)

  • The role of expertise in the evaluation of non-profit organizations

    • Particular conceptions of expertise and ‘valid’ information can serve to elevate the interests of certain actors in non-profit organizations whilst disenfranchising others (Ebrahim, 2002); Greene, 1999)

Analysis of evaluation techniques

Logical framework

• Developed throughout the 1970s as a planning and evaluation tool primarily for use by large bilateral and multi-lateral donor organizations

• 4x4 matrix

  • Vertically, the project is translated into a series of categories

    • Inputs

    • Outputs

    • Purpose

    • Goal

  • Horizontally, the categories are described using:

    • Narrative summary

    • Objectively verifiable indicators

    • Means of verification

    • Important assumptions

• Designers and proponents were Rosenberg and Posner (1979) of Practical Concepts Incorporated

• Developed in response to two perceived problems with existing evaluation approaches

  • 1) Viewed as unclear and subjective

  • 2) Evaluations were sites for disagreement that was considered unproductive

• LFA views the projects as a “set of interlocking hypotheses: if inputs, then outputs; if outputs, then purpose” (RP, 1979)

  • Chain of causality

• Evaluation based on evidence

  • OVIs were “means for establishing what conditions will signal successful achievement of the project objectives”

  • Prove to outsiders that results have indeed been achieved

    • Avoid “misunderstanding or … different interpretations by those involved in the project” (RP, 1979)

• Detailed specification of objectives and indicators has two effects:

  • 1) Remove any possibility for conflict and discussions are thus “constructive”

  • 2) Provides clarity that being blamed for issues apparently outside one’s control is no longer possible

• Very little evaluation required

  • A simple comparison of actual results to pre-set standards

• Real expertise relates to designing projects, not their evaluation

Most significant change

• Developed in 1990s as an approach to evaluating complex social development programs

• Regular collection and interpretation of stories about important changes in the program

• Used to evaluate programs in both developing and developed economies

• Principal designers and proponents were Davies and Dart

• Developed in response to several perceived deficiencies in existing evaluation practice

  • 1) Evaluations were focused on indicators that are “abstract” (DD, 2003)

  • 2) Do not provide “a rich picture of what is happening [but an] overly simplified picture”

    • Developments reduced to a single number

  • 3) Existing approaches focus on examining intended rather than unintended changes

• No comparison to a pre-defined set of objectives

• The evaluators task is to encourage people to write stories, to help with their selection and to motivate and inspire people

  • Skills required are different to LFA evaluators

• MSC is overtly political

  • Disagreements and conflict are encouraged

Social return on investment

• SROI developed in the 1990s as an approach to analyzing the value created by social enterprises

  • Roberts Enterprise Development Fund

• Envisioned a set of SROI metrics along with organization data, project descriptions and case studies of participant experiences

• Predominantly used in developed economies

• REDF (2001) distinguishes “Transactive Philanthropy” from “Investment Philanthropy”

  • Problem with TP is “there is often no real connection made between the dollars one provides non-profit organizations and the social value generated from that support”

  • An IP approach makes stronger connections between the provision of dollars and social value created

• Key focus of the SROI is quantifying

  • i.e. “returning” to the community

    • Index of Return is established

• SROI metrics make two connected translations:

  • 1) Effects of the activities of a social enterprise are expressed in a financial return

  • 2) Measure of return, along with a measure of investment, is used to make a further translation into an index

• Analysis requires certain skills

  • Financial literacy, an affinity with Excel and an understanding of the time value of money and DCF

• SROI also encourages provision of contextual information

  • But this is viewed as supplemental and “being behind the results”

• Seeks to adjust outcomes to reflect the influence of outside factors

  • Adjusted for “deadweight” and things that would have happened anyway

    • By using control groups

• Evaluator is seen as a researcher

  • Independent and objective (NEF, 2007)

Outline of evaluation logics

• Logics of evaluation are the broad cultural beliefs and rules that structure cognition and shape evaluation practice in non-profit organizations (Friendland & Alford, 1991)

• Evaluation...

Buy the full version of these notes or essay plans and more in our Accounting in Non-Governmental Organisations Notes.