Measuring Impact: recommended resources #ORblog

O.R. in the Third sector

Measuring impact: sources of information

1. There is an enormous amount of material available on this topic. If your project requires impact measurement, outcomes measurement or evaluation, you should first review as much of the material below as you have time for – the charity may well be aware of it, even if you aren’t. CES and NPC, in particular, are well known in the charity world and are partially responsible for the recent increased importance attached to impact measurement and evaluation.

Outcomes, impact, evaluation

2. Some organisations use these terms interchangeably, some don’t – there is not much consistency. A useful guide to the terminology is produced by the Big Lottery Fund; worth reading if only because you may have been asked to undertake the analysis to support an application for funding to Big Lottery or to another funder that takes a similar approach. www.biglotteryfund.org.uk/er_impact_measurement.pdf
3. Optional extra reading, or for reference if you feel the need: http://www.jargonbusters.org.uk/

Charities Evaluation Services http://www.ces-vol.org.uk/tools-and-resources

4. CES have useful material on planning for monitoring and evaluation, and outcomes and outcome indicators
5. They discuss purpose of evaluating impact:

5.1  reflect on and develop work

5.2  report to funders

5.3  identify gaps in services

5.4  bid for new work

and have a nice template (used by a Pro Bono O.R. volunteer carrying out a project for Elfrida   Rathbone Camden) demonstrating impact and effectiveness.

Outcome	Outcome indicators	Data collection	When and who by	How this will be used

Impact measurement and ‘theory of change’

6. The charity/think-tank New Philanthropy Capital (NPC) has been instrumental in promulgating the importance of impact measurement.  Its methodology, based on the idea of a ‘theory of change’ (a mapping of actions to outputs and outcomes which should be a familiar way of looking at the world to an O.R. professional), is now widely accepted. It is described in: http://www.thinknpc.org/publications/npcs-four-pillar-approach/
7. For more tools:  http://www.thinknpc.org/blog/7-practical-outcomes-tools/
8. To read about different approaches in practice: http://socialvalueint.org/wp-content/uploads/2015/01/SIAA-Challenge-Report_2014.pdf

Social return on investment (SROI)

9. SROI is a principles-based method for measuring extra-financial value (i.e., environmental and social value not currently reflected in conventional financial accounts) relative to resources invested. It can be used by any entity to evaluate impact on stakeholders, identify ways to improve performance, and enhance the performance of investments.  Useful introductions are:

9.1 http://socialvalueuk.org/

9.2 http://en.wikipedia.org/wiki/Social_return_on_investment

9.3 http://evpa.eu.com/wp-content/uploads/2010/09/SOCIAL-EVALUATOR-SROI-an-introduction.pdf

9.4 http://evpa.eu.com/publication/a-practical-guide-to-measuring-and-managing-impact-2015/

HMT Green Book: https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-governent

10. The Green Book is a guide to best practice appraisal (before investment) and evaluation (post-implementation) of public sector initiatives, which applies equally well to any initiative where there is no market value of the outcome.
11. It covers:

11.1   Valuing costs and benefits where there is no market value (value, utility, welfare, well-being)

11.2   Weighting and scoring

11.3   Critical success factors (met or not)

11.4   Discounting

11.5   Risk management

11.6   Valuing risks (uses ‘expected value’)

11.7   Performance management and measurement (tracking the success of a policy, programme or     

          project in achieving its objectives and in securing the expected benefits)

11.8   Benefits categories:

11.9   Financial quantitative

11.10  Non-financial quantitative

11.11  Non-financial qualitative

11.12  Outcomes qualitative and quantitative

11.13  Objectives and outcomes; outputs and targets.

11.14  Social Cost Benefit Analysis which seeks to assess the net value of a policy or project to

           society as a whole

11.15  Utility: the satisfaction a person gets from consumption of a good, or to a change in their  welfare or well-being

Specific types of charity

12. If your charity works with offenders or their families, you should be aware of Clinks: http://www.clinks.org/support-evaluation-and-effectiveness/demonstrating-outcomes
13. If your charity works with young people, you should be aware of Project Oracle: www.project-oracle.com  and http://inspiringimpact.org/listing/project-oracle/
14. If your charity is involved in overseas aid or development, you should be aware of the Logframe approach (“Logical framework”, which you can find out more about via google): http://www.usip.org/sites/default/files/PW77.pdf  and https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/253889/using-revised-logical-framework-external.pdf

Pro Bono Economics  http://www.probonoeconomics.com/

Pro Bono Economics matches volunteer economists with charities to address questions around measurement, results, impact and value. Volunteers produce a full, peer-reviewed report on each project, and all reports are published on the website, constituting a valuable resource.