October 2010
Logic mapping: 
hints and tips

for better transport evaluations. 

Developed by the Tavistock Institute 	

Dr Dione Hills 

The Tavistock Institute
30 Tabernacle Street
London EC2A 4UE	
T
F
W	+44 (0)20 7417 0407
+44 (0)20 7457 0566 
tavinstitute.org	

	

 

Although this report was commissioned by the Department for Transport (DfT), the recommendations are those of the authors and do not necessarily represent the views of the DfT. While the DfT has made every effort to ensure the information in this document is accurate, DfT does not guarantee the accuracy, completeness or usefulness of that information; and it cannot accept liability for any loss or damages of any kind resulting from reliance on the information or guidance this document contains.

Table of contents 
1.	Logic mapping and its uses
1.1.	Summary and purpose of this guide
1.2.	What is logic mapping?
1.3.	Key components of a logic map
1.4.	The use of logic mapping
1.5.	Use of logic mapping in evaluation
2.	Step-by-stepguide to logic mapping
2.1.	Getting started
2.2.	Step one: identifying the issue
2.3.	Step two: identifying the final impacts
2.4.	Step three: identify outcomes
2.5.	Step four: identifying the outputs
2.6.	Step five: creating a list of inputs
2.7.	Step six: establishing the underlying rationale
3.	Using logic maps as part of an evaluation strategy
3.1.	Defining the evaluation questions
3.2.	Deciding what evaluation strategy to use
3.3.	Using logic mapping to establish evaluation criteria
3.4.	Using logic maps to identify data sources
3.5.	Using logic maps for analysis of findings
3.6.	Revising your logic map
3.7.	Using logic maps to communicate evaluation findings
Appendix A: Resources
Appendix B: Template for logic mapping

1.	Logic mapping and its uses
1.1.	Summary and purpose of this guide
This guide to logic mapping has been developed as an aid to the evaluation of transport interventions. It was developed as a practical resource to accompany the Department for Transport’s (DfT) Guidance for transport impact evaluations: choosing an evaluation approach to achieve better attribution with the aim of giving more in-depth advice on developing logic maps. One of the key principles in DfT's evaluation guidance is the idea that evaluation strategies would be strengthened if some initial logic mapping of the intervention being evaluated were undertaken. This document uses some of the experience in developing logic maps to highlight a number of key hints and tips and to set out the practical steps in creating and using logic maps. 
This guidance is intended to be accessible to a wide audience. Whilst it has been designed to assist transport evaluators, the techniques promoted within Chapter 2 of the guide will also be of interest to policy makers as well as local authorities and partnership organisations as a tool to support the planning and design of interventions. 
The three chapters take the reader through the following step-by-step processes, with tips, ideas, practical suggestions and common hurdles highlighted throughout. 
Chapter 1: Logic mapping and its uses
What are logic maps, what are their components and what is their purpose? 
Chapter 2: Step-by-step guide to logic mapping
How do you go about populating each element of a logic map?  Best practice, useful hints and tips.
Chapter 3: Using logic maps as part of an evaluation strategy
How can logic maps be used throughout the evaluation process?  E.g. defining the evaluation questions, deciding on the evaluation criteria and data sources or analysing the data. 
Examples and advice on where to turn for further guidance are also provided throughout and summarised in Appendix A. 
Evaluation planners are advised to read all three sections of this document before putting the logic mapping into practice to ensure the key issues are considered in the logic mapping process.
1.2.	What is logic mapping?
Logic mapping is not new: it is widely used in programme planning and, particularly in the field of transport, as part of the appraisal process for new interventions.  It is referred to by a number of different terms: “outcome mapping”, developing “programme logic", “intervention logic” and "programme theory” which all refer to similar processes (although there may be a difference of emphasis and focus – see section 1.4 below).  
What is relatively new, however, is the growing interest in use of logic mapping as a framework for enhancing the focus and robustness of evaluation activities. This approach is particularly recommended as part of a ‘theory based’ or 'Theory of Change’ approach to evaluation, but can also be valuable alongside other evaluation approaches. 
One way of describing logic mapping is to see it a systematic and visual way of presenting the key steps required in order to turn a set of resources or inputs into activities that are designed to lead to a specific set of changes or outcomes.
1.3.	Key components of a logic map
In order to achieve this, logic mapping requires you to identify and describe a number of key elements in your intervention. These typically include:
•	The issues being addressed and the context within which the intervention takes place
•	The inputs – resources and activities – required in order to achieve intervention's objectives 
•	Outputs (e.g. in terms of target groups to be engaged, roads built, products developed); 
•	Outcomes (i.e. short and medium-term results, such as changes in traffic flow levels and modal shifts); and
•	Impacts (i.e. long-term results such as better quality of life, improved health, environmental benefits etc).  

Figure 1: Components of an intervention logic map

Issues addressed and context in which it is taking place

What is invested, e.g. money, skills, people, activities 

What has been produced? 

Short and medium term
results 

Long- term outcomes 

Context 

Input

Output

Outcomes

Impact

Tip: The map is not the territory
Some people worry about the terminology used in logic mapping and whether they are using this correctly. For example, what is the difference between an input, output or outcome? Part of the difficulty is that the map is trying to illustrate something that is a continuous flow, and often an iterative process, in which outputs from one activity becoming the input to another. Dividing and labelling different steps is often quite an arbitrary exercise. Further confusion arises from the fact that different approaches to logic mapping use different terminology (e.g. calling outcomes impacts or vice versa).
It generally helps to remember that – like the London Underground map – a logic map is an abstract representation designed to help you to find your way, not an entirely accurate and detailed representation of reality. The key question to ask is ‘Is the logic map helpful in designing an evaluation or in communicating with others?’ not ‘Am I using the right terminology?’ or ‘Is this a truly accurate representation of reality?’.
Logic maps generally are ‘read’ from left to right, leading you through a time sequence from the initial concept, through implementation to short and longer term results. It follows a chain of reasoning of ‘if…then…:
Example of  ‘if…then’ reasoning 	
If	then	if	then	if	then	If	Then		
There is congestion due to heavy traffic at peak times.		Bus lanes put in together with better   information on bus services		Bus travel becomes a more attractive way of travelling to work 		A sufficient number of commuters change from car to bus travel 		Less congestion, better air quality, improved efficiency etc. 	

Visual representation of your logic model
A key element of logic mapping is the representation of these steps in a visual form, which might be a flowchart (which can be organized vertically or horizontally), table, or even a circular sequence.  A table based ‘template’ for drawing up a logic map has been provided in Appendix B, and additional examples and templates which can be used for the development of your own maps can be found at. http://www.uwex.edu/ces/pdande/evaluation/evallogicmodelworksheets.html

1.4.	The use of logic mapping
Logic mapping is widely used in the planning and design of new interventions, in the management and, increasingly, in the evaluation of interventions post implementation. A number of different ‘types’ of logic mapping can be identified in the literature – including those with an ‘outcome’ focus, those with an ‘activity’ focus and those with a ‘theory’ focus . ‘Modelling’ undertaken as part of the appraisal process of large scale transport schemes could also be seen as a type of logic mapping.
The type of logic modelling most widely used in evaluation is one which focuses on the underlying ‘theory’ of an intervention, recommended as part of what has become known as  ‘theory-based’ evaluation approaches.
1.5.	Use of logic mapping in evaluation 
Theory-based evaluation approaches were developed to address the challenge of gathering robust evidence about the success of complex community based interventions where an experimental research design could not be used. Most widely used in the UK are ‘Theory of Change’ and ‘Realistic evaluation’ strategies .
These evaluation approaches are particularly helpful in the evaluation of complex interventions, i.e. those in which several different actions are taking place at the same time, and where the links between the actions and their anticipated outcomes are not straightforward. They are also particularly useful when the objectives are likely to take some years to achieve or are very difficult to assess. Theory of Change logic mapping helps the evaluator to work out how progress has been made along an anticipated path towards the final impacts, even if it is not possible to gather evidence that this has been achieved.
Theory-based approaches also help focus an evaluation on those aspects of an intervention for which existing knowledge or evidence is limited. For example, although there is an evidence base showing a relationship between levels of car use and increases in levels of CO2, there is still much to be learned about the best ways of encouraging the public to change from car use to other, more sustainable, forms of transport.  
They can also be useful in exploring the underlying ‘mechanisms’ by which an action leads to a certain result (especially where this is unclear). It can also help where contextual factors may be very important – information that may be crucial when making decisions about further implementation – for instance knowing whether this intervention is suitable in a particular environment and whether changes need to be made to the basic design in order to improve its effectiveness. Both of these are central to Realistic evaluation strategies .  

2.	Step-by-step guide to logic mapping
This section outlines a series of steps that will enable you to draw up a logic map for your intervention. 
2.1.	Getting started 
Logic mapping might be used as part of the process of planning an intervention, or as part of designing its evaluation. If the intervention is already planned, then it can be done as a ‘paper’ exercise – based on a review of the intervention documentation. Or it might be based with interviews with some of the key organisations or individuals with a ‘stake’ in it.  (This could include funders, key policy makers, those involved in its management and delivery, and representatives of different groups affected by the intervention). 
In community programmes, in which the Theory of Change evaluation strategy was developed, it is recommended that key stakeholders work together on developing the logic map, as this ensures that all different views about the intervention, and how its aims and objectives will be achieved, are taken into account. This is a good way of engaging stakeholders in evaluation activities, and can provide a rich source of data concerning the underlying rationale for the programme.
Although logic maps are read from left to right, they are generally developed from right to left. In other words – it is best to start with the issue being addressed then consider the impact or change that intervention is intended to achieve (the objectives of the intervention) before working backwards through the steps required in order to achieve these objectives.

The issue addressed and the context in which it is located

What is invested, e.g. money, skills, people, activities 

What has been produced? 

Short and medium- term
results 

Long- term outcomes 

Step 2: articulate anticipated impacts

Step 3 to 6: 
Analysis of the logic of the intervention 

Step 1: identify the issue 

Context 

Input

Output

Outcomes

Impact

Tip: Running a logic mapping workshop
Invite key people involved in your intervention to a workshop meeting - you will need at least three hours. Explain that the purpose of the workshop is to draw up a ‘map’ of the intervention that will help in the design of an effective evaluation strategy.
Invite participants to describe all the issues that the intervention will address, encouraging them to explain the underlying rationale and considerations made about the broader social, political and economic context which the intervention will be set against (step one below). This could be done on a flip chart or written on pieces of paper that are then placed on a large sheet of paper, chart or wall. Issues are written down on the far left of the chart.
Invite them to do similar exercises for impacts – which are placed on the far right of the chart (step two below). Then ask them to write down what outcomes, outputs and activities that will be required in order to achieve these changes (steps two to five below). (Each can be written in different colours or using different coloured paper).
As each set of notes is completed, invite participants to explain their thoughts on how each step will lead to the next and what 'success' at each step would look like.
Link or cluster those issues, activities and outputs that appear to relate to different kinds of outcome. (E.g. tackling perceptions of buses being inconvenient is addressed by providing easy to read bus maps and timetables, reducing fear of crime is addressed by increasing use of security cameras).
One thing that can emerge is a difficulty in reaching a consensus about what are the desired outcomes and how these will be achieved.  However, such discussions can be useful in identifying different ideas about how the intervention works, as well as helping to ensure that everyone’s views will be represented in the evaluation plan - which in turn will increase interest in the evaluation results.
2.2.	Step one: identifying the issue
This step involves describing the problem that your intervention is attempting to solve or the issues that it will address. Also important is understanding the context in which this is located, including background factors which may be influencing the need for change. This step should help to articulate the rational for the intervention. 

Context 

Input

Output

Outcomes

Impact

The issue addressed and the context in which it is located

What is invested, e.g. money, skills, people, activities 

What has been produced? 

Short and medium- term
results 

Long- term outcomes 

Useful questions to ask to identify the context and issues
•	What are the stated objectives of this intervention?
•	What is the particular problem that has been identified – who has identified this, and why at this particular point in time? 
•	What is the evidence indicating that this is a problem?
•	What national transport policies does the intervention originate from and support? 
•	What sub-regional issues and priorities does the intervention originate from and support (transport, economic development, social inclusion objectives, health…)? 
•	What local issues and priorities does the intervention originate from and support (transport, local economic development, social inclusion objectives health, environment…)?
•	Are there any other contextual factors that may influence the ability of the intervention to achieve its outcomes and impacts?

Walk in to Work Out example
The Walk in to Work Out (WITWO) programme was set up in Scotland as a campaign to encourage more people to walk to work.

The context for this initiative included:
- Research indicating high levels of mortality from heart disease in the population 
- Evidence which showed a proven link in many cases with poor diet and lack of exercise
- National government objectives to increase physical activity (to improve health, reduce obesity)
- Sustainable transport policy objectives aimed at decreasing the use of cars and increasing travel by other means, particularly travel to work 
- Little established knowledge about how active commuting behaviour can be increased

Tip: Contextual factors
Contextual factors are an important part of the consideration of the issue being addressed, and what other factors might be influencing the decision to undertake an intervention (and are therefore placed in the first box on the left hand side of the map).

However, the context may also be changing during the life of the intervention, in ways that might influence the outcome. For example, changes in car tax or the price of petrol, or another big building development which had not been anticipated, may also be influencing travel behaviour as well as your own intervention. 
 
Some people put in a ‘context’ box along the bottom of their map to ensure that contextual changes of this kind are also tracked during the course of the evaluation.
2.3.	
Step two: identifying the final impacts
Typically the final impacts ‘box’ of a logic map reflects the overarching objectives of your intervention: what is ultimately hoped to be achieved. 
When this involves large scale investment, this may also incorporate national policy objectives such as reducing carbon emissions.  In the local or sub-regional context, it may reflect specific transport objectives incorporated in Local Transport Plans, or broader objectives such a enhancing the local economy.

Context 

Input

Output

Outcomes

Impact

The issue addressed and the context in which it is located

What is invested, e.g. money, skills, people, activities 

What has been produced? 

Short and medium- term
results 

Long- term outcomes 

 
Questions to ask in identifying impacts:
•	What is the intervention looking to achieve in the long-term? For instance: support the UK economy, contribution to climate change objectives or improved health of the population etc.   
•	What national or local policy objectives will this intervention address?
Example
WITWO programme impacts 
- Reduced mortality / morbidity from chronic heart disease 
- Improved mental health
- Reduced pollution
- Improved commuting environment 
- Increased number of population reaching physical activity targets 

Tip: Which impacts are most relevant?
Your stakeholders might identify a large number of possible ‘impacts’ of the programme or project you are evaluating. It is best to focus on the most important of these – the ones related to the primary issue or problem being addressed (identified in step one), or the particular policy or funding stream under which it is being funded. 

2.4.	Step three: identify outcomes 
The step involves identifying what short to medium-term outcomes (for example, in terms of changes in public behaviour, attitude or knowledge) will be required if the ultimate impacts identified in step two are to be achieved.

The issue addressed and the context in which it is located

What is invested, e.g. money, skills, people, activities 

What has been produced? 

Short and medium- term results 

Long- term outcomes 

Context 

Input

Output

Outcomes

Impact

Useful questions to ask in identifying outcomes:
•	What is the intervention looking to achieve in the short to medium-term?  (For instance: less congestion, raised awareness, partnership working, better skills, and change of attitude and / or behaviour).  
•	How would we know that we were ‘on course’ to achieve our final objectives?
•	What kind of changes (in terms of individual behaviour, or in the organisations involved) would we expect to see as a result of intervention activities?

In identifying the outcomes of your intervention it helps the evaluation if these are SMART: i.e. Specific, Measurable, Achievable, Realistic and Time bound. You might also want to begin to consider the RE AIM questions posed in the criteria section below (section 3.3). 
You might also want to use a ‘phased’ approach to outcomes, identifying both immediate and more intermediate outcomes (part way between ‘outputs’ and ultimate ‘impacts’. This was done in the walk in to work map (see page 17). 
Example:
Immediate and longer-term outputs identified in the WITWO project
- Increased knowledge of safe / feasible walking / cycling routes 
- Increased cognitive behaviour skills for active commuting
- Increased goal setting and individual behaviour change
- Enhanced motivation for active commuting

Medium-term outcomes in the WITWO project
- Reduced number or length of car trips
- Increased active commuting in study population
- Increased proportion of Scottish employers adopting / implementing WITWO
- Increase active commuting to work in Scottish population

Tip: Timescales and logic maps
While working out how long it will take to complete different steps in the logic chain is widely accepted when these are used for management purposes, it is often something overlooked in an evaluation. However, it is particularly important to understand how long it is likely to take to achieve certain types of outputs (how long will it take to roll out a particular intervention across a large geographical area), outcomes (how long does it take for levels of knowledge or types of behaviour to change) and impacts (at what point to you assess when the local economy has benefited, or air quality improved) if an appropriate assessment of their success or otherwise is to be made. 

The anticipated time scale between inputs, outcomes and impacts is also a factor to be taken into account in deciding the most appropriate evaluation strategy to use (see ‘Guidance for Better impact evaluation’ Chapter 5).

2.5.	Step four: identifying the outputs 
The next step is to identify what outputs are required in order to achieve these outcomes. Typically this involves identifying what actual activities (number, type, and frequency) will be delivered, what number of people or organisations will be engaged in these and what their characteristics are. In many cases it will also be relevant to get feedback on the quality of the intervention, for instance in terms of participants' feedback (e.g. how complete or accessible is the infrastructure provided, whether the promotional campaigns have been clear or whether training provided is appropriate or timely?)

The issue addressed and the context in which it is located

What is invested, e.g. money, skills, people, activities 

What has been produced? 

Short and medium- term results 

Long- term outcomes 

Context 

Input

Output

Outcomes

Impact

Useful questions to address when identifying outputs.
•	What activities will directly result from the intervention? E.g.: building new road or rail infrastructure; street furniture; delivering training; information or awareness campaigns; passing regulation; provision of public transport priority facilities; walking and cycling facilities; parking controls; travel plans introduced.
•	What participation will directly result from the intervention (who will be reached)? E.g.: types of transport users, partners, agencies, decision-makers, groups in society, areas of a specific town/city. 
•	What kind of response will people need to have to the intervention if it is to be successful (interested, satisfied, engaged)?

Example: 
Outputs from WITWO (alongside the longer-term outputs previously described)
- Number of active commuting packs produced
- Employers and study participants recruited 
- Monitoring / reinforcement of behaviour change
- Publication and dissemination of findings 
- Development and dissemination of national pack
Tip: Unanticipated consequences
One criticism made of evaluation approaches is that they place too much emphasis on assessing anticipated outputs, outcomes and impacts and fail to record other changes which might be equally beneficial – or possibly sufficiently negative to out-weigh the beneficial impacts – which were not specifically planned for or anticipated.

One way to address this is to add further boxes into your model in relation to outputs, outcomes and impacts which reflect such factors, to ensure that some account is taken of these. Your stakeholders may help you to identify these. 

2.6.	Step five: creating a list of inputs 
This step involves being as specific as possible about the various elements that will make up the programme – the resources invested, the staff required (including any training they will need), the partners or organisations to be engaged, the technology and activities that will be put in place.

The issue addressed and the context in which it is located

What is invested, e.g. money, skills, people, activities 

What has been produced? 

Short and medium- term results 

Long- term outcomes 

Context 

Input

Output

Outcomes

Impact

Useful questions to ask about inputs:
•	What financial resources are being invested in implementing the intervention?
•	What other resources are being invested? E.g. people and (partner) organisations, skills, equipment, technology (e.g. electronic road signs), research or appraisal, etc. 
•	What activities will be undertaken? E.g. new routes put in place, campaigns launched, and travel plans developed etc.
Example: 
Inputs in the WITWO project
-Development of the active commuting packs
-Outreach work to recruit employers and study participants 
-Monitoring / reinforcement of behaviour change

Tip: Inputs and activities
You might want to be quite detailed in terms of identifying all the different resources (staffing, funding, different organisations involved in delivery) and a long list of different activities in the ‘input’ stage of your logic map, or you might want to keep it quite simple – just the main inputs and actions taken. To some extent this will depend on whether the focus of your evaluation is on processes or impacts (see section 3.1 below). 

Please note that the distinction between inputs and outputs can be particularly confusing at this point. Generally speaking, inputs are what have to be put in place in order to produce outputs. However, you might also, in this context, like to go back to the ‘Tip’ in section 1.3: the map is not the territory. Step three of the 'Guidance for transport impact evaluations' shows a simplified logic map for the “Walk in to Work Out Programme” (WITWO)
2.7.	Step six: establishing the underlying causal logic 
Most plans – in the transport field as elsewhere – have both a set of stated objectives, and a number of less clearly stated assumptions about how a particular activity or intervention will lead to a particular change. In the transport field, some of these will be an assumed part of technical experience, while others will be embedded in guidance. A key task to undertake either while drawing up your logic map (when reviewing the rationale for the intervention), or once it is complete, is to begin to articulate some of the ‘if…then…’ links between different steps in the map. These may be stated in programme documentation or found in policy or guidance documents. Those which are less explicit may need to be identified through discussion with different stakeholders involved in, or influencing the design of, the intervention. Another useful task at this stage can also be to review the available research to identify what evidence already exists concerning specific links in the map.
Useful questions to consider when establishing the underlying causal logic:
•	Why do you believe that activity X will lead to output Y and/or outcome Z?
•	Does anyone have another explanation for why activity X would lead to outcome Z?
•	Is there any research evidence linking activity X and output Y, or output Y with outcome Z?
•	Will activity X always lead to outcome Z or only under some circumstances or with some target groups?
•	What might get in the way of activity X leading to outcome Y?
Tip: using logic maps to identify and test hypotheses 
A hypothesis can be defined as ‘a supposition made on the basis of limited evidence as a starting point for further investigation’. In other words, it sets out a proposition (or set of propositions) that will be tested through the collection of data – this is the basis of your evaluation design. The underlying causal logic identified through the mapping is one way of establishing a set of hypotheses for testing. For example: To test the hypothesis that people will only be motivated to take up active commuting if general advice and guidance is accompanied by some positive reinforcement it would be possible to make a comparison between a group of people provided with only advice and guidance, with another group that is provided with positive reinforcement as well.

3.	Using logic maps as part of an evaluation strategy
In this section we spell out in greater detail how you can use a logic map in the design of your evaluation.
3.1.	Defining the evaluation questions 
Logic maps can be very helpful in identifying a set of evaluation questions. These are the overarching questions that your evaluation will seek to answer (these are not the specific questions that might be asked of participants in activities). 
Example: Overarching evaluation questions from WITWO:
Can a cognitive behavioural intervention increase walking or cycling to work?

The evaluation questions will depend in part on the purpose for which you are undertaking the evaluation (for more information on different types of question addressing different evaluation purposes see the ‘Guidance for transport impact evaluations' Step 4). 
A process focus
For example, if you are undertaking your evaluation in order to improve your intervention as it is being implemented, then you might want to focus on process questions – which will lead to a particular interest on the context, inputs and outputs. 

Outcomes

Output

Impact

Input

Context 

The issue addressed and the context in which it is located

What is invested, e.g. money, skills, people, activities 

What has been produced? 

Short and medium- term
results 

Long- term outcomes 

Examples of process related questions
•	Have the activities addressed the issues identified?
•	What obstacles have been identified and how have these been overcome?
•	Which of the activities have been particularly successful? What examples of good practice can be identified?
•	Have the sections of the population targeted been effectively engaged (and in sufficient numbers)?
•	Is there anything we need to do to improve delivery?

An impact focus
On the other hand, if you are undertaking your evaluation in order to demonstrate to your funders or other stakeholders that you have been successful in achieving your aims, or in providing evidence of how effective a particular intervention has been, then you are likely to be particularly interested in evaluation questions relating to the outcomes and impacts elements of your logic map.

Context 

Input

Output

Outcomes

Impact

Long- term outcomes 

Short and medium- term
results 

What has been produced? 

What is invested, e.g. money, skills, people, activities 

The issue addressed and the context in which it is located

Examples of impact evaluation questions
•	Is this intervention a useful way of changing travel behaviour?
•	Is this programme something that should be implemented more widely?
•	Has this intervention represented good value for money?

3.2.	Deciding what evaluation strategy to use
The Guidance for transport impact evaluations describes three different approaches to ‘impact evaluations’: output, experimental and theory based. The guidance takes you through six steps in deciding which strategy is best for your evaluation, taking into account factors such as the nature, complexity and time scale of the intervention and the purpose of the evaluation. Mapping the intervention logic is step three and helps you to establish which factors are present in your intervention which may influence this decision.
3.3.	Using logic mapping to establish evaluation criteria
Another useful aspect of logic mapping is that it can help in the establishment of a set of success criteria.
This can be achieved by establishing specific, time limited targets for each step in the map. These targets may be those which have been: set out in an original business case / strategy paper, determined by a particular policy framework (e.g. a commitment to deliver a particular change), benchmarked against similar interventions elsewhere or through establishing a set of realistic targets in discussions with relevant stakeholders. 
It is useful to establish criteria for inputs and outputs as well as outcomes and impacts. Sometimes described as ‘process’ evaluation criteria, the former are particularly useful in establishing whether the intervention is on course to achieve its ultimate objectives. For example
•	we will produce and deliver door to door X many leaflets by X date
•	we will implement X miles of improved road surface by Y date
•	we will have X schools signed up and delivering the new road safety programme by Y date

Outcome and impact targets are generally set by translating broad objectives into more specific and measurable ones. For example
•	there will be an X% reduction in road traffic at junction Y by Z date
•	there will be an X% reduction in serious child casualties by Y date
•	there will be an X% increase in physical activity by Y date.

Tip: the RE-AIM framework 
One useful resource in establishing criteria is provided by the RE AIM framework, an American public health framework designed to encourage programme planners, evaluators, funders, and policy-makers to pay more attention to programme elements needed to ensure sustainable adoption and implementation of interventions. The title refers to the key elements of the RE-AIM framework, i.e.:
Reach the target population 
Effectiveness or Efficacy 
Adoption by target settings or institutions 
Implementation - consistency of delivery of intervention 
Maintenance of intervention effects in individuals and settings over time 

Useful question to use when applying the RE-AIM framework

Reach – How do I reach the targeted population with the intervention? (Or have I reached the target population?)
Efficacy – How do I know my intervention is effective?
Adoption – How do I develop organisational support to deliver my intervention?
Implementation – How do I ensure the intervention is delivered properly?
Maintenance - What are the long-term effects of the intervention?
3.4.	Using logic maps to identify data sources
Another invaluable use of logic maps is in the identification of what data or information you will require to assess progress towards your ultimate objectives. For this purpose we recommend the use of the template in Appendix B (or similar) adding in the column titles below:

Exercise: Mapping the information you need for different stages in your logic map 
Stage of project	If….
Initial
issue/ context	Then…
Rationale
for
intervention	Then….
Actions
taken	Then…
Short-term
Outputs	Then…
Interim
Outcomes	Then…
Long-term
objectives	
Information needed	Understanding of the contextual background for the intervention and measure the behaviours / targets before the delivery of the intervention (i.e. establish baseline)	Assumptions underpinning the intervention including articulating hypotheses about how the intervention will cause the intended change	Information about staffing, resources, activities planned	Information on activities undertaken, numbers and characteristics of people or organisations involved, feedback on quality of intervention	Data on changes in travel behaviour, or changes in attitude and knowledge, changes in organisation policies and practice	Changes in road use, air quality, environment, economic activity in target areas, health and quality of life etc
	
Information already available from existing sources 	Initial research, project proposal, appraisal, consultations, data local socio-economic conditions and existing transport network	Project plans, local and national  policy documents / strategy papers / business case,
existing research and evaluation evidence base	Regular financial monitoring, contractors reports, project management records	Ongoing traffic or accident monitoring,
project management records on delivery,
public transport patronage data, user surveys	Ongoing traffic or accident monitoring, user surveys 	Local traffic or accident statistics, community surveys etc.	
Gaps in existing data	No baseline data on behaviour of target population / beneficiaries	Unclear exactly how intervention is intended to change behaviour 	No data being collected on staff numbers or involvement of partners in  delivery, the types of challenges being faced during delivery and how these were overcome 	No data on numbers and characteristics of those targeted by / population using the intervention and how they compare to those not using the intervention 	No data on changes in behaviour, attitudes, knowledge in specific target groups and why or understanding why others have not changed behaviour	Data too general to identify changes which can be reliably attributed to the scheme	
New data to be collected (and methods to be used)	Survey of target population  / beneficiaries prior to implementation	Review behaviour change research / theory and draw on baseline data to understand current behaviours 	Bespoke monitoring of staff resources used and activities undertaken, qualitative research with stakeholders to understand the deliver process and how challenges faced were overcome	Bespoke monitoring of target population / beneficiaries and how they are engaging with the intervention	Post intervention survey of target group	Follow-up surveys / monitoring to track change in the longer-term	

3.5.	Using logic maps for analysis of findings
Analysis of data can be particularly difficult for those less experienced in evaluation methods. Again, the logic map, together with the evaluation questions you posed and the criteria you established, can provide you with a helpful framework or checklist that can assist you in this task.
Example of how logic map can be used to summarise data collected
	Actions taken	Short-term outputs	Interim outcomes	Long-term impacts	
Criteria established	What activities were delivered?	What was the % reach of target group / the numbers of users of the intervention?	Does the evidence indicate that the desired change has occurred? 	Has this led to changes in congestion, health, carbon etc?	
Data collected related to these criteria	Does the evidence show that the activities planned are now in place, within budget and of a suitable quality?
	Were the usage levels as anticipated? What is the reason for any variation?	What level of change has been observed? Can this be attributed to the intervention or have other factors influenced the change? Were there any unanticipated outcomes?	What conclusions can be made from the evidence that the outcomes have led to the anticipated impacts? 	
Was intervention successful and why / why not?	Was the intervention implemented in line with delivery targets? If not, what was the cause of any setbacks?	To what extent were the target population reached by the intervention / adopted by target organisations? 	Was the intervention effective in changing behaviour / attitudes / levels of knowledge? 
Did the intervention achieve targets for usage?	Are the outcomes maintained in the longer-term in order to deliver the anticipated impacts? 	
3.6.	Revising your logic map
Once you have completed your analysis, it is then advisable to revise your logic map in order to take account of the learning derived from the evaluation. This will now represent a new and better evidenced version of how an intervention of this kind can lead to certain outcomes and will be particularly useful for others wishing to consider and implement similar interventions in the future, as well as your own organisation in making amendments to policy and practice in the shorter term. 
Hint: revisiting the stakeholder workshop
It is particularly useful to go back to the stakeholders who helped to develop the logic map in the first place, with your ‘new’ version, and check with them whether the new version still makes sense to them. At this stage you will also be able to spell out more clearly some of the underlying rationale for different steps in the intervention, and provide evidence concerning the effectiveness (or otherwise) of different approaches to meeting the programme goals. 
3.7.	Using logic maps to communicate evaluation findings
Some evaluators use their logic map as a way of structuring their evaluation reports or the presentation of findings. The logic map provides the basis for providing a readable ‘story’ of the intervention as well as helping to ensure that elements often overlooked in evaluation reports are included (e.g. detailed information about the intervention itself, the context within which it was undertaken or the unanticipated effects). The logic map itself can also provide a helpful ‘visual aid’ to illustrate these various points.
Appendix A: Resources   
This guidance provides a basic guide to logic mapping. However, there are a number of other useful resources if you wish to go into this topic in greater depth, or if you find you are having difficulty with any of the steps outlined in the guidance. 

This guidance itself was produced for the Department of Transport as an additional resource to support the use of their guidance on better attribution: Guidance for transport impact evaluations: choosing an evaluation approach to achieve better attribution http://www.dft.gov.uk/pgr/evaluation/evaluationguidance/transportimpact/
Help with logic mapping and Theory of Change evaluation strategies

HELP WITH LOGIC MAPPING AND THEORY OF CHANGE	
Organisation	Details of support provided	Contact/
Web-link	
Theory of change.org	A website set up to support people using this evaluation methodology and has a number of useful resources including papers on its development and use in various contexts. They are currently developing (in pilot phase) an online logic mapping tool TOCO.	
http://www.theoryofchange.org/

The Kellogg Foundation	Provides a useful (and detailed) logic model development guide.	W. K. Kellogg Foundation - Logic Model Development Guide	
University of Wisconsin	Provides a number of resources including: 
A power point presentation with a step-by-step guide to logic mapping and different templates for creating a logic map.
	http://www.uwex.edu/ces/pdande/evaluation
	
RE AIM	Has a number of tools and resources related to the RE AIM framework described in section 3.4. This includes presentations and tools for calculating the effectiveness of reach, adoption and impacts.	RE-AIM.org	
Compendium Institute	An open forum for the ongoing development and dissemination of the methodology and software tools including logic mapping tools.	http://compendium.open.ac.uk/institute/index.htm
	
Tavistock Institute	Provides evaluation services including support to those undertaking their own evaluation activities (training, coaching, and facilitation of logic mapping workshops). 
	http://www.tavinstitute.org/work/research/evaluation.php	

Other resources
There are also many useful websites that provide guidance on how to undertake evaluation, including step-by-step advice on the use of different data collection methods. 
GUIDANCE ON EVALUATION METHODS & APPROACHES	
Organisation	Details of support provided	Contact/
Web-link	
Department for Transport	Guidance for transport impact evaluations 	http://www.dft.gov.uk/pgr/evaluation/evaluationguidance/transportimpact/	
Prove and Improve: a quality and impact toolkit   (New Economics Foundation)	This site is constructed to provide step-by-step guidance on evaluation: it includes how-to guides on measuring impact, practical guides on a range of tools and approaches you can use, and a downloadable library of measurement, evaluation and quality resources.  	http://www.proveandimprove.org/new/tools/index.php	
Evaluation support Scotland	Contains broad range of evaluation tools, tool-kits and guides on every aspect of evaluation, including an online step-by guide on evaluation called ‘Evaluation Pathway’.	http://evaluationsupportscotland.org.uk/evaluation/index.asp	
Social Research Association: (SRA)	Guidance on many aspects of research, including research ethics, commissioning research. 	http://www.the-sra.org.uk
	
UK Evaluation Society  (UKES)	The UKES run a selection of forums, training days and events on evaluation – ranging from introductory level to more specialist course.   	http://www.evaluation.org.uk/events.aspx 	

The following books and reports have also been referred to in the text
Blamey, A. and Mackenzie, M. (2007) “Theories of Change and Realistic Evaluation” Evaluation, Vol. 13 No. 4, pp. 439-455
Chen, H.T. (1990) Theory-Driven Evaluations, Thousand Oaks, CA: Sage
Connell, J. P. and Kubisch, A. C. (1998) Applying a Theory of Change Approach to the Evaluation of Comprehensive Community Initiatives: Progress, Prospects, and Problems. http://www.aspeninstitute.org/policy-work/community-change/community-change-evaluation 
Hills, D. and Junge, K. (2010) Guidance for transport impact evaluations: choosing an evaluation approach to achieve better attribution, DfT http://www.dft.gov.uk/pgr/evaluation/evaluationguidance/transportimpact/ 
Mutrie, N., Carney, C., Blamey, A., Crawford, F., Aitchison, T., & Whitelaw, A., (2002) 'Walk in to Work Out’ Journal of Epidemiology and Community Health, 56(6), 407-412.
Pawson, R. and Tilley, N. (1997) Realist Evaluation, London: Sage 
Appendix B: Template for logic mapping
A	B	C	D	E	F	
If….
Initial
issue/ context	Then…
Rationale
for
intervention	Then….
Actions
Taken, resources required	Then…
Short-term
outputs	Then…
Interim
outcomes	Then…
Long-term
impacts