Evaluating Diet-Related Policy
Published on 22 March 2010 in Food, health and wellbeing
Introduction
Public health policies and community programmes are being implemented to try and improve the diet and health of the population. It is important, however, to know what impact they are having on health, and understand which elements of them are working. Policies tend to be complex with interacting elements and, unlike many research trials, external influences often cannot be controlled. For example, a research trial might investigate the impact of reducing fat in the diet on body weight, whereas a policy is more likely to include a range of actions all at once (e.g. increase fruit and vegetable intake, reduce sugary drinks and increase exercise, to promote a healthy weight).
This means that there will be multiple outcomes occurring at different times which make evaluating the impact of policy more complex. In addition, it may take several years before an impact on health can be seen, so it is important to identify early indictors which will show the direction of travel. Developing an evaluation framework during the planning stage of a programme can help identify early indicators as well as set realistic expectations of what public health policies can deliver within a specified timeframe.
Key Points
- Evaluating the impact of policy is essential, but given the complexity of assessing this at a population level, a different approach is needed to give an early indication of the direction of travel.
- Evaluation should be an integral part of the planning stage of the programme, allowing enough time and resources for baseline data to be collected before it is implemented.
- There are several methods for planning an evaluation framework, including techniques such as concept mapping and logic modelling. These methods help design the evaluation, identify short-, medium- and long-term outcomes and estimate the time required to detect a change in the outcomes.
Research Undertaken
Research at the Rowett Institute of Nutrition and Health was carried out to develop an evaluation timeline for evaluating diet-related policy. The implementation of the Schools (Health Promotion and Nutrition) Act (Scotland) 2007 in primary schools was used as a model to develop the timeline. The Act has several elements including setting food and nutrient requirements for all food and drinks provided in school, aiming to increase the uptake of school meals and reduce the stigma of free school meals, placing health promotion at the heart of a school’s activities and requiring local authorities to consider sustainable development when procuring food and drink.
Concept mapping is a stakeholder driven approach for planning an evaluation. This project engaged with a diverse group of stakeholders including policy makers, researchers, local authorities, NHS, schools and parents. Thirty eight stakeholders representing these groups initially participated in a brainstorming session, focus groups or one-to-one interviews where they generated the range of likely outcomes resulting from the implementation of the Schools Act. The outcomes generated included both positive and negative outcomes, as well as highlighting possible unintended consequences (i.e. increasing food waste as a result of children not eating the food prepared in schools). In total they generated over 300 possible outcomes, but removing duplication left a final set of 85 to be used in the next stage of the concept mapping process. Using an on-line web-based system a larger group of 111 stakeholders completed the next tasks of sorting the outcomes into higher level themes and rating the outcomes on three parameters: likelihood of the outcome occurring, the importance of it occurring and the anticipated earliest time to impact, using a 5 or 7 point rating scale.
Overall, the 85 outcome statements were sorted into 13 themes: long-term health, food literacy, adverse economic consequences, dietary intake, food culture, local food economy, diet choice, educational benefits, undesirable eating behaviours, well-being, impact on the family and attitudes towards food. The ratings highlight differences between the importance and likelihood of outcomes occurring as well as the anticipated earliest time to impact. ‘Long-term health’, for example, was rated as highly important but less likely to occur and had the longest on the time to measurable impact (>5 years). This compared to ‘adverse economic consequences’ which was rated as less important than health but more likely to occur and would be seen earlier after the implementation of the Act (<2 years). Ratings of the anticipated time to impact were used to develop a timeline to indicate how long it is likely to take after the implementation of the Act before changes might be seen.
Policy Implications
To understand and improve population health it is essential to evaluate the impact of public health and diet related policies. This means setting in place an evaluation framework that includes short-, medium- and long-term outcomes. Relevant short-term outcomes are important to feed back into the policy cycle to give an indication of whether the policy is heading in the intended direction. Concept mapping is one technique for designing an evaluation that can be used for complex programmes where there are multiple interacting elements.
It has been described as a method for ‘simplifying complexity’. A strength of this approach is that it combines knowledge of the evidence and effectiveness of interventions from those working in research with the experience of people working in the communities who have an insight to the practicalities of running these programmes in the ‘real world’. This combination provides a more realistic perspective of the likely outcomes and timeline.
Author
Dr Jennie Macdiarmid j.macdiarmid@abdn.ac.uk
