PhD research: ‘Evaluations of sustainability projects must and can be more to the point’

Irrelevant questions, too many criteria and limited standpoints. According to RUG researcher Frans Sijtsma , a lot can go wrong with evaluations of large-scale sustainability projects.‘It’s impossible to do it completely right, but things must and can be much more to the point.’

Example: cockle fishing in the Wadden Sea . For four years this was evaluated and then stopped as a result of the findings. ‘The fishing was for an international market. Although it’s mainly the Spanish who eat cockles, the report concentrated only on the Netherlands.’ Sijtsma thinks that a global investigation would have led to different conclusions.

Global scale

Sijtsma: ‘In the same period, sixteen illegal Chinese immigrants were killed while fishing for cockles by hand in the UK, and the Dutch are now trying to transfer their fishing to the coast of Mauritania, which also has beautiful natural resources. The Dutch cockle fishing industry was a very well-organized sector, with very good boats, all of which were monitored. You have to look at the wider picture – if the Spanish continue to eat cockles, they’ll have to come from somewhere.’

An evaluation of sustainability – an assessment of the effects on economic development, extreme poverty and a deteriorating environment – are too often examined only at a local level. This is one of Sijtsma’s main criticisms of current analysis methods. ‘Of course it’s impossible to always get it right. Ideally you’d like to trace all the effects but you can’t. Everything is linked – trade with China, CO ₂emissions on a global scale. But it should all be much more to the point, and it could be.’

Simplicity and clarity

In his thesis, Project evaluation, sustainability and accountability, Sijtsma introduces a new analysis method for sustainability projects. In it he combines two of the most important current methods – the cost-benefit analysis (CBA) and the multi-criteria analysis (MCA). Although they are two useful and theoretically well-founded methods, they are not able to cope with complicated material such as sustainability. ‘Cost-benefit analyses take money as the assessment factor, thus they can manage to express CO₂emissions, landscape pollution and even traffic victims in terms of money. This can become very confusing for people very fast. The multi-criteria analysis is a wonderful technique that wants to take account of everything, but then you often end up with a long list of criteria, which becomes unwieldy and often counts things twice.’ According to Sijtsma, a good combination of the strong points of both systems will result in simplicity and clarity. ‘The strength of the cost-benefit analysis is its strict causality, a precise analysis of what is cause and what effect. The multi-criteria analysis looks further than just economic terms. A good evaluation has to confine itself to the most important criteria, which are set in advance and after thorough discussion. Data must be presented in natural ways (CO₂emissions in kilotons, travel time in hours) so that everyone knows what you’re talking about.’

Openness

Analysts must stop always trying to say what the ‘best choice’ is, thinks Sijtsma. They must support their decision-making with clear data. A clear best choice is often, and certainly when talking about sustainability, just not possible. ‘If you go too far in your evaluation, you’ll overplay your hand. That was what happened in the reports about the Zuiderzeelijn, for example. The RUG thought that it was viable, the research bureau Ecorys did not. They disagreed on the extent and valuation of part of the effects. Why didn’t they just say, this is what we know, and the rest is just guesswork? Even customers often just want a clear result, even if that’s not possible, and are perhaps less interested in how it is achieved.’ Sijtsma is in favour of an evaluation where everyone can see and understand exactly what is being measured. It’s easy to make things complicated, in his opinion. ‘The trick is to keep the analysis simple. But perhaps not everyone wants that much openness, and it may be in the interests of some parties to have an unclear evaluation.'

Curriculum Vitae

Frans J. Sijtsma studied Economics in Groningen and has worked for the RUG since 1992. He has conducted lots of research in the wide field of spatial economics: agriculture, nature policy, countryside, goods transport, regional labour market, industrial estates, sustainability. He is the coordinator of the Economics and Management & Organization Science Shop and lecturer in Economic Geography. His defence of his thesis (19 October from 1.15 - 2.15 p.m. ) will be broadcast live on www.rug.nl/kennisdebat/  (in Dutch only).