Audit Committee report on quality of PBL work published

The audit committee, at the same time, was of the opinion that PBL should improve its process of quality assurance. In this context, the committee has recommended that a uniform external review procedure be established, and that important results and methodological innovations are communicated more often through peer-reviewed publications. In addition, the committee offered recommendations for further improvement of the quality and relevance of PBL work.

From August to December 2012, PBL was evaluated by an international audit committee of prominent scientists led by Lea Kauppi, Director General of the Finnish Environment Institute SYKE and Professor at the University of Helsinki. The committee met from 12 to 16 November of last year, in the Netherlands, for the purpose of this audit. The audit was conducted at the request of the external PBL advisory committee, chaired by Professor Wim van de Donk. The PBL advisory committee monitors the scientific quality and social relevance of PBL work. The audit committee completed its report this week and presented it to the PBL advisory committee.

One of the general conclusions of the audit committee was that PBL, in these times of government cut backs, has made some effective strategic choices based on a thorough analysis of PBL’s role as policy advisor and the changing context within which it operates.

For a number of themes, the international audit committee indicated a number of strong points as well as recommendations for improvement of the weaker points. A selection of these findings is presented below.

Science–policy–society interface: Interaction between PBL, science, policy and society

Strong points

• PBL fulfils a carefully considered combination of roles at the science–policy interface – the audit committee is of the opinion that PBL thus has defined an international benchmark;
• PBL is correctly utilising its independent position in placing relevant subjects on both policy and societal agendas;
• PBL analyses pay increasing attention to policy implementations and governance;
• PBL has developed state-of-the-art guidance documents for dealing with uncertainty and stakeholder participation.

Recommendations

• PBL should indicate more clearly how it intends to operate on various scales (local, regional, national, European and global);
• PBL should continue the current trend of increased stakeholder participation.

Scientific quality control

Strong points

• PBL has a clear understanding of the meaning of scientific quality in the context of providing independent policy advice;
• PBL uses an elaborate review procedure for its main products;
• PBL has properly functioning guidelines and procedures for verifying external data;
• The coordinating and advisory role of the chief scientist.

Recommendations

• PBL should draft a uniform procedure for external review and document review results within projects;
• PBL more often should communicate important results and methodological innovations through peer-reviewed publications;
• PBL should not cut back its investment in strategic research.

Organisation and human resources

Strong points

• PBL has motivated and competent employees;
• PBL has a positive organisational culture;
• PBL is a learning organisation, as is also clear from its critical self-evaluation;
• PBL has in-house staff trainings through the PBL Academy.

Recommendations

• PBL should develop a human resource strategy to achieve the required, necessary staff reductions while also making room for new expertise;
• In relation to the recommended increase in external financing, the government should allow PBL to carry over external funds from one budget year to the next;
• PBL should strive for more competences in the areas of interdisciplinary collaboration, stakeholder participation and in raising external financing.

Research projects closely scrutinised

The audit committee also reviewed eight PBL projects that were completed in recent years. This led to a number of conclusions and recommendations related to each project. These results also served as input for the overall conclusion about the quality of the PBL work. In addition, this also led to the following two general recommendations:

• PBL should improve the awareness and implementation of its state-of-the-art Guidance for Uncertainty Assessment and Communication.
• PBL should apply its expertise in the field of governance not only to certain PBL areas but in all PBL work.

PBL response

PBL plans to publish a response to the audit committee report, in consultation with the PBL advisory committee. This response will include measures and strategies related to all recommendations. The PBL advisory committee, which had requested the audit and has already received an initial verbal report by the audit committee, supports the recommendations and has requested PBL to address them in their response. The PBL response in the area of scientific quality assurance will include: improvement in the application of methodological guidance documents, making external review procedures more uniform and encourage peer-reviewed publications.

Audit committee

In preparation of the audit, PBL has conducted a self-evaluation: the PBL Self-evaluation Report, May 2008 – May 2012. This report has served as input for the audit committee. The committee visited PBL from 12 to 16 November of last year, and during this time met with the PBL advisory committee, PBL management and staff, researchers of research institutes, universities, CPB Netherlands Bureau for Economic Policy Analysis and Statistics Netherlands (CBS), and with Dutch policymakers.

Members of the audit committee:

1. Lea Kauppi (chair), Director General, Finnish Environment Institute & Associate Professor of Limnology and Land Use Impacts, University of Helsinki, Helsinki, Finland
2. Leen Hordijk, Professor Emeritus of Environmental Systems Analysis, Wageningen University, Wageningen, the Netherlands, & Principal Advisor and Head of the Modelling Taskforce of the Joint Research Centre, European Commission, Brussels, Belgium
3. Judith Innes, Professor Emerita of City and Regional Planning, UC Berkeley, Berkeley, CA, United States
4. Sheila Jasanoff, Pforzheimer Professor of Science and Technology Studies, Harvard Kennedy School, Harvard University, Cambridge, MA, United States
5. Pushpam Kumar, Chief, Ecosystem Services Economics Unit, United Nations Environment Programme, Nairobi, Kenya & Reader in Environmental Economics, University of Liverpool, Liverpool, United Kingdom
6. Wolfgang Lutz, Professor of Applied Statistics, WU - Vienna University of Economics and Business, Vienna & Leader of the World Population Program of the International Institute for Applied Systems Analysis, Laxenburg, Austria
7. Richard Moss, Senior Staff Scientist, PNNL Joint Global Change Research Institute at the University of Maryland & Visiting Senior Research Scientist, Maryland’s Earth Systems Science Interdisciplinary Center, College Park, MD, United States
8. Jeroen van den Bergh, Professor at the University of Barcelona (Universitat Autònoma de Barcelona), Spain & Professor of Environmental and Resource Economics at the VU University Amsterdam, the Netherlands

The audit committee was supported by an external secretariat in the hands of Dr Femke Merkx. The audit committee’s findings and recommendations as well as their full results have been published in the Audit Report.