OSR response to the Sturgis review

Introduction

In January 2023, OSR commissioned Professor Patrick Sturgis of the London School of Economics to review the way that we carried out a regulatory review of the Covid Infection Survey, run by the Office for National Statistics (ONS). We found Professor Sturgis’ review helpful and intend to use the model of external review more regularly to get an external perspective on our work.

We welcome Professor Sturgis’ evaluation that our review did a good job of identifying the strengths of the survey and of noting where improvements were needed, but that there were weaknesses in the ways that the findings and recommendations of the review were communicated to ONS and in how progress against objectives was monitored and enforced. We also welcome Professor Sturgis’ recommendations to improve both the way that we present our findings and how we subsequently follow up on the actions that we require or recommend be taken.

We accept all the recommendations from the review, noting that for our core assessment and systemic review projects we already largely work in a way that is consistent with Professor Sturgis’ suggestions. Our priority will therefore be to consider how to extend these important principles to our smaller and more ad-hoc reviews. We will do this in parallel with considering what further improvements we can make to our existing assessment and systemic review processes. We expect to invite a further review of our processes once we have made the suggested changes.

The remainder of this document includes the recommendations from Professor Sturgis’ review and our response to each.

Recommendations and responses

Recommendation 1: Ensure that the recommendations in its reviews and assessments provide more explicit guidance on the actions that need to be taken by the producer and with a clearly specified time frame for completion for each recommendation.

OSR Response: We agree and will implement this as stated.

We will continue to develop our recommendations to make them clearer. A key way that we aim to do this is to be explicit on the desired beneficial outcome from making the change, or the risk that the change is expected to manage. We will also aim wherever possible to provide guidance on the “how”, and in doing so we will recognise that as the producer body looks into taking action, or because circumstances change, there may be alternative ways of delivering the desired outcomes and impact. We will also ensure that we are clear whether our suggested actions are required, recommended or suggested.

Our assessment reports currently include a timescale for reporting back to OSR; we will ensure that we are clear on required timescales for action to be taken resulting from other projects, including being clear in those cases where there may be no specific timescale, or the timescale is flexible.

Recommendation 2: Require that producers publish a response to a review within six weeks of the review’s publication, setting out how it intends to act on all recommendations in that review.

OSR Response: We cannot strictly require producers to publish a response but agree that we should ask producers to publish a response and will work through the practicalities of extending this to our non-assessment work.

It is already widespread practice within our assessments for producer bodies to respond to the assessment describing their intended actions – either through a high-level letter, a formal action plan, or both. We also request or are offered such responses in a small proportion of our shorter reviews. We will consider how to apply this principle proportionately across our different regulatory projects, recognising that some of our reviews are short and the findings may not lead to the need for a formal action plan. We recognise that publishing a response within six weeks may be a suitable default and will seek to agree with producer teams the appropriate timescale for a response in each project. We will pay particular regard to resourcing pressures within producer teams especially during busy times of the statistical production process.

Recommendation 3: Monitor and report on progress against the recommendations in its reviews by regularly updating the review Annex, in which the findings and recommendations are enumerated against the pillars of the Code [of Practice for Statistics]. Review reports should be considered ‘live’ documents, with progress against recommendations updated when milestones and deadlines become due.

OSR Response: We agree in principle and will work through the practicalities of extending this to our non-assessment work.

For each of our assessments we publish details of actions taken by the producer body, and OSR’s response, alongside the confirmation or awarding of National Statistics designation^[1]. We will consider how to develop this practice in the following ways:

• Weighing the potential benefits of publishing progress updates to the table more frequently against the administrative burden of so doing.
• Applying proportionately to our other projects, taking specific care to avoid being burdensome for either OSR or producer bodies especially in the smallest reviews.

Recommendation 4: Implement a penalty for failure to comply with review recommendations within the specified timeline. This might be a requirement by the head of the unit being reviewed to write a letter to the National Statistician explaining why the recommendation has not been actioned.

OSR Response: We agree that we should consider how to respond to failure to comply with our recommendations and will work through the practicalities of extending this to our non-assessment work.

We consider that as a minimum it would be appropriate for producer teams to write to the Director General for Regulation (as head of OSR) rather than the National Statistician to describe why recommendations haven’t been actioned, although recognise that we have no formal powers to require this. Writing to the Director General for Regulation reflects our current practice for other correspondence about failure to comply with the Code of Practice for Statistics. We will consider what further interventions may be appropriate, recognising that we have few formal powers or sanctions that we can apply beyond the removal of National Statistics for statistics that are designated as such.

[1] For example the recently concluded assessment of Scottish prison population statistics.

Office for Statistics Regulation

18 May 2023

PDF Version: Professor Patrick Sturgis Review – OSR Response