Assessment of compliance with the Code of Practice for Statistics – 2021 Census in Northern Ireland

Published:
17 November 2021
Last updated:
16 November 2021

Executive summary

Why this assessment is needed

The Census is one of the most important sources of data and statistics, informing decisions about almost every aspect of life within the UK. It allows users – including government, local authorities, academics, and commercial businesses – access to important information on the people and households of the UK and helps people get a better understanding of the places in which they live and work.

The Office for Statistics Regulation (OSR) is carrying out assessments of the UK Censuses produced by the Office for National Statistics (ONS), the National Records of Scotland (NRS) and the Northern Ireland Statistics and Research Agency (NISRA). The assessments will allow OSR to recommend whether the Census outputs should be designated as National Statistics, in accordance with the requirements of the Statistics and Registration Service Act 2007, when they are first released.

It is essential that the data and statistics from the Censuses are reliable and provide valuable insights, meeting the rigorous standards of trustworthiness, quality and value outlined in the Code of Practice for Statistics. This assessment report focuses on the 2021 Census in Northern Ireland, produced by NISRA, and aims to identify areas of strength, good practice and innovation in NISRA’s Census planning and development as well as identifying some areas where improvements need to be made.

Back to top

What we found

NISRA has successfully delivered Census operations for Northern Ireland during the COVID-19 pandemic, overcoming many challenges to its systems and service delivery. Census return rates in Northern Ireland reflected a high level of participation, with 97 percent of occupied households returning Census questionnaires. The Census team in NISRA has worked flexibly to effectively reprioritise resources and modify its plans for the Census collection period, ensuring live operations were undertaken safely for both the public and NISRA staff. This has been a significant achievement for the core Census team and all involved in operational delivery.

The data from Census 2021 will provide a snapshot of life in Northern Ireland at this unprecedented time. Data collected during this time may well be unusual or changed from what might have been expected, particularly on topics such as employment and economic activity. NISRA now has an important role in understanding and explaining the impacts on the data to users, supporting the appropriate use of Census data, and seeking to address any unmet user needs as a result.

NISRA launched its outputs consultation in October 2021, to gather views from data users and stakeholders on its plans for Census outputs. Consultation events have been arranged to promote and explain its current plans, with both general interest and targeted audience groups. This will be an extremely important exercise to understand the needs of a wide and varied group of users, as NISRA finalises its plans to deliver high quality and valuable Census data and statistics. In its work to improve how Census data will be disseminated to users, NISRA has worked collaboratively with ONS and the Central Statistics Office (CSO) in the Republic of Ireland to utilise new tools which will increase the accessibility and flexibility of Census data. The real value of the 2021 Census in Northern Ireland will be realised when Census data and statistics are released. NISRA is committed to successfully achieving that aim.

As NISRA works to process and produce Census data and statistics for Northern Ireland, it must also continue to prepare its supporting information on quality, data sources, and methods. During the earlier stages in Census planning, limited detailed information was published on NISRA’s research, developments or judgements on quality and methods while these were still being finalised. NISRA should continue its work to ensure that Census outputs are accompanied by finalised information on quality to support users and assure them on the data sources and methods used in their preparation.

Back to top

Requirements

We have identified several ways NISRA needs to implement improvements to support appropriate use of the data and meet users’ needs to comply with the highest standards of the Code.

NISRA has made little information widely available on the steps it is taking to investigate, or otherwise meet, user needs for data affected by COVID-19. This is a significant gap in the assurances offered by NISRA on its plans to address impacts on data.

Requirement 1

In order to support society’s need for information, NISRA needs to clearly communicate how 2021 Census data may be impacted by COVID-19 and how it plans to address any unmet user needs. NISRA should ensure this information is communicated in an accessible and timely way, being open on plans, developments and progress even where definitive answers or solutions are still being sought.

Further steps need to be taken by NISRA to communicate plans and provide more-detailed information, when available, to users of UK population estimates, UK Census data and Ireland-Northern Ireland outputs. As the provision of UK data and statistics is the responsibility of ONS, NISRA should signpost users to related ONS materials on the UK population estimates and UK Census data as this becomes available.

Requirement 2

To assure Northern Ireland users of how their data needs will be addressed, NISRA needs to provide users with transparent, accessible and timely information on how UK population estimates for 2021, UK Census data and Ireland-Northern Ireland data will be provided. NISRA should continue to work with, and align communications with, ONS, NRS and CSO to explain any impacts on data quality and describe where user needs may or may not be met as a result.

To assure users of the value and quality of Census data, NISRA should ensure its plans to provide information on quality – including information on data collection and processing, quality assurance activities and quality measures, methods and use of administrative data, and NISRA’s judgement on appropriate use of Census data – are delivered as planned.

Requirement 3

NISRA should ensure finalised documentation on quality, information and judgements on suitable data sources, and methods and their application are complete. All supporting information should be sufficiently open and easily available to Census data users alongside its range of Census outputs.

With such a wide and varied set of users of Census data, NISRA needs to engage with user groups with different requirements and interests. This includes special interest groups or those from a topic-focussed perspective or when, for example, considering the needs of users with different levels of expertise or accessibility requirements.

Requirement 4

In order to ensure the relevance of data and statistics to users, NISRA needs to continue to develop and enhance its user engagement activities, connecting with a broad range of users and embracing challenge. NISRA should continually review and seek to implement improvements in its engagement strategies and should ensure its decision making is open and transparent, being clear where users’ needs can or cannot be met.

NISRA aims for a first release of Census Population and Household estimates by summer 2022, with all other planned Census releases being published by summer 2023. NISRA is also committed to producing Census outputs that meet users’ needs and that are timely, accessible and flexible. Producing timely and accurate data from the Census is vital to ensuring high public value.

Requirement 5

NISRA needs to deliver its aims in relation to timely, accessible and flexible Census outputs – while ensuring sufficient data quality and supporting appropriate use of the data. It should clearly communicate its plans and timelines for outputs at the earliest opportunity, updating and revising these as soon as more detail is available or to reflect any changes to its plans.

Having accessible and easily findable information supports the appropriate use of data and statistics.

Requirement 6

To best support Census data users, NISRA needs to continue to improve its webpage navigation for current materials. NISRA’s plans for a separate website or webpages for Census outputs themselves will require sufficient consideration of its navigation and accessibility. NISRA should keep webpages and content refreshed and current.

Back to top

Judgement on National Statistics Status

We have identified six requirements for NISRA to address in order to ensure the high standards of public value, quality and trustworthiness associated with National Statistics designation are met.

Once NISRA has demonstrated that the improvements covered by these requirements have been made or provided sufficient assurance that our expectation for the data and statistics will be met, OSR will recommend to the UK Statistics Authority that National Statistics status for these statistics be confirmed. NISRA is aiming to meet the requirements of this report by spring 2022 so a designation decision can be made ahead of first Census outputs in summer 2022.

Back to top
Download PDF version (194.75 KB)

Assessment of compliance with the Code of Practice for Statistics – 2021 Census in England and Wales

Published:
17 November 2021
Last updated:
2 December 2021

Executive Summary

Why this assessment is needed

The Census is one of the most important sources of data and statistics, informing decisions about almost every aspect of life within the UK. It allows users – including government, local authorities, academics, and commercial businesses – access to important information on the people and households of the UK and helps people get a better understanding of the places in which they live and work.

The Office for Statistics Regulation (OSR) is carrying out assessments of the UK Censuses produced by the Office for National Statistics (ONS), the National Records of Scotland (NRS) and the Northern Ireland Statistics and Research Agency (NISRA). The assessments will allow OSR to recommend whether the Census outputs should be designated as National Statistics, in accordance with the requirements of the Statistics and Registration Service Act 2007, when they are first released.

It is essential that the data and statistics from the Censuses are reliable and provide valuable insights, meeting the standards of trustworthiness, quality and value outlined in the Code of Practice for Statistics. This assessment report focuses on the 2021 Census in England and Wales, produced by ONS, and aims to identify areas of strength, good practice and innovation in ONS’s Census planning and development as well as identifying some areas where improvements need to be made.

This webpage was updated on 2 December 2021 to add a PDF downloadable version of the report.

Back to top

What we found

ONS has successfully delivered Census operations for England and Wales at a time of significant change and in unusual circumstances. The response to Census saw ONS exceed the return rate targets that it set for itself, both overall and for local authority areas; 97 percent of occupied households completed Census returns and all local authority areas saw over 90 percent occupied household return rates. It has worked hard to ensure the safety and wellbeing of its staff and the general public in doing so and these efforts should not be understated. The assessment team recognises the hard work and commitment of ONS to achieve this – from senior leaders, across the numerous specialist teams, and of course including those working in the field.

The data from Census 2021 will provide a unique snapshot of life in England and Wales, taken during the COVID-19 pandemic. Data collected during this time, on topics such as employment, travel, and household status, may well be unusual or changed from what might have been expected. ONS now has an important role in understanding the impacts on the data and seeking to address any unmet user needs as a result.

ONS should continue its work to ensure that Census outputs are accompanied by finalised information on quality to support users and assure them on the data sources and methods used in their preparation. This will be particularly important for 2021 Census data when it comes to areas of change – for example, on data particularly affected by COVID-19, for new Census questions, or where concerns from users have been raised such as on data collected on sex as, in response to a High Court order, ONS made a change to its online guidance during live operations. ONS has a responsibility to support and assure users of the quality of Census data and its plans to deliver this alongside Census outputs.

To enable enhanced quality assurance arrangements in Census data processing, ONS now aims to release its first Census outputs slightly later than originally planned, in late spring 2022, and all other estimates within 24 months of Census. ONS is developing new tools and reporting formats with the aim of providing users with more-flexible, timely access to data and delivering greater clarity and insight to support the use of Census data and statistics. Delivering high quality Census outputs at the earliest opportunity is vital to delivering public value from the Census.

ONS has recently completed a large-scale consultation exercise with data users and stakeholders on its plans for Census outputs. This is one of a range of different user engagement exercises which has been undertaken throughout ONS’s work on Census. ONS utilises various tools and approaches as part of its engagement to gather user views, get expert advice, update stakeholders of its plans, and explain its decision making – this includes formal consultations, user advisory groups, webinars and public events.

Having open and effective dialogue with users and stakeholders is vital in demonstrating that ONS is a trustworthy organisation that actively listens and responds to users’ views. We consider that, for some users, elements of ONS communications and engagement could have been handled better and confidence in ONS’s trustworthiness may have reduced; ONS’s communications linked to the Census sex question was an example of this. ONS needs to learn lessons from its experiences – both good and bad – for future Census user engagement and more broadly across the organisation.

Back to top

Requirements

We have identified several ways ONS needs to implement improvements to support appropriate use of the data and meet users’ needs to comply with the highest standards of the Code.

ONS has made little information widely available on the steps it is taking to investigate, or otherwise meet, users’ needs for data affected by COVID-19. This is a significant gap in the assurances offered by ONS on its plans to address impacts on data.

Requirement 1: In order to support society’s need for information, ONS needs to clearly communicate how 2021 Census data may be impacted by COVID-19 and how it plans to address any unmet user needs. ONS should ensure this information is communicated in an accessible and timely way, being open on plans, developments and progress even where definitive answers or solutions are still being sought.

Further steps need to be taken by ONS to communicate its plans and provide more-detailed information, when available, to users of UK population estimates and UK Census data in particular.

Requirement 2: To assure users of how their data needs will be addressed, ONS needs to provide users with transparent, accessible and timely information on how it will provide UK population estimates for 2021 and UK Census data. ONS should continue to work with, and align communications with, NRS and NISRA to explain any impacts on data quality and describe where user needs may or may not be met as a result.

ONS made a change to its guidance for the Census sex question during live Census collection operations and while it told us that it expects this to have minimal impact on the Census data, ONS needs to publish suitable evidence to support this.

Requirement 3: ONS should be open and transparent in publishing its plans to evaluate and mitigate any risk to data given the change in the sex question guidance during live Census collection operations. ONS should provide appropriate assurances to users of the quality of these data and any implications for use should be clearly explained, including at disaggregated levels, alongside Census outputs.

To assure users of the value and quality of Census data, ONS should ensure its plans to provide information on quality – including information on data collection and processing, quality assurance activities and quality measures, methods and use of administrative data, and ONS’s judgement on appropriate use of Census data – are delivered as planned.

Requirement 4 :ONS should ensure finalised documentation on quality, information and judgements on suitable data sources, and methods and their application are complete. All supporting information should be sufficiently open and easily available to Census data users alongside its range of Census outputs.

With such a wide and varied set of users of Census data, ONS needs to engage with user groups with different requirements and interests. This includes from special interest groups or topic-focussed perspective or when, for example, considering the needs of users with different levels of expertise or accessibility requirements.

Requirement 5: In order to ensure the relevance of data and statistics to users, ONS needs to continue to develop and enhance its user engagement activities, connecting with a broad range of users and embracing challenge. ONS should continually review and seek to implement improvements in its engagement strategies and should ensure its decision making is open and transparent, explaining where users’ needs can or cannot be met.

Producing timely and accurate data from the Census is vital to ensuring high public value. We recognise the improvements to ONS’s quality assurance processes – and how this reflects ONS’s commitment to quality – although this has affected the release schedule for Census outputs.

Requirement 6: ONS needs to continue its efforts to deliver timely, accessible and flexible Census outputs – while ensuring sufficient data quality and supporting appropriate use of the data – mitigating any risks to further delay to the release of Census data and statistics. It should clearly communicate its plans and timelines for outputs at the earliest opportunity, updating and revising these as soon as more detail is available or to reflect any changes to its plans.

The current ONS Census webpages contain a wide range of materials, research, plans, and reports, focussed on providing information to aid with transparency throughout its planning and development stages although it is still difficult to find materials and navigate through the ONS Census webpages.

Requirement 7:

To best support Census data users, ONS needs to continue to improve its webpage navigation for current materials. ONS’s plans for a separate website or webpages for Census outputs themselves will require sufficient consideration of its navigation and accessibility. ONS should keep webpages and content refreshed and current.

 

Back to top

Judgement on National Statistics Status

We have identified seven requirements for ONS to address in order to ensure the high standards of public value, quality and trustworthiness associated with National Statistics designation are met.

Once ONS has demonstrated that the improvements covered by these requirements have been made or provided sufficient assurance that our expectation for the data and statistics will be met, OSR will recommend to the UK Statistics Authority that National Statistics status for these statistics be confirmed. ONS is aiming to meet the requirements of this report in the coming months so a designation decision can be made ahead of first Census outputs in late spring 2022.

 

Back to top
Download PDF version (376.99 KB)

The Code of Practice: Guidance for Models

 This guidance is in alpha phase, meaning that we welcome any comments and feedback from everyone. Please get in touch with regulation@statistics.gov.uk. We aim to release an updated version of this guidance in early 2022 based on a review of this work.

This document provides guidance on how the principles in the Code of Practice for Statistics can support good practice in designing, developing and using models. The document has been created to cover both traditional statistical techniques, such as linear regressions, and newer techniques including machine learning, when used to create statistics or generate data used to inform decision making and public policy. In the guidance a number of tick box statements have been included so that you can apply the principles in your work.

Part I explores the planning and designing of a model. It provides steps to ensure your planned work meets the expectations of the Code of Practice before you begin to develop the model. The two main factors which should guide your decision to create or use a model even before referring to specific elements of the Code of Practice are the context and purpose of the work. This is so you can demonstrate the appropriateness of your chosen technique. You should consider why you want to continue to use of change your current approach; what alternatives that also need to be considered; whether you can meet the quality of existing statistics with the approach; and whether learning and capability is sufficiently prioritised to achieve your aims.

You must also understand the ethics when considering the use of your model (§ Ethical considerations). You must ensure that the use of the chosen technique is ethically appropriate. This includes knowing the provenance and biases of the data, as well as knowing legal requirements such as Data Protection Legislation, the Human Rights Act 1998, the Statistics Registration and Service Act 2007 and the common law duty of confidence. The UK Statistics Authority’s (UKSA) Centre for Applied Data Ethics’ ethics self-assessment form can be used to assist this process.

Key in making sure your model succeeds is whether the responsible team is sufficiently skilled to undertake this work (Professional capability). Techniques such as machine learning may require different skillsets compared to traditional statistical techniques. However, it is important that the team also knows how to apply these traditional statistical techniques to avoid use of complex techniques if not necessary. If the team does not have the sufficient skills, you must decide whether to upskill them or bring in specialist resource. It is important to note, however, that the team must have the appropriate knowledge and skills to manage both the implementation and maintenance of the model.

Building capability and staying up to date with the latest techniques achieves development goals (Innovation and Improvement). You should understand where these skills sit within your team and organisations overall development plan. You should make sure learning is sufficient prioritised in your area, and whether the chosen techniques are the best use of available resource.

Like in the Code of Practice, the Chief Statistician or Head of Profession, or those with equivalent responsibility, should have sole authority for deciding on methods used for published statistics in their organisation (Transparent processes and management). This is also true for models that are used in decision making. The responsible individual needs to be aware of the methodological choices that have been taken in designing the model. Creating a chain of model accountability allows anyone involved in the project to know who to go to if something goes wrong or an error is identified.

Part II focuses on the steps you should take to best develop and use your new model to serve the public good. Part of this is ensuring that users of the model, or data and statistics generated by the model, are at the heart of any decisions to change the way these statistics or decisions are made (Relevance to users). This engagement should continue once the model is introduced to ensure user need is factored into the continuous development and monitoring of the model.

The model documentation and data should also be accessible to all (Accessibility). It is good practice to make data used by and generated from models open-access where applicable and appropriate. In some cases, this may not be possible. Model code should be findable, accessible, inter-operable, and re-usable.

You should collaborate with experts in both the type of model being used or developed and the subject matter which the data concern (Clarity and insight). This is to ensure any new insights drawn from the model are aligned with the experts’ understanding. It shall also help you identify potential errors or bias in your model. You should be transparent about the involvement of expert groups involved in the model creation. You should produce documentation that can describe the model and work to a range of users.

Crucially, you must know how your model works and know who you will need to communicate your model to (Explanation and Interpretaion). There is a risk that poor communication leads to misuse or misinterpretation of the model. This in turn could lead to over or under reliance on its outputs. Public acceptability is related to how well you can describe your model and its outputs. There should also be stringent quality assurance processes that can satisfy developers and those who are accountable for the model. Quality assurances show that the model is fit for purpose and generating outputs that can be trusted.

 

Back to top
Download PDF version (163.24 KB)

Defining the Public Good in Applications to Access Public Data

Published:
28 May 2021
Last updated:
7 June 2021

Summary

This report is about how researchers see their research as serving the public good, or providing public benefits. Researchers can apply to gain access to data held by government departments so that they can analyse it themselves to answer questions they have. In order to gain access, researchers must apply to the National Statistician’s Data Ethics Advisory Committee (NSDEC) or the Research Accreditation Panel (RAP).

When researchers do this, they must show that their work will serve the public good in some way. There are different ways of defining what the public good is, or how public benefits can be delivered by research, but little is known about what researchers think the public good is.

We wanted to carry out an analysis to understand more about this. We analysed a selection of applications to the NSDEC and the RAP to see what researchers said when they were asked what public benefits their work would provide.

Our findings (through a quantitative analysis) showed that the researchers often said their work would provide evidence which would help public policy decision-making or help make decisions which may benefit the economy, society or quality of life (in the UK). This might be because researchers want to show the real-world impact their research could have. Researchers might also think that this improves the chance of their application being successful so they can access the data they want.

Another frequently mentioned public benefit in applications to the NSDEC was to improve the quality, coverage or presentation of existing statistics but this was one of the least frequently mentioned public benefits in applications to the RAP. This shows that researchers applying to the different processes may want to achieve different aims in their work.

The least mentioned public benefit was to replicate, validate or challenge Official Statistics. This may suggest that researchers want to avoid saying that they will directly challenge data.

Looking more closely at the text in the applications (qualitative analysis) showed that some researchers say their work will help to improve decisions to spend public money. Other researchers say that their work will help to provide more detailed information about different regions in the UK. Lastly, several researchers said their research would enable more research collaborations to take place, and could help to improve the linking of datasets. This means that the public good provided by data can be extended even further.

There are limits on how much we can read into this work because we analysed applications and did not directly ask researchers what they think about the public good. However, the study does tell us what researchers focused on in their applications and what public good they would like their work to serve. This is important information for us in OSR as it helps us to develop our understanding about what the public good means to different people.

Back to top
Download PDF version (315.74 KB)

Executive summary

Why we did this review

Poverty is an important social and economic issue in the UK. Yet there is currently no universally accepted definition of poverty. The evidence base for poverty in the UK has largely been driven by international best practice and successive government strategies aimed at eradicating poverty.

The concept of poverty means different things to different people. This makes it difficult to define and measure. Despite this challenge, it is important for central and local governments to understand and address the nature of poverty in the areas they serve.

The Department for Work and Pensions (DWP) and the Office for National Statistics (ONS) are the primary producers of official statistics on income-based poverty. However, there are a number of other official statistics producers working in this space, including the Northern Ireland, Scottish and Welsh Governments. For the purposes of this report, where we refer to ‘statistics producers’, this includes all of the producers who contribute to the income-based poverty statistics landscape. Where recommendations apply to specific producers, we will refer to them by name.

A Government Statistical Service (GSS) Income and Earnings Coherence Steering Group was established in 2020, aimed at addressing the coherence and accessibility of income and earnings statistics. The group is made up of statistical leaders across DWP, HMRC and ONS, as well as representatives from the devolved administrations and academia, who are striving to improve the evidence base on income-based poverty, as well as income more broadly.

There are also several prominent organisations outside of government that contribute to the wider evidence base on poverty. These include the Social Metrics Commission (SMC), which was formed in early 2016 with the goal of creating new poverty measures for the UK, as well as think tanks such as the Institute for Fiscal Studies, the Resolution Foundation and the Joseph Rowntree Foundation.

When poverty is discussed in the public domain, it is often painted as a single statistic or trend which can mask the complexity of the underlying issue. The fact that there are multiple approaches to measuring poverty also means that measures can be used selectively, to suit a particular argument or point of view.

We want to ensure that statistics on poverty provide a robust evidence base for national and local policy development and decision making. We champion the need for statistics to support a wide range of uses, including by charities, researchers and individuals.

Back to top

What we found

Three strategic improvements are needed to support and deliver statistics that fully meet users’ needs. These would represent a step-change in the way the statistics on income-based poverty are produced and will require continued joined up, collaborative working, to be achieved.

  • The GSS Income and Earnings Coherence Steering Group needs to continue to collaborate and demonstrate leadership of the income-based poverty statistics landscape, to move away from producing a series of individual outputs to a more coherent and comprehensive evidence base.
  • Statistics producers need to better understand how the income-based poverty statistics are being used across policy and service provision and how the evidence base can be improved.
  • Innovation is needed for the statistics to deliver their full potential and serve the public good. Opportunities for data linkage should be maximised and data gaps should be addressed, building on work already underway in the GSS to explore the use of administrative data and its integration with social surveys.

Information needs in the poverty space are multi-faceted and encompass a range of specialist interests and priorities. To meet these broad needs, poverty is most helpfully viewed as a basket of main measures. As such, one measure could not adequately meet all the differing needs that users have for poverty statistics.

The current landscape of income-based poverty statistics is difficult for many to navigate and there is scope for signposting between the different statistics to be improved. The accessibility of language used in statistical bulletins and guidance accompanying the statistics could also be enhanced to support users’ understanding.

The number of people falling under the headline poverty line, drawn at 60% of median income, has remained stable over the past few years at around 14 million individuals. Focusing on this headline measure of poverty can mask important insights into the different levels of poverty experienced by different groups.

Whilst this review is focused on income-based poverty, poverty is closely linked to many other aspects of people’s lives, from employment prospects to health outcomes. Users we spoke to felt that the best mechanism for understanding people’s ‘lived experience’ of poverty is through qualitative research. Such research is currently carried out by a number of organisations outside of government, including the Joseph Rowntree Foundation.

Material deprivation is often used as a proxy for understanding the lived experience of poverty. The existing material deprivation statistics could be enhanced to ensure the questions are reflective of essential items and services in society today, and that they are an appropriate discriminator of who is ‘deprived’ and who is not.

Household surveys, which underpin most of the income-based poverty statistics, contain a number of data gaps. Users expressed concerns about the exclusion of the homeless and under-coverage of individuals with no recourse to public funds. There is also a lack of robust, granular data on ethnicity or sub-regional breakdowns in the data.

There is untapped potential within administrative data to augment and improve existing income-based poverty statistics. Administrative data could be used to address historical issues with sample-based surveys such as timeliness and benefit-underreporting. The opportunities for greater use of administrative data are already being explored by DWP and ONS.

There remains a substantial role for sample-based surveys in this space to ask the questions that administrative data cannot capture. These include questions on family structure, housing costs, certain sources of income and lived experience. However, there are limitations to these surveys which should be made more visible for less-expert users.

Equivalisation scales are used in reporting on income-based poverty statistics to adjust household income, taking into account household size and composition. Many users told us that the current modified OECD scale used by official statistics producers in the UK is outdated and arbitrary. It also fails to account for a number of ’inescapable’ costs such as disability, childcare and commuting costs. Users told us that there could be value in developing equivalisation methods for income statistics which are tailored more specifically by age and other demographic characteristics. ONS have already conducted some initial research into alternative methods of equivalisation.

Statistics producers we spoke to as part of this review are engaged with the subject of poverty and understand how they contribute to the evidence base. Importantly, whilst they endeavour to provide clear briefing on complex data, the information is still sometimes misunderstood and misused by politicians.

There is a lack of transparent communication of DWP’s development priorities and plans for income-based poverty statistics. Decisions around development of the statistics need to be communicated openly to enhance confidence in the data. Government departments need to take a wider view of user needs and look beyond immediate policy needs.

The GSS Income and Earnings Coherence Steering Group provides a cross-GSS vehicle to help producers address the recommendations set out in this report. The group has developed a vision statement and coherence and accessibility improvement workplan for Income and Earnings statistics in general, which will highlight the steps taken by the producers already with regards to the user need we have highlighted in this report. The group had plans to publish these outputs around the end of May 2021 at the time of our review.

 

Back to top

Our recommendations

The existing statistics on income-based poverty provide a good foundation for decision making but there are opportunities to improve the evidence base provided by official statistics. We have identified the following detailed recommendations for producers of poverty statistics:

Improve the accessibility of language and guidance

  • Producers should look to provide clearer and more detailed signposting to other income-based poverty statistics in their bulletins.
  • Producers should ensure supporting guidance is accessible to lay users and clear on the appropriate uses and quality of the statistics.
  • Producers should consider the helpfulness of the language used in the poverty bulletins and accompanying guidance, to ensure that it does not risk confusing or misleading less-experienced users.
  • DWP and ONS should ensure they are clear about the strengths and limitations of household surveys, particularly with regards to missing groups, and clearly set out the implicit and explicit assumptions that underline them.

Address data gaps to enhance insight

  • Producers should do more to draw out the necessary insights to allow users to understand the nature of poverty and how this varies between groups at differing levels of poverty, as identified above.
  • DWP and ONS need to understand why experts are funding their own data collections and analysis and consider whether this reflects weaknesses in the existing official statistics.
  • To increase the public value of the existing statistics, DWP should:
    • review the current set of questions which underpin material deprivation and determine a way to compare material deprivation across groups, in collaboration with other producers across the GSS who use these questions.
    • increase the consistency in the way it reports material deprivation, as it currently reports material deprivation of children in households with less than 50% and 70% of median income but not at 60%.
  • DWP and ONS should address the ethnicity data gap, as part of the wider GSS response to the findings of the Commission on Race and Ethnic Disparities’ report.
  • DWP should consider the potential to extend the low-income families at a local area level analyses to working-age adults without children and pensioners.

Review existing methods and maximise use of administrative data

  • DWP and ONS, building on existing work to explore the feasibility and potential of social survey and administrative data integration, should explore whether integration can help improve the timeliness and robustness of income-based poverty statistics.
  • DWP and ONS should prioritise work to address under-reporting at the bottom end of the income distribution. They should consider a multifaceted approach to solving this problem, such as data linkage and making greater use of administrative data.
  • DWP and ONS should look to understand and address concerns about access when introducing administrative data into the production of income-based poverty statistics.
  • DWP and ONS should determine the user need for a single data source on household incomes by exploring the feasibility of consolidating the existing social surveys, as part of their existing plans in the new combined GSS Income and Earnings Coherence Work Plan. This could either be used to inform different publications, or to form the basis of a single set of statistics constructed from a consolidated data source, based on an understanding of user needs.
  • DWP and ONS should look to better understand the non-response bias of their surveys, and ensure they are transparent with users about any potential bias.
  • DWP and ONS should consider leading a review of equivalisation methods, in collaboration with other producers, building on the initial work conducted by ONS.

Command confidence in the statistics through trustworthy production

  • DWP and ONS should assess how the SMC recommendations can be implemented in their own work to enhance the public value of their statistics. Any planned developments to the statistics should also be communicated in an open and transparent way.

 

Back to top
Download PDF version (363.67 KB)

Misleadingness: A follow-up thinkpiece

Published:
17 May 2021
Last updated:
20 May 2021

Misleadingness thinkpiece

Why we did it

At the Office for Statistics Regulation, we are often asked if we consider a particular use of statistics to be misleading. These questions can come from members of the public, politicians and organisations and we welcome them, because the interest in whether uses of statistics are misleading or not shows that people care about the appropriate use of statistics.

We always look carefully at these cases and seek to reach a judgement, but in investigating them, we find it is often not clear what is meant by something being “misleading”. The word is used to cover a wide range of situations and sometimes it seems as though the judgement we are being asked to make revolves around the merits of the argument that the user is making, rather than the use of statistics in itself.

So over the last year we have been thinking about the idea of misleadingness – what it is, and how we should approach it in the context of our work. We wanted to go beyond merely technical criteria and think about the impact of uses of statistics on audiences. Our first step was to publish a think-piece in May 2020, which we developed with input from Jenny Saul, a philosopher who has written and thought extensively about misleadingness.

Our think-piece explored three approaches to judging misleadingness:

1: Materiality and intention – an approach which focuses on the significance of the statement being made. What were the intentions of the speaker?

2: Audience – an approach which focuses on audience understanding. Were the audience misled about what the statistics were telling them?

3: Case-based – an approach which focuses on particular features of the presentation of statistics. Is the style of presentation unclear and likely to mislead?

We concluded that the most appropriate definition of misleadingness in the context of our work as statistics regulator was:

“We are concerned when, on a question of significant public interest, the way statistics are used is likely to leave a reasonable person believing something which the full statistical evidence would not support.”

We also determined that none of the three approaches was likely to be effective on its own. Instead, the think-piece tentatively concluded that a blended approach was likely to work best.

The paper that follows provides an update on our thinking based on conversations we’ve had and feedback we’ve received since we published the think-piece.

Back to top

Who we spoke to

After the initial publication of the misleadingness piece, we received feedback from a number of sources, including:

  • Further input from Jenny Saul, the philosopher who worked on the first think-piece
  • Outcomes from a seminar held with Jenny Saul, other philosophers she suggested[1], Ofcom and the Advertising Standards Authority
  • A meeting with the Royal Statistical Society (RSS) Data Ethics and Governance Section Committee
  • Individual feedback from the chairs of the RSS Data Ethics Committee
  • Feedback from a small number of other individuals

[1] Eliot Michaelson, Kings College London; Andreas Stokke, Uppsala University; Neri Marsilli, University of Barcelona; Alexandra Freeman, University of Cambridge; Jonathan Webber, University of Cardiff

Back to top

What we found

Overall support for the approach in the think-piece

• Overall, people welcomed the think-piece; it was valued as much as a trigger for discussion as for its content.

• One clear outcome was a recognition of the benefits of bringing together statistical, philosophical and regulatory approaches. Several people who provided feedback commented positively on this way of working.

• Having a clear statement of principles is helpful but we also need to recognise an irreducible complexity. Professor Kevin McConway of the Open University pointed out to us that it will always be difficult to produce a definitive document that describes every possible situation of misleadingness.

Back to top

Distinguishing production and use

• A strong sentiment from the feedback was the need to distinguish production and use. The production of statistics by Government departments and ONS requires rigorous collection and presentation of data, in line with the Code of Practice for Statistics. Once statistics have been published (ie produced), they are available for use, including by politicians. ‘Production’ can be thought of as an upstream activity, and ‘use’ as downstream. In the think-piece, we are focussing on the downstream element of ‘use’.

• Although the paper focuses on downstream ‘use’, we should recognise that the way statistics are produced can raise risks of misinterpretation and hence be used in a misleading way. OSR frequently addresses issues with production, such as poor presentation or incomplete commentary, in our regular reviews of statistics. This work lies outside the scope of this paper.

• In thinking about ‘use’, we recognise that there is often a range of actors involved in presenting a claim about statistics: the Government body that produces and publishes the statistics; the communications team that presents information drawing on the statistics; media interpretation and summary of what is said; social media reuse of short segments of what is said; and many more actors. It is not OSR’s role to intervene at all points in this chain. Our role is to focus on how prominent politicians take the statistics and use them in their own communications – for example, speeches, press releases, social media statements. There are other organisations, including Ofcom and press regulators, who consider the work of various media actors.

• In terms of ‘use’, there will always be a risk that too much weight is put on a particular set of statistics. As Paul Allin, chair of the Statistics User Forum, told us: “Statistics rely on precise definitions of things being measured, but result almost invariably in some imprecision on the measurement. Statistics are not strict accounts and may have confidence or error limits.”

Back to top

Intention not a helpful concept to guide our judgement

• Intention is not a helpful basis for guiding or supporting the OSR’s judgements about misleadingness. Both regulators and philosophers agreed that deciding someone has intended to mislead is difficult, subjective, and likely to lead to unnecessary controversy.

• It is far better to consider likely impact on audience, rather than intentions of the speaker. This approach is consistent with that taken when judging misleadingness in other contexts, for example by the Advertising Standards Authority and Ofcom.

• Judging intention may be important to some people – for example, journalists wishing to understand and explain the factors behind particular decisions or arguments. But OSR is forming a view on the appropriate use and interpretation of the statistics, not judging the motivations and drivers of the person using the statistics.

• Although judging the intention of the speaker may not be the right approach for OSR, when considering materiality we will look at whether the use of statistics is significant – and one element in this consideration can be whether a particular use is repeated over time or part of a prepared communication (speech, political ad, etc). These factors will inform how we take forward a case – for example, how strongly we express any concerns we may have.

Back to top

Aspects not covered by the initial think-piece

• Some specific issues arose that need further consideration:

  1. There were some risk factors that the original think-piece did not consider, for example the use of incomplete statistical evidence (eg placing too much weight on early results of a new policy) or recency (eg placing too much weight on the latest data, even if changes the new data appear to show are not meaningfully different from past data).
  2. Many of the cases that OSR will deal with are relatively simple – for example, false statements that should be corrected, or use of unpublished data. In these cases, misleadingness would not be considered. In more complex or ambiguous cases where it is harder to reach a judgement, OSR would consider whether a statement has been misleading.

 

Back to top

Use of statistics in political communication

• The think-piece is relatively silent on the role of intermediaries. In one of our conversations we discussed a scenario in which a speech that is carefully constructed, well researched and uses statistics appropriately, is summarised in a single soundbite in media reporting. It was suggested that the speaker may actually intend this outcome, knowing that a careful speech will inevitably be packaged into a soundbite that could be misleading.

• As noted above, in the case of media intermediaries, the OSR approach would typically focus on the content of the original communication, not the media reporting of it. In the same way that we can’t assume the intentions of a speaker, it is similarly difficult to comment directly on the interpretation made by intermediaries. We can however give our view on the correct interpretation of the underlying statistics.

• One particular feature of political rhetoric, highlighted to us by Thomas King of the RSS Data Ethics section, is that different actors can draw widely different conclusions from the same underlying evidence. The point of democratic discussion is that different arguments are put forward; different narratives are presented; and different visions of good policy and the public interest are articulated. OSR’s role is not to judge these different perspectives, nor to limit the use of statistics to support them. Instead, our role is more humble: we simply try to ensure that the statistics are used in a way that does not give a misleading impression of the statistical picture.

• It is not OSR’s ambition to be an arbiter of political debate, nor would it be appropriate. Our role is to protect the role of statistics in public debate – that is, to ensure that their content and any caveats are respected in the way that they are used.

Back to top

Evolving the thinkpiece

Based on the findings from above, we have evolved how we consider these questions, by downplaying intention, recognising complexity, adding in further risk factors, and being clearer on the circumstances in which it is relevant to consider misleadingness.

For simple cases which are about false statements these considerations are not relevant. (An example is provided in the annex in which a clear misstatement about education funding was brought to our attention and was subsequently corrected). However, for complex cases, which are about the interpretation and weight put on statistics, these considerations are relevant. In all complex cases, we would use the core definition below to guide our judgment.

Old definition:

“We are concerned when, on a question of significant public interest, the way statistics are used is likely to leave a reasonable person believing something which the full statistical evidence would not support.”

New definition:

“We are concerned when, on a question of significant public interest, the way statistics are used is likely to leave audiences believing something which the relevant statistical evidence would not support.”

Back to top

Approach

Each piece of casework will be subjected to the same initial consideration, asking the following question:

Is this a question of ‘use’ of statistics, or of ‘production’?

If it is the latter, OSR will consider the issue in line with our interventions policy and the Code of Practice for Statistics, and look to address the question:

We are concerned when, on a question of significant public interest, the way statistics are used is likely to leave audiences believing something which the relevant statistical evidence would not support.

In addressing this question, there are three aspects to consider:

1: The nature of the issue

It is important to start with the issue and the context. This will enable consideration of whether relevant audiences are likely to be misled about a particular set of statistics, and whether there is any evidence that they have been misled. [This is based on approach 2 from the original think-piece]

There could be a range of audiences, of course, ranging from technically knowledgeable specialists to the general public, and OSR should consider which of these audiences is most relevant in considering the way the statistics have been used.

2: Risk factors

There are some recurring features of the way statistics are used that constitute risk factors – factors that can give audiences a different impression from that provided by the full, underlying evidence. [This is an extended version of approach 3 from the original think-piece]

 

Back to top

The risk factors are:

1. There is selectivity of data points to support a claim which other data points do not support. (for example, from a time series)

2. There is selectivity of a metric to support a claim which other related metrics do not support (for example absolute figures rather than percentage or cash terms rather than real terms)

3. The language used does not fully represent the available statistics (for example implying the statistics represent a much broader or narrower definition than appropriate.)

4. There are methodological choices which lead to potential bias in the presented figures

5. No source or methodology is given, making it likely that a hearer could draw inaccurate conclusions about what the available statistics represent

6. Poor quality data is used, making it likely that the hearer will believe something which is untrue

7. There is an inappropriate choice of graph axis or data

8. The causality of a statistic is overstated, making it likely that the hearer will believe there is stronger evidence to support a causal link than exists

9. There is an error in the statistic used – for example, the figures for the wrong year are used to describe a change over time

10. NEW There is undue weight put on recent or new data

11. NEW There is too much emphasis on data that are incomplete. (For example, early results from a trial)

Back to top

3: Materiality

Not all uses are as prominent as each other. It is important to consider the context of the use of statistics and ask the following questions:

• Is it a one-off or repeated use?
• Is it on a subject that the speaker has formal responsibility for?
• Is it part of a prepared speech or not?
• What is the public profile of the person using the statistics?

The answer to these questions will determine how significant the issue appears to be, with a one-off remark being less significant than a repeated use. [This is based on approach 1 from the original think-piece, but with no consideration of intention]

Back to top

Next steps

Although we already employ many of the concepts in this think-piece in our ongoing work, it is not yet finalised and we will continue to explore how it operates in practice. We would also welcome further comment to guide future updates and improvements.

If you’d like to get in touch with us about this document, please email us.

 

Back to top
Download PDF version (295.85 KB)