Findings: Data quality

There are opportunities to innovate and improve data quality

Greater use of administrative data would improve poverty statistics

Many of the leading indicators on income-based poverty rely on social surveys such as DWP’s FRS and ONS’s Household Finance Survey (HFS). Whilst there are many advantages to the survey-based approach, users we spoke to identified that there is untapped potential within administrative data to further augment and improve existing income-based poverty statistics.

One area of potential that DWP and ONS are already working on is the use of linked administrative data to improve the timeliness of income-based poverty statistics. The household income surveys underpinning these statistics are produced annually and can be lagged by up to 15 months from the reference period. The COVID-19 pandemic has increased users appetite for timely information and, in regards to poverty, it will be over a year before the effects of the pandemic on poverty will be seen in the data. We are pleased to hear DWP and ONS are considering the timeliness of the income-based poverty statistics as a priority, to enhance the public value of these statistics. ONS has already developed a set of admin-based income statistics.

The SMC’s recommendations for measuring poverty noted the absence of liquid assets, such as savings, in the existing official statistics on poverty and explored how this affects the data. While wealth and assets are not components of income, the absence of them in income-based poverty measures can lead to households wrongly appearing to be poor, if they report low incomes but have high levels of wealth in the form of assets. We recommend DWP and ONS, building on existing work to explore the feasibility and potential of social survey and administrative data integration, explore whether integration can help improve the timeliness and robustness of income-based poverty statistics.

As with most surveys of their kind, the household income surveys produced by ONS and DWP have historically had problems with undercounting of benefit receipts in the data they collect. In a submission to our review, the Resolution Foundation raised the issue of benefit under-reporting in social surveys. It found, for example, that around £40 billion a year in benefits was missing from the FRS in 2016/17.This is due to a number of issues, such as people forgetting or underestimating certain sources of their income when they respond to surveys, or respondents not being willing to disclose that they are on benefits. DWP and ONS do acknowledge the existence of this issue in their bulletins, and are considering steps to address benefit misreporting in the FRS.

Another known limitation of sample-based surveys of household income is that they consistently under-report income at both the top and bottom of the income distribution. In its effects of taxes and benefits publication, ONS recently introduced an adjustment to address survey under-coverage of the richest households, using administrative tax data. DWP have also been using admin data to adjust for under-coverage of high-income households for a number of years. We are pleased to see the approach that both producers have adopted here. We encourage DWP and ONS to prioritise work to address under-reporting at the bottom end of the income distribution. They should consider a multifaceted approach to solving this problem, such as data linkage and making greater use of administrative data.

We found that the opportunities for administrative data need to be balanced with accessibility concerns. When speaking to producers in the devolved nations, we heard concerns that the devolution of Scottish benefits may cause short-term disruption to their access to benefits data, which would impact the timescales for achieving the ambition of introducing administrative data into the HBAI statistics.

We also heard from researchers who had concerns about access to the FRS microdata being increasingly restricted if the dataset becomes larger and more sensitive with the addition of administrative data. We recommend that DWP and ONS understand and address concerns about access when introducing administrative data into the production of income-based poverty statistics. We are pleased to hear from DWP that they are alert to these issues and are working with the Scottish Government to develop a solution.

There is still a role for social surveys but their limitations should be made clearer

There remains a substantial role for social surveys in this space to ask the questions that administrative data cannot capture – including around family structure, housing costs, certain sources of income and lived experience. Surveys play a vital role in uncovering answers to key questions around poverty and will continue to do so going forward.

For the first time, ONS’s Household Income Inequality statistics for 2019/20 were produced using a new combined data source called the Household Finance Survey (HFS). This combines data from the Living Costs and Food Survey (LCF) and the Survey on Living Conditions (SLC). The combined data source provides a sample survey of around 17,000 private households in the UK. This is just under the sample size for DWP’s Family Resources Survey (FRS) in 2019/20 of 19,000 households. Users told us that it is confusing to have two sets of statistics on household income where it is not clear what the relative strengths of each series are. They expressed a desire to have a single and more complete source of data on household incomes. ONS and DWP told us that the new combined strategic vision and GSS Income and Earnings Coherence Work Plan reflects all aspects of striving to improve coherence. We recommend that DWP and ONS should determine the user need for a single data source on household incomes by exploring the feasibility of consolidating the existing social surveys, as part of their existing plans in the new combined strategic vision and GSS Income and Earnings Coherence Work Plan. This could either be used to inform different publications, or to form the basis of a single set of statistics constructed from a consolidated data source, based on an understanding of user needs.

A number of users also cited concerns around the non-response bias of surveys such as the FRS, which can lead to certain demographic groups being overrepresented in the statistics at the expense of others. Producers should also ensure they are transparent with users about potential bias in survey response rates, and how this affects the reliability of results. We are pleased to hear DWP is taking forward a large scale boost of the FRS from April 2022, and of ONS’s research into targeted sample boosts in its household finance surveys, to better measure groups that are currently underrepresented. We recommend that DWP and ONS look to better understand the non-response bias of their surveys, and ensure they are transparent with users about any potential bias. We acknowledge that there are difficulties associated with this, given the fact that the FRS is an address-based survey. DWP is currently exploring the feasibility of different approaches to understanding non-response bias in the FRS, including linking sampled addresses to DWP held data, and we welcome any further progress on this work.

Existing methods of equivalisation should be reviewed

Equivalisation scales are used in reporting on income-based poverty statistics to adjust household income, taking into account household size and composition, to compare household units. The modified OECD scale is the model used by official statistics producers in the UK and, although it is widely used across European countries, a number of users expressed a view that the current methods of equivalisation used in the UK are outdated, unhelpful and arbitrary. ONS has already conducted some initial research into alternative methods of equivalisation.

Users told us that there could be value in developing equivalisation methods for income statistics which are tailored more specifically by age and other demographic characteristics. The SMC has conducted initial research into the appropriateness of the current approach and the possibility of developing a new, more detailed scale. The SMC’s recommendation for measuring poverty has highlighted that ‘inescapable costs’ faced by families are not captured in the existing equivalisation scales and therefore misrepresent the disposable income a family is left with. These costs include disability, childcare and commuting costs.

A current lack of reliable data available on these inescapable costs is a significant barrier to developing a robust equivalisation scale that is able to take these disparities into account. Such scales would also rely on a number of broad assumptions about the spending patterns and needs of the groups they affect.

We recognise that trying to develop an equivalisation scale which perfectly represents all households is an impossible task, and that attempting to account for too many different household structures could contribute to over-complicating a landscape of statistics and data that is already complex. Users told us the strengths of the current methods are that they allow for a consistent time series of poverty measures and that they enable comparisons with other countries. We consider, however, that improving equivalisation methods does not have to be done at the expense of a consistent time series. For example, DWP were able to move smoothly from the original McClements scale, when it first adopted the OECD scale in 2005.

Some of the producers we spoke to noted the wider value that could be gained from a cross-cutting review of equivalisation methods used across government, as there are currently discrepancies in the approach taken by different government departments and policies. This review could take the form of a literature review on income equivalisation used in poverty statistics in the short term, followed by a longer-term strategic initiative across government looking at equivalisation methods more broadly. DWP and ONS should consider leading a review of equivalisation methods, in collaboration with other producers, building on the initial work conducted by ONS.