The critical judgment about the suitability of the administrative data for use in producing official statistics should be pragmatic and proportionate, in which some statistics will need greater levels of assurance than others. The levels can be determined by an evaluation of:
- the likelihood of quality concerns arising in the administrative data that may affect the quality of the statistics
- the nature of the public interest served by the statistics
The Risk/Profile Matrix
Level of risk of data quality concerns
Use the following descriptions to help determine whether the administrative data should be regarded as low, medium or high risk of having quality concerns:
Low risk – the data may have a low risk of data quality concerns in situations in which there is a clear agreement about what data will be provided, when, how, and by whom; when there is a good appreciation of the context in which the data are collected, and the producer accepts that the quality standards being applied meet the statistical needs.
Medium risk – the data may be regarded as having a medium risk of data quality concerns when high risk factors have been moderated through the use of safeguards, for example, integrated financial audit and operational checks, and effective communication arrangements. It is also appropriate to consider the extent of the contribution of the administrative data to the official statistics, for example, in cases where the statistics are produced in combination with other data types, such as survey or census data.
High risk – the data may have a high risk of data quality issues when there are many different data collection bodies, intermediary data supplier bodies, and complex data collection processes with limited independent verification or oversight.
For example:
- The risk is lower when a supplier team is located within the same organisation as the statistics producer but increased when there are multiple data collection agencies and/or intermediary data suppliers.
- The risk is lower when operational processes include teams whose function it is to quality assure and verify the recorded data and where internal audit is an established mechanism with the body. The risk is higher when no internal audit occurs and where limited external and independent challenge occurs to the data, such as, provided by regulator bodies.
- The risk is lower when clear agreements are in place and communication is regular and found to be effective by all partners in the data supply process. The risk of data quality issues is increased when roles and responsibilities are not clear and no formal agreement is in place for the supply of data.
Official Statistics examples of different risk ratings of data quality concerns
Low risk:
Birth registrations – legally mandated information provision; direct information provided by key source (parents/guardians); face to face data entry (by registrar); centralised collection with common IT system for provision (by local registry offices to national statistical institute, ONS); built-in validation system checks.
Medium risk:
DWP’s Work Programme – multiple providers, payment by results offset by integrated financial audit and operational checks
High risk:
ONS’s police recorded crime – multiple data suppliers with variable recording practices, subjective interpretation of information, use of local targets to drive performance, lack of external scrutiny, indirect relationship between statistical producer and data suppliers with intermediary statistical producer receiving and processing administrative data
Public interest profile of the statistics
Statistical producers and assessors should use the following descriptions to help determine whether the administrative data should be regarded as having low, medium or high levels of public interest (representing the ‘public good’):
Low profile – politically neutral subject; interest limited to niche user base, and limited media interest
Medium profile – wider user and media interest; moderate economic and/or political sensitivity
High profile – economically important, reflected in market sensitivity; high political sensitivity, reflected by Select Committee hearings; substantial media coverage of policies and statistics; important public health issues; collection required by legislation.
Official Statistics examples of different public interest profile ratings
Lower:
Ofqual’s vocational qualifications – not contentious, small user base, specific interest
Medium:
HSCIC’s adult social service activity statistics – public interest in adequacies of locally provided social care services
Higher:
ONS’s police recorded crime – strong public interest in crime, fear of crime, effectiveness of police forces, public accountability of government and police, strong public concern in mis-recording of crime recording by police forces, investigation by Public Administration Select Committee