Administrative Data Quality Assurance Toolkit

QA matrix

The QA Matrix helps assessors and producers to determine the types of assurance and documentation required to inform users about the quality assurance arrangements for administrative data. This guidance can support a judgment about the suitability of the data and to identify examples of practices that meet the different levels of assurance.

Producers are responsible for judging the appropriate level of assurance. Producers should be able to explain the basis of their judgments of the chosen levels of assurance.

‘No assurance’ (A0) is not compliant with the Code of Practice for Official Statistics.

The need for investigation and documentation increases at each level of assurance from ‘Basic’ (A1) to ‘Enhanced’ (A2) to ‘Comprehensive’ (A3). It may be appropriate for the levels of assurance to vary among the four practice areas; for example, given specific circumstances it may be appropriate for ‘Communication’ to be Basic (A1), while ‘Data Collection QA’ be Enhanced (A2) and for both ‘Operational Context’ and ‘Producer’s QA’ to be Comprehensive (A3).

Assessors will make an evaluation of what they regard as the appropriate level of assurance for the administrative data during an Assessment of official or National Statistics based on administrative data.

The Authority may decide that given the level of risk of quality issues and the public interest profile of the statistics that a higher level of assurance is appropriate than that judged by the statistics producer. In these cases, assurance levels A1 or A2 may be viewed as not compliant with the Code. The Assessment will identify the specific areas of practice that are required for compliance with the Code.

Four practice areas associated with data quality

Operational context and admin data collection

environment and processes for compiling the administrative data
factors which affect data quality and cause bias
safeguards which minimise the risks
role of performance measurements and targets; potential for distortive effects

Communication with data supply partners

collaborative relationships with data collectors, suppliers, IT specialists, policy and operational officials
formal agreements detailing arrangements
regular engagement with collectors, suppliers and users

QA principles, standards and checks by data suppliers

data assurance arrangements in data collection and supply
quality information about the data from suppliers
role of operational inspection and internal/external audit in data assurance process

Producers' QA investigations & documentation

QA checks carried out by statistics producer
quality indicators for input data and output statistics
strengths and limitations of the data in relation to use
explanation for users about the data quality and impact on the statistics

Levels of assurance for four areas of practice related to quality assurance of administrative data regularly provided for producing official statistics (see Annex A to see the QA Matrix on one page):

Practice area 1: Operational context & administrative data collection

Level of assurance	Operational context & administrative data collection
A0: No assurance	Operational context and administrative data collection by supplier not investigated, managed or documented
A1: Basic assurance Statistical producer has reviewed and published a summary of the administrative data QA arrangements	Consider the following types of activities: Producer has provided users with an outline of the administrative data collection process, Illustrated the administrative data collection process and main stages, Outlined the operational context, Identified actions taken to minimise risks to quality, Identified and summarised the implications for accuracy and quality of data, including the impact of any changes in the context or collection arrangements
A2: Enhanced assurance Statistical producer has evaluated the administrative data QA arrangements and published a fuller description of the assurance	Consider the following types of activities: Producer has provided users with a fuller description of the operational context and administrative data collection arrangements, eg: a process map detailing the data collection processes, explanations for classifications, Identified and summarised potential sources of bias and error in administrative system, Identified and described safeguards taken to minimise risks to data quality, Provided a detailed description of the implications for accuracy and quality of data, including the impact of any changes in the context or collection arrangements
A3: Comprehensive assurance Statistical producer has investigated the administrative data QA arrangements, identified the results of independent audit, and published detailed documentation about the assurance and audit	Consider the following types of activities: Producer has provided users with a detailed description of the administrative system and operational context: explained why the data are collected, who by and how, identified differences across areas in the collection and recording of the data, identified issues for individual data items, such as whether objective or based on subjective recording, missing and/or imputed, poorly recorded, Identified issues in design and definition of targets, Identified and described potential sources of bias and error in the administrative system, Identified and explained any safeguards used to minimise the risks to data quality, Provided a detailed and specific description of the implications for accuracy and quality of the data, including the impact of any changes in the context or collection arrangements

Practice area 2: Communication with data supply partners

Level of Assurance	Communication with data supply partners
A0: No assurance	No communication
A1: Basic assurance Statistical producer has reviewed and published a summary of the administrative data QA arrangements	Consider the following types of activities: Producer has outlined the data provision arrangements including: annual statement of needs, timing and format of data supply, coordination of data sign-off from data supplier, Fed back identified errors to data suppliers and recorded their response, Sought the views of statistics users about the data and resolved any quality issues reported
A2: Enhanced assurance Statistical producer has evaluated the administrative data QA arrangements and published a fuller description of the assurance	Consider the following types of activities: Producer has agreed and documented: data requirements for statistical purposes, legal basis for data supply, data transfer process, arrangements for data protection, sign-off arrangements by data suppliers, Established an effective mode of communication with contacts (eg with data collector and supplier bodies, IT systems, operational/policy officials) to discuss the ongoing statistical needs in the data collection system and quality of supplied data, Sought the views/experiences of statistics users and resolved any quality issues reported
A3: Comprehensive assurance Statistical producer has investigated the administrative data QA arrangements, identified the results of independent audit, and published detailed documentation about the assurance and audit	Consider the following types of activities: Producer has established/maintained collaborative relationships, Has a written agreement specifying: roles and responsibilities, legal basis for data supply, data supply and transfer process, security and confidentiality protection, schedule for data provision, content specification, Used a change management process, Regularly communicated with the data collector and supplier bodies, IT systems, operational/policy officials eg newsletters, conferences, attending data supplier/IT system group meetings, Regularly engaged statistics users, resolved any reported quality issues, and held user group conferences

Practice area 3: QA principles, standards and checks applied by data suppliers

Level of Assurance	QA principles, standards and checks applied by data suppliers
A0: No assurance	No description of suppliers' QA procedures and standards
A1: Basic assurance Statistical producer has reviewed and published a summary of the administrative data QA arrangements	Consider the following types of activities: Producer has knowledge of suppliers' QA checks and published a brief description, Identified whether audits are conducted on the admin data (such as internal or operational audits, external audit such as by regulator), Described the implications for the statistics
A2: Enhanced assurance Statistical producer has evaluated the administrative data QA arrangements and published a fuller description of the assurance	Consider the following types of activities: Producer has provided a fuller description of the main QA principles, quality indicators and checks used by the data suppliers, Described the role of relevant information management or governance groups in data quality management, Described the role of audit of the admin data within the collection and operational settings, Described the implications for the statistics for the quality issues identified by data supply bodies and regulators
A3: Comprehensive assurance Statistical producer has investigated the administrative data QA arrangements, identified the results of independent audit, and published detailed documentation about the assurance and audit	Consider the following types of activities: Producer has described the data suppliers' principles, standards (quality indicators) and quality checks, Reviewed quality reports for the received data (such as input quality indicators for data accuracy, coverage and completeness), Identified and documented the findings of investigations and audits conducted on the admin data and associated targets (such as internal and operational audits, and external audits by regulators and professional bodies), Described the implications for the statistics and determined whether the data continue to be satisfactory for official statistics purposes

Practice area 4: Producer’s QA investigations & documentation

Level of Assurance	Producer's QA investigations & documentation
A0: No assurance	No description of own QA checks
A1: Basic assurance Statistical producer has reviewed and published a summary of the administrative data QA arrangements	Consider the following types of activities: Producer has established regular QA checks on the received admin data, Published a description of its own QA checks on the admin data, Outlined the general approach and overall findings, Identified the strengths and limitations of the admin data, Explained the likely degree of risk to the quality of the admin data
A2: Enhanced assurance Statistical producer has evaluated the administrative data QA arrangements and published a fuller description of the assurance	Consider the following types of activities: Producer has provided a fuller description of its own QA checks on the admin data, Detailed the general approach and findings for specific quality indicators, Identified the strengths and limitations of the admin data, Explained the likely degree of risk to the quality of the admin data
A3: Comprehensive assurance Statistical producer has investigated the administrative data QA arrangements, identified the results of independent audit, and published detailed documentation about the assurance and audit	Consider the following types of activities: Producer has provided a detailed description of its own QA checks on the admin data (including validation, sense and consistency checks), Given quantitative (and where appropriate qualitative) metrics for specific quality indicators (such as input, process and output quality metrics), Undertaken comparisons with other relevant data sources (such as survey or other admin data), Identified possible distortive effects of performance measurements and targets, Identified the strengths and limitations of the admin data and any constraints on use for producing statistics, Explained the likely degree of risk to the quality of the admin data

« Previous

Download PDF version (243.41 KB)