Quality assurance of administrative data standard has three levels of assurance: basic, enhanced and comprehensive.
This case example illustrates enhanced quality assurance of administrative data used by DWP to produce Work Programme statistics, particularly using financial audit and inspection checks.
Content:
Background to the statistics
The data sources and suppliers
Approach to the quality assurance of the administrative data
Outline of DWP’s quality assurance approach
- Initial checks quality assurance conducted during data development
- Routine quality assurance
- Audit and inspection assurance
- Automated check
- Post-payment check on a random sample
- Corroboration against another data source
- Data processing – adjustment for error
- Feedback check with data suppliers
- External audit
Background to the statistics
DWP’s Work Programme is the government scheme in Great Britain to assist people who are long-term unemployed into sustained employment. The service is provided by employment support organisations through 40 contracts with 15 prime providers. These providers work with a larger number of sub-contractor organisations. Claimants are randomly assigned by the local Job Centre Plus to a prime provider in their area.
[From DWP’s statistical release: Quarterly Work Programme National Statistics to Sep 2015]
The data sources and suppliers
The providers are paid when the claimants complete defined periods in work (usually after 6 months, but 3 months for those that are hard to place e.g. ex-prisoners) receiving the job outcome payment. The providers then can receive further payments for each additional continuous 4 weeks in employment – i.e. sustainment payments.
DWP’s Work Programme statistical release provides an overview of the monthly intake of people referred to the Work Programme, tracked over time, published in quarterly reports.
[From DWP’s statistical release: Quarterly Work Programme National Statistics to Sep 2015]
Approach to the quality assurance of the administrative data
DWP describes its approach to quality assurance for the Work Programme statistics in its background information note. Section 3.8 describes the quality assurance arrangements, summarised below:
Initial checks quality assurance conducted during data development:
- reliability, completeness and level of disclosure of individual variables
- levels of duplicate, missing or contradictory information
- consistency across computer systems (Labour Market System (LMS) and Provider and Referral Payment system (PRaP)) and with management information, via cross checking
- monthly build up of figures in system data
- trends and variation in characteristic, time series and geographical breakdowns
- a methodology to allocate early clerical outcomes to months
- consistent results during data visualisation and tabulation tool development via dual methodology testing and manual checking
- trends and differences in post payment adjustment factors via secondary quality assurance
Routine quality assurance:
- duplicate, missing or contradictory information
- accordance across computer systems (LMS and PRaP) and with management information
- trends and variation in characteristic, time series and geographical breakdowns
- trends and differences in post payment adjustment factors
- dual methodology testing of data visualisation
- automated checks against tolerances for a sample of tabulation tool tables
Individual variables remain complete and consistent with existing sources, comparisons across systems remain predominantly within a 0.1% tolerance, including across key breakdowns and time series.
Quality assurance in operational system:
DWP conducts a series of checks during the processing of the Work Programme payment claims. These include checks by operational staff and financial auditors and corroboration against administrative data held by HM Revenue and Customs (HMRC).
Automated check
All Job Outcome payment claims are subject to an ‘off-benefit check’ before payment. This involves an automated check to match participant information on DWP’s Customer Information System, to ensure that participants for whom Job Outcome payments are claimed are not claiming benefit.
The automated off-benefit check has a window of 15 days in which the check is applied, to allow for minor discrepancies between the details of the provider’s claim and the details on Departmental systems. Job Outcome payment claims that fail this automated check are removed from the system (unless they can be validated) and not paid. Claims which pass the off-benefit check are released for payment, and are then subject to further post payment in-work checks.
Post-payment check on a random sample
Post-payment validation is performed every month to strengthen the controls against fraud and error in the Job Outcome payments reported to DWP by the Work Programme providers. This process involves selecting a sample of 17 claims per contract from the total population of Job Outcome payments that passed the automated off-benefit check and that were subsequently paid in that month. The results of 6 rounds of validation, one for each month, are brought together every six months (April-September, October-March) to provide validation rates.
Corroboration against another data source
The sample is matched against HMRC P45 data to validate employment. Those that fail the HMRC check are validated by confirming employment with either the employer or the individual. Job Outcome payments that are found to be invalid are used to calculate the error rate which is extrapolated from the total population. The results of 6 rounds of validation (one for each month) are brought together every six months to provide quarterly error rates used in the National statistics.
The primary purpose of the error rate is to extrapolate financial recoveries against all payments made to a contract in the extrapolation period based on the error rate, rather than for the sampled claims alone. Once the percentage of error has been calculated from the sample, the error rate is applied to the total paid to providers for the relevant six month period, and the provider is then required to pay this back to the Department.
Data processing – adjustment for error
The error rates for the 40 contracts are used to derive adjustment factors which are then used to rate the official statistics to reflect final Job Outcome payments made to providers. The adjustment factor is derived using the number of the Job Outcomes which fail the post-payment validation process divided by the total number of Job Outcomes sampled. This ratio is applied to Job Outcomes (less the sample and those already validated) to adjust the official statistics.
Feedback check with data suppliers
Once the validation process has been completed, Work Programme providers have the opportunity to challenge its results. Time is allowed for providers to challenge and for DWP to assess and arbitrate any challenge. This process can take up to approximately 3 months, so that the official statistics on Job Outcomes for some providers may be revised slightly in the following quarterly release. DWP says that the affect of these revisions have so far been minimal.
The end to end post-payment validation process takes approximately 8 and 1/2 months to complete. The routine sampling, checks and production of error rates take just over 1 month and these are performed on the previous six months Job Outcomes payments.
External audit
In addition to the quality assurance of the statistics, assurance of the underlying administrative data is also carried out. The National Audit Office published a report in August 2013 on their review of the data systems associated with the Work Programme Business Plan Transparency indicator.