Dear Tony

MOD statistics on regional expenditure with UK industry and commerce and supported employment

As you are aware, we recently completed our compliance check of the Ministry of Defence’s (MOD) Regional expenditure with UK industry and commerce and supported employment statistics against the Code of Practice for Statistics. The recent spending review has intensified public interest in defence spending, and these statistics support understanding and debate about job creation and prosperity across the UK.

We found many positives in our review, particularly around the clarity and insight of the bulletin and transparency of methods. We welcome the recent development of estimates of the number of indirect job estimates supported through MOD expenditure, as this meets a user need for this information. Key strengths include:

• The team communicates the statistics widely within MOD and engaged with internal stakeholders as it developed the statistics. The statistics have had impact – for example, they were featured in the MOD Integrated Review Command Paper published last year. The team also has a constructive relationship with ADS Group, a trade body representing the aerospace, defence, security and space sector, which publishes similar expenditure and jobs estimates for the whole of the UK defence sector. The team has discussed methods and assumptions with ADS Group.
• The bulletin is engaging and insightful. It provides commentary on changes in expenditure and jobs over time and differences between regions and, where possible, links these to specific MOD projects or contracts. The statistics are presented in a variety of ways, enhancing their accessibility for a wide audience. For example, expenditure is presented per capita and direct jobs per 100,000 Full-Time Equivalent, making them easily comparable across regions for all users. The data tables and visualisations, in particular the regional maps and charts, are informative and aid understanding of the statistics.
• The background quality report contains detailed information about the methods used, including assumptions made and limitations. The description of the indirect jobs methodology is especially thorough and demonstrates good practice in guiding users through a complex process. The flowchart is clear and accessible and the full worked example is helpful.
• The team is automating production of the statistics through implementing a Reproducible Analytical Pipeline (RAP). It has automated large parts of data processing for expenditure data and is using an open-source programming language. We commend the team for adopting RAP principles as this has enabled the team to bring forward the release of the statistics by two months and strengthen quality assurance processes.
• The bulletin is transparent about the provisional nature of the latest jobs estimates; due the coronavirus pandemic, employment and turnover data from the Office for National Statistics (ONS) have been delayed until summer 2021. It is good that the team carried out a review of historic data using MOD expenditure with a one-year lag on ONS employment and turnover data to understand the likely impact of the mismatch. The likely impact, and possible level of change that might occur when the figures are updated, are clearly communicated to users.

Our review also identified several ways in which we consider you could enhance the trustworthiness, quality and value of these statistics as part of your ongoing development:

• Despite the extensive use of the statistics by policy teams, there are alternative publications that cover similar areas so the team should encourage and support senior colleagues to use statistics that are robust and most appropriate for the situation. For example, there is a related set of statistics published by the ADS Group, which were developed after MOD discontinued the original statistics in 2009 and therefore filled a gap. We encourage the team to build confidence in its own statistics by making a stronger case for the value they add. In addition, to help users understand the coherence and comparability of the MOD and ADS Group statistics, the bulletin and quality report should be clearer about what the MOD statistics do and when they should be used alongside, or instead of, the ADS Group statistics.
• The team and others in MOD have told us that there is a lot of interest in these statistics outside MOD, and the bulletin highlights the range of potential users. So far, user engagement has been limited to users within MOD and the ADS Group, but the team could gain useful feedback to help it further improve the statistics by widening its engagement. It should attempt to understand the extent to which the statistics are used by those outside MOD and be proactive in engaging with these users. The team might find our regulatory guidance on user engagement and the Government Statistical Service’s user engagement top tips helpful when planning user engagement activities.
• The statistics draw on a wide range of data sources, including MOD financial data, Supply-Use Tables and Input-Output Analytical Tables, population estimates, and labour market and business surveys. The quality report contains a brief description of each source but lacks information about the quality, strengths and limitations of some sources. For example, the jobs figures are estimates derived from labour market and business surveys, but neither the bulletin nor the quality report mentions uncertainty around the estimates. The extent and nature of any uncertainty should be clearly explained. This is particularly important for granular breakdowns, which are more uncertain. To aid user understanding of the quality of the data, the team should publish more-detailed information about the quality of all data sources.
• The quality report would also benefit from additional information about certain aspects of the methods used, including the potential impact of assumptions. The team told us it has done some work on the impact of the alignment between the MOD Standard Industrial Classifications (SIC) and those used by ONS, and it might be helpful to publish this sensitivity analysis for expert users. There is also room for improving the accessibility of methods information for non-expert users. For instance, a worked example for direct jobs, modelled on the clear indirect jobs example, would help users understand how these figures are derived.

Thank you to your team for their positive engagement during this review: we look forward to continuing to engage with you and the team and we hope our findings inform the development of these statistics. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further or if we can offer further assistance as these statistics continue to develop.

Yours sincerely

 

Mark Pont

Assessment Programme Lead