Mark Pont to Tony O’Connor: Armed forces continuous attitude survey statistics

Dear Tony

ARMED FORCES CONTINUOUS ATTITUDE SURVEY STATISTICS

As you are aware, we recently completed our compliance check of the Ministry of Defence’s (MOD) Armed Forces Continuous Attitude Survey statistics against the Code of Practice for Statistics. Following a constructive conversation with yourself and the other MOD statisticians responsible for these statistics, I am pleased to confirm that these statistics should continue to be designated as National Statistics.

We found many positives in the way that MOD produces and presents these statistics, which enhance their value and quality, including:

  • the team has moved from publishing individual tables to using visualisations to illustrate the data. This helps make technical data more engaging for non-technical users
  • additionally, the team has begun to add context to the report; this adds a great deal of value and aids in interpreting the data and extracting the main messages. Your commitment to further expand on this and willingness to look at what is done in other surveys, with the families’ survey being discussed, is also very encouraging
  • the team, along with other analytical teams in MOD, has been working to ensure its statistical bulletins meet the new accessibility requirements for public sector organisations. This work will ensure a broad range of users are able to access the statistics easily
  • the background quality report published with the statistics provides a lot of helpful information about data sources and issues. In particular, the report includes useful and clear information on the context in which the statistics are produced and where they are used within the MOD. There is also further information about methods and the section on statistical weighting and its links to response rates is very clear and informative
  • the data used to produce the visualisations in the report are available to download from the associated website in a format that encourages re-use. Currently the data collected are in the form of a five-point Likert scale that are recategorised to three points and the team has agreed to look into releasing the untransformed Likert scale numbers to allow more-nuanced re-use

Our review also identified several ways in which we consider that you could further enhance the trustworthiness and value of these statistics:

  • the value of government statistics is determined by what they contribute to the public good and these statistics are among the most widely accessed statistical publications in the MOD. While there is good engagement with the main users within MOD there is little engagement with other users. In the 2014 formal review (which covered coherence with other MOD statistics and wider user engagement) many of the user requests from outside MOD resulted in changes to the survey or work on putting the data in context with wider MOD data. Although a similar review scheduled for 2019 was cancelled, it would be good to find a way to ensure that as wide a range of users’ views as possible can feed into the statistics
  • that the planned 2019 formal review could not go ahead due to resource constraints is understandable. However, we consider that trustworthiness could be improved if this decision were to be mentioned, for example in the background quality report, rather than merely removing all mention of the review

The importance of this survey cannot be understated. Its contribution to decisions around pay and benefits are obvious but equally clear is that the high level of online access shows that there is a large level of wider interest around it. The value of the survey results comes in large part because of the good response rate. There is a risk that lack of coordination of surveys across MOD could cause respondent fatigue and damage the response rate for this survey. We consider that stronger central survey governance would be helpful in providing appropriate reassurances about the future value of these statistics, and the quality of other surveys within the MOD, and we encourage you to consider how such arrangements could be resourced and put in place.

Thank you to you and your teams for your positive engagement during this review. Our Crime and Security team will continue to engage with you on progress in the coming months. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further.

I am copying this letter to Paula Fredersdorff, Deputy Head of Profession for Statistics and Kerry Flecknoe, Armed Forces Continuous Attitude Survey lead.

Yours sincerely

 

Mark Pont

Assessment Programme Lead

Mark Pont to Tony O’Connor: Finance and Economic series: Trade, Industry and Contracts Statistics

Dear Tony

FINANCE AND ECONOMICS SERIES: TRADE, INDUSTRY AND CONTRACTS STATISTICS

As you are aware, we recently completed our short review of compliance with the Code of Practice for Statistics of the Ministry of Defence’s (MOD) Trade, industry and contracts statistics. I am pleased to confirm that these statistics should continue to be designated as National Statistics.

We chose to review these statistics as they make an important contribution to the public accountability of MOD. We reviewed the statistics against the three pillars of the Code – trustworthiness, quality and value – and, following a constructive conversation with the MOD statisticians who produce the statistics, this letter outlines the key findings of our review.

We found many positive aspects in the way that MOD produces and presents these statistics, which enhance their value and quality, including:

  • The team uses helpful visualisations to illustrate the data. In particular, the use of heat maps to demonstrate expenditure break downs helps make technical data more engaging for non-technical users.
  • To ensure the statistics can be easily understood by non-technical users, the statistics team has agreed to provide definitions of technical and departmental terms at their first use in the publication. This will enhance the clarity of the statistics.
  • The team, along with other analytical teams in MOD, has been working effectively to ensure its statistical bulletins meet the new accessibility requirements for public sector organisations, which came into effect in September 2019. This work will ensure a broad range of users are able to access the statistics easily.
  • The quality report accompanying the statistics provides a lot of helpful information about data sources and about data issues. The Background and Context section – which provides information on data tables that used to be included in the bulletin, but no longer are – is clear and valuable to users, as it explains why these data are no longer provided and informs users that the team is working to reinstate them. In addition, the Relevance section helps users to understand how these statistics contribute to public debate.
  • The team is collaborating well with commercial and finance colleagues within MOD, to be able to improve the quality of data on competitive and non-competitive contracts, which could not be included in the 2018/19 statistics. This has included discussions to ensure operational databases correctly process the data fields needed by the team for its analysis and providing training for commercial officers to make sure they understand the importance of the all the data (including the non-mandatory competitive markers) and how to complete the data forms. We are pleased to hear that the statistics team can now retrospectively include data for 2018/19 in the 2019/20 publication and is taking steps to try to ensure that data for 2019/20 can be included as well.

The engagement between analysts and colleagues in other parts of MOD is really positive and will help colleagues who play an indirect part in the statistical production process to understand how the data they collect are used and the impact their work can have on the final statistics. High-quality analysis and statistics, which in turn will inform better organisational delivery, will always depend on high-quality input data, which are correct at the point of entry. It can also help ensure analytical requirements are considered when changes are made to operational and IT systems.

Continued collaboration with others across MOD will be important to continue to improve the quality of these data and statistics, and to reinstate other tables that have not been included in the bulletin in recent years, due to underlying data quality issues: tables showing the number of contracts and expenditure with small and medium-sized enterprises, for example. This will boost the value of the statistics for policy and decision makers.

During our discussion, you also told us that several MOD statistics are now produced using Reproducible Analytical Pipelines (RAP) and that work is ongoing to identify further statistics that would most benefit from the adopting this approach. It is exciting to hear about these developments, which can help with quality management, presentation and offer efficiency savings: we look forward to seeing how this work develops.

Thank you to you and your teams for your positive engagement during this review. Our Crime and Security team will continue to engage with you on progress in the coming months. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further.

I am copying this letter to Paula Fredersdorff, Deputy Head of Profession for Statistics; Chris Harvey; and Julia Edwards.

Yours sincerely

Mark Pont

Assessment Programme Lead