Dear Tony

ARMED FORCES CONTINUOUS ATTITUDE SURVEY STATISTICS

As you are aware, we recently completed our compliance check of the Ministry of Defence’s (MOD) Armed Forces Continuous Attitude Survey statistics against the Code of Practice for Statistics. Following a constructive conversation with yourself and the other MOD statisticians responsible for these statistics, I am pleased to confirm that these statistics should continue to be designated as National Statistics.

We found many positives in the way that MOD produces and presents these statistics, which enhance their value and quality, including:

• the team has moved from publishing individual tables to using visualisations to illustrate the data. This helps make technical data more engaging for non-technical users
• additionally, the team has begun to add context to the report; this adds a great deal of value and aids in interpreting the data and extracting the main messages. Your commitment to further expand on this and willingness to look at what is done in other surveys, with the families’ survey being discussed, is also very encouraging
• the team, along with other analytical teams in MOD, has been working to ensure its statistical bulletins meet the new accessibility requirements for public sector organisations. This work will ensure a broad range of users are able to access the statistics easily
• the background quality report published with the statistics provides a lot of helpful information about data sources and issues. In particular, the report includes useful and clear information on the context in which the statistics are produced and where they are used within the MOD. There is also further information about methods and the section on statistical weighting and its links to response rates is very clear and informative
• the data used to produce the visualisations in the report are available to download from the associated website in a format that encourages re-use. Currently the data collected are in the form of a five-point Likert scale that are recategorised to three points and the team has agreed to look into releasing the untransformed Likert scale numbers to allow more-nuanced re-use

Our review also identified several ways in which we consider that you could further enhance the trustworthiness and value of these statistics:

• the value of government statistics is determined by what they contribute to the public good and these statistics are among the most widely accessed statistical publications in the MOD. While there is good engagement with the main users within MOD there is little engagement with other users. In the 2014 formal review (which covered coherence with other MOD statistics and wider user engagement) many of the user requests from outside MOD resulted in changes to the survey or work on putting the data in context with wider MOD data. Although a similar review scheduled for 2019 was cancelled, it would be good to find a way to ensure that as wide a range of users’ views as possible can feed into the statistics
• that the planned 2019 formal review could not go ahead due to resource constraints is understandable. However, we consider that trustworthiness could be improved if this decision were to be mentioned, for example in the background quality report, rather than merely removing all mention of the review

The importance of this survey cannot be understated. Its contribution to decisions around pay and benefits are obvious but equally clear is that the high level of online access shows that there is a large level of wider interest around it. The value of the survey results comes in large part because of the good response rate. There is a risk that lack of coordination of surveys across MOD could cause respondent fatigue and damage the response rate for this survey. We consider that stronger central survey governance would be helpful in providing appropriate reassurances about the future value of these statistics, and the quality of other surveys within the MOD, and we encourage you to consider how such arrangements could be resourced and put in place.

Thank you to you and your teams for your positive engagement during this review. Our Crime and Security team will continue to engage with you on progress in the coming months. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further.

I am copying this letter to Paula Fredersdorff, Deputy Head of Profession for Statistics and Kerry Flecknoe, Armed Forces Continuous Attitude Survey lead.

Yours sincerely

 

Mark Pont

Assessment Programme Lead