M&A Note 1/2009

This paper is prepared by the Monitoring and Assessment Team to inform the Board’s consideration of the case, following the letter by the Chair of the Authority to the Permanent Secretary at 10, Downing Street in December 2008.(1) The paper relates to a Press Notice(2) and accompanying Fact Sheet(3) issued on 11 December by the Home Office. It considers aspects of the documents against the standards set out in the Statistics Authority’s Code of Practice for Official Statistics, to be published on 6 January 2009.(4)

Context – National Statistics and official statistics
Compliance with the Code is a statutory requirement in relation to National Statistics; it is good practice in relation to all official statistics.

Section 8 of the Statistics and Registration Service Act 2007 requires the Board to “monitor the production and publication of official statistics”; to “report any concerns it has about the quality of any official statistics, good practice in relation to any official statistics, or the comprehensiveness of any official statistics, to the person responsible for those statistics”; and allows the Board to “publish its findings or any report under this section”. Section 7(4)(a) of the Act specifically includes “impartiality” as a dimension of “quality”.

The statistics included in the 11 December documents were not released in the form of a statistical report, or otherwise issued by statistics professionals. The hospital admissions statistics were released against the advice of the relevant senior statisticians in the NHS Information Centre.

The Code is explicit that it applies to “all public bodies that produce official statistics”, not just to statisticians (Preamble, (ii)).

The preamble also says (xiii) that “it is implicit in the Code that there will be sufficient managerial separation between staff responsible for official statistics and other staff of the organisation to ensure clear lines of accountability for observance of the Code”.

Relevant statements of practice in the Code
Annex A lists the relevant provisions (practices) of the Code with which the documents seem to be inconsistent. A number of these practices relate to Principle 3, on Integrity. Others relate to Protocol 2, on Release Practices. We have concerns about the presentation of statistics in the Home Office Fact Sheet – specifically, about:

• Unclear description • Selective or otherwise inappropriate comparisons
• Lack of contextual information • Inappropriate conclusions being drawn
• Unsubstantiated claims.

These concerns are outlined in Annex B.

Monitoring & Assessment Team, UK Statistics Authority

---

1. https://uksa.statisticsauthority.gov.uk/reports—correspondence/correspondence/index.html
2. http://press.homeoffice.gov.uk/press-releases/Tackling-knives-six-months-on
3. http://press.homeoffice.gov.uk/documents/Tackling_Knives_Action_Prog1.pdf
4. https://uksa.statisticsauthority.gov.uk/assessment/code-of-practice/index.html

Annex A

List of practices from the Code with which the 11 December release of knife crime statistics appears to be inconsistent

• Publish statistical reports according to a published timetable that takes account of user needs. (Principle 1, practice 4)
• Issue statistical reports separately from any other statement or comment about the figures and ensure that no statement or comment – based on prior knowledge – is issued to the press or published ahead of the publication of the statistics. (3, 1)
• Ensure that those producing statistical reports are protected from any political pressures that might influence the production or presentation of the statistics. (3, 2)
• Ensure that the relevant statistical Head of Profession has the sole responsibility for deciding on statistical methods, standards and procedures, and on the content and timing of statistical releases. (3, 3)
• Ensure that official statistics are produced according to scientific principles. Publish details of the methods adopted, including explanations of why particular choices were made. (4, 1)
• Ensure that official statistics are produced to a level of quality that meets users’ needs, and that users are informed about the quality of statistical outputs, including estimates of the main sources of bias and other errors, and other aspects of the European Statistical System definition of quality (4, 2)
• Provide information on the quality and reliability of statistics in relation to the range of potential uses, and on methods, procedures, and classifications. (8, 1)
• Prepare and disseminate commentary and analysis that aid interpretation, and provide factual information about the policy or operational context of official statistics. Adopt formats for the presentation of statistics in graphs, tables and maps that enhance clarity, interpretability and consistency. (8, 2)
• Publish statistical reports in an orderly manner, in accordance with Protocol 2 (2, 1)Protocol 2: Release Practices
  Statistical reports should be released into the public domain in an orderly manner that promotes public confidence and gives equal access to all, subject to relevant legislation.Practices
  1. Release statistical reports as soon as they are judged ready so that there is no opportunity, or perception of opportunity, for the release to be withheld or delayed.
  2. Publish a timetable of statistical releases for twelve months ahead.
  4. Issue statistical releases at the standard time of 9.30am on a weekday to maintain consistency and to permit time for users to understand and respond to the information during normal working hours.
  6. Include the name and contact details of the responsible statistician in statistical reports.
  9. Ensure that government statements issued alongside official statistics, and referring to or based upon them:
  a. contain a prominent link to the statistical release and clearly refer to the source of the statistics;
  b. are labelled clearly as policy statements (or ministerial statements) and are readily distinguished from a statistical release and
  c. meet basic professional standards (for example, statistics should be cited accurately, and charts should be drawn in an accurate and impartial way).
• Ensure that no action is taken within the producer body, or public statement made, that might undermine confidence in the independence of the statistics when released (Protocol 3, 4)

Annex B

Monitoring and Assessment Team’s concerns about the presentation of statistics in the Home Office Fact Sheet

Unclear description
No information is given about the source or quality of the statistics, nor the use to which the data may legitimately be put. No information is given about whether data quality is likely to be the same in the ‘Tackling Knives Action Programme’ (TKAP) areas and other areas, and for the periods under comparison.

Statistics based on surveys should be accompanied by information about the survey design, the response rate and numbers of respondents. For some other statistics in the Fact Sheet more information on matters of timing would be appropriate – such as the period in which knife/other weapon crime was committed, as opposed to the period in which the offender was sentenced.

The Fact Sheet mixes references to: teenagers, young people, young victims (under 20), and offenders. Further, it mixes references to “assault by a sharp object (including knives)”, “serious knife crimes”, “knife crime”. It also refers to “youth violence”, which is not necessarily directly related to knife crime. Terms (such as “young people”, “knife crime” etc) are not defined.

Selective or otherwise inappropriate comparisons
Data for October 2008 and June 2008 are compared several times. This is not self-evidently appropriate because seasonal factors may affect such comparisons. Furthermore, monthly figures are inherently more volatile than averaged figures.

Some statistics refer to the 9 English TKAP areas, some to the 10 in England and Wales, without clear explanation or justification.

The proportion of people caught with a knife and charged is given only for London, as is the change in the numbers of young victims of knife crime.

Lack of contextual information
“Hospital admissions 27% lower…” refers only to the 9 TKAP areas in England. There is no record of whether admissions were lower in the other areas – no figures for a ‘control’ group.

There is no indication of how knife crime has changed for the groups not reported upon.

“Over 105,000 stop and searches for offensive weapons” – no context is given for whether this number has gone up or down, or how it compares with that for other areas. Similar points apply to the statement about the number of shops tested for underage sales.

Youth violence is reported as being 30% lower in Halloween week than in the previous year. “Halloween week” is not a recognised period for statistical comparisons. And no evidence is given about the reasons for this change – it could be because of the weather or other external factors.

Drawing inappropriate conclusions
Some of the conclusions drawn are based on small numbers, and may therefore be unsafe.

Unsubstantiated claims
No evidence is given to back up the claim (in the HO Press Release) that ‘those caught with knives are now three times more likely to be sent to prison’.