Chapter 2: Quality

Findings

2.1 PaNS data are produced from an online panel survey run by a contractor. The PaNS team at Natural England has a strong relationship with this data provider with regular contact ensuring continuous data quality improvements. There are known issues with the representativeness of panel samples and Natural England has published a good summary of the limitations within the published survey methods and technical details for the People and Nature Survey. Natural England also publishes detailed information about the quality assurance checks that the data go through within the PaNS team and by the data suppliers, some of which were developed in collaboration between the teams.

2.2 At the start of this assessment some parts of the Technical Report did not reflect current developments. It is good that the team reviewed and refreshed the Technical Report as a matter of priority and published an updated version to reflect the latest plans in November 2022.

2.3 Users we spoke to reported that they have confidence in the quality of the survey data and that they found the quality of the PaNS data appears to be improved compared to its predecessor the MENE survey (see para 1.1).

2.4 Natural England had planned to parallel run PaNS with MENE to enable backwards harmonisation between the two surveys and provide a consistent time series. However, COVID-19 restrictions stopped this from happening as MENE was a face-to-face survey. The team sought advice from ONS’s Methodology Advisory Service who concluded that the backwards harmonisation would not be feasible due to the time that has elapsed since MENE stopped, the different survey modes used (MENE was a face-to-face survey and PaNS is an online panel survey) and changes to the questions. The team understands that additional work needs to be done to explain the discontinuity between the two sets of statistics to users and plans to explore the issues during its next Steering Group meeting later this year.  Users we spoke to also expressed a need for information on the comparability between MENE and PaNS so that they could look at the longer-term trends in people’s interactions with nature.

2.5 Work to implement Reproducible Analytical Pipelines (RAP) is ongoing and is increasing the automation within the quality assurance processes. Introducing automation into the production of statistics helps to reduce the potential for human error and also frees up time, allowing statistics producers to spend more time on other aspects of the production process. We have published a review into the use of RAP principles and overcoming barriers which may be useful to the team as a more automated approach is considered.

2.6 Within the survey, participants are given the option to answer ‘Don’t know’ or ‘Prefer not to say’ which for the latest headline figure of ‘People who had visited green and natural spaces in the last 14 days’, accounts for around 10 per cent of answers. This level of ‘Don’t know’/’Prefer not to say’ could affect the interpretation of the statistics as it is not known what impact the true number of visits made by these participants would have on the average number of visits, and how this changes over time. It is reassuring to know that the data supplier checks for high levels of these types of answers, but there is potential for further work to be done to understand why survey participants are answering with “Don’t know” such as through adding a follow-up question within the survey. This further work should help to identify if there is any potential misunderstanding of some of the survey questions, the need for any adjustments (such as the introduction of specific guidance) and by reducing the number of ‘Don’t know” answers, it should allow a more robust dataset to be produced.