Findings

Communication of the quality of the statistics

1.9 The assessments of children’s progress against the Curriculum for Excellence (CfE) levels are based on teacher professional judgements in schools. These professional judgements are based on a wide range of evidence collected by teachers during the ongoing assessment of children and young people’s learning. This includes observing children and young people at work, assessing their work in class, standardised assessments and assessing children and young people’s knowledge and understanding by talking to them about their learning.

1.10 Through our stakeholder engagement, we heard from users within Scottish Government that they had a good level of trust in the quality of the statistics. However, we also identified that some academics had concerns about the quality of the statistics, primarily as a result of the use of teacher judgements.

1.11 Whilst we recognise that there will always be an element of subjectivity to these teacher judgements, based on the evidence gathered as part of this assessment, we are content that there is a strong quality assurance framework in place and that the process is robust. An overview of the quality assurance process is as follows:

• The teacher judgements are based on national benchmarks set by Education Scotland and are informed by a range of evidence.
• Education Scotland works closely with schools to provide guidance on the teacher judgements, to improve data quality and to support moderation activities. The      moderation process aims to ensure consistency in judgements across schools and Local Authorities (LAs) and is supported by the use of Quality Assurance and Moderation Support Officers.
• Moderation is undertaken at different levels as set out in Scottish Government’s 2019 Achievement of CfE Levels – Official Statistics Evidence paper.
• Teacher judgements are submitted to the corresponding Local Authority (LA) by schools. Each LA undertakes its own quality assurance before submitting the data to Scottish Government in line with the data specification.
• The data is submitted through ScotXed and is subject to a number of automatic validation checks that must be passed.
• Once a successful submission has been received, a second round of quality assurance is conducted by Scottish Government and potential issues are queried with LAs.
• As part of the submission process, LAs also provide information on the quality of the data through a survey.

1.12 It is our view that many of the concerns regarding the consistency of teacher judgements were a result of a lack of detailed information available to users on the quality assurance processes in place. In order to strengthen the credibility of the Achievement of CfE Levels statistics, Scottish Government should provide users with a greater level of detail on the quality assurance processes that are undertaken at each stage of data collection and processing.

1.13 As noted above, Scottish Government collects information on data quality from each LA through a survey. This information is useful in monitoring overall data quality and Scottish Government publishes a brief summary of this information in the methodology section of the background notes to the statistics. In order to communicate potential uncertainty and to ensure that users are able to draw reliable conclusions from the statistics, Scottish Government should provide more detail on the information gathered from the quality survey where possible. It would also be beneficial for Scottish Government to demonstrate their commitment to continuous improvement by setting out how the information in this survey is used to improve the quality of future data collections.

1.14 Scottish Government provides a good level of detail about the methods used to produce the statistics in the background notes section of the bulletin. This includes information on pupils in special schools or standalone special units, derived variables, Gaelic-medium education and data matching. Scottish Government also provides information on the limitations of the data. One such limitation is the way in which pupils with complex needs are treated across different LAs, with some LAs placing these pupils into special schools/standalone special units and others integrating into mainstream schools. In order to support users in drawing accurate comparisons between LAs, we encourage Scottish Government to provide additional information on this limitation, such as the approach taken by each LA where known.

1.15 There have been several changes to the statistics since they were introduced in 2015/16 including the relabelling from official statistics in development to official statistics in 2018/19. In addition, COVID-19 prevented the statistics being collected in 2019/2020 and impacted the 2020/2021 collection. As this is still a relatively new collection this means that the comparable time series is somewhat limited, but Scottish Government is confident that the series will be stable in future. Scottish Government provides useful information in the background notes on the uncertainty of comparing the statistics over time due to changing methodology and how this affects how they can be interpreted. This includes a clear flowchart demonstrating how the data and the robustness of the statistics have changed over time. Further information around the impact of COVID-19 on data collection and quality can also be found in the background notes with detail on how many LAs were impacted.

1.16 Due to the devolved nature of education, we acknowledge that it can be difficult to draw direct comparisons across the UK nations. It is positive that Scottish Government provides signposting links to the relevant websites for the other UK nations within the statistical bulletin. We encourage Scottish Government to provide more information, where possible, on the key differences and the level of comparability with the other UK nations in order to support research and analysis at a UK level.

User engagement

1.17 We identified a varied user base for the Achievement of CfE Levels statistics within the education sector in Scotland including Scottish Government, Education Scotland, LAs and schools. The Achievement of CfE Levels data are the only pupil attainment measures in Scotland prior to secondary school formal exams, so Scottish Government policy makers and other officials are regular users of the statistics in order to evaluate the success of the Curriculum for Excellence. Users within Scottish Government told us that they regularly communicated with the statistics team on issues such uncertainty and confidence levels to support their own interpretation when presenting the data to ministers.

1.18 The data are also used alongside a suite of measures to assess progress towards closing the poverty-related attainment gap as part of the National Improvement Framework. Understanding the levels of deprivation using the Scottish Index of Multiple Deprivation (SIMD) and how they affect pupil attainment is particularly important for LAs and academic users undertaking research in this area.

1.19 HM Inspectors within Education Scotland evaluate detailed school performance by analysing raw data at school level which they obtain through a data sharing agreement with Scottish Government.

1.20 In general, the users that we spoke to were aware of the developments that had been made to the statistics since their introduction including the relabelling from official statistics in development when they were first launched in 2015/16 to official statistics in 2018/19. Users reported that the gap in data collection during the school closures as a result of COVID-19 had affected the year-on-year comparability in recent years but that the reasons had been explained well in the bulletin.

1.21 There are relatively few users of the Achievement of CfE Levels statistics outside of the education sector and these users were primarily in academia. We heard from a few academics that they were aware of the statistics but that they were cautious about using them due to their concerns regarding the perceived quality of the statistics. We consider that building stronger relationships with the academic community will enable Scottish Government to address these concerns and strengthen the credibility of the statistics.

1.22 Given that these statistics provide a unique perspective of attainment throughout the Scottish education system we consider that there is the potential to widen the user base further including to parents, researchers, and other interested parties. Expanding the user base would increase the value offered by the statistics and would enable Scottish Government to provide further assurance on the reliability of teacher judgements. We recognise the current resource constraints on the Achievement of CfE Levels team, however we consider that, where possible, Scottish Government should seek to understand their potential user base beyond the education sector and proactively engage with these potential users to raise awareness of the statistics and increase trust in the statistics.

1.23 Some users outside of Scottish Government that we spoke to, including those in academia, felt that there had been minimal engagement with them by Scottish Government on these statistics in recent years. The Code of Practice for Statistics states that users and other stakeholders should be involved to help prioritise statistical plans. In order to maximise the public value from this important data source, we recommend that Scottish Government engages proactively with users to seek feedback on the statistics including potential areas for improvement. To support this engagement and to keep users informed, Scottish Government should publish a workplan detailing plans for the ongoing development of the statistics.

Public value

1.24 The Achievement of CfE Levels statistics are published on a timely basis with the collection taking place in August, based on a census date in June, and the statistical bulletin published in December.

1.25 On the whole, users were content with the frequency and timeliness of the statistics. The timeliness of the data was mentioned by some users who thought there was slight time lag between collecting the data from schools based on a census in June and publishing the statistical outputs in December. However, most users understood the reasons for this being due to the rigorous quality checks that the school submissions go through combined with the schools’ summer holidays and the start of the academic year. In general, the annual frequency was considered adequate as long as there was assured quality and users understood the need to balance this with the burden on teachers.

1.26 The Achievement of CfE Levels statistical bulletin presents detailed breakdowns by various demographic and socio-economic variables including Scottish Index of Multiple Deprivation (SIMD), pupil characteristics, and by LA. Generally, the users that we spoke to were happy with the level of detail that was provided in the statistical bulletin. We heard how the statistics help users to understand the national picture, trends across councils, gender breakdowns and further information about pupils with additional support needs (ASN), amongst other areas of interest.

1.27 Since the introduction of the statistics in 2015/16, Scottish Government has undertaken a large amount of work in developing the statistics and embedding the quality assurance processes. Scottish Government is now confident that the production of the statistics has stabilised and so we consider this to be a good opportunity for Scottish Government to explore additional insights that could be drawn from the statistics where available resources allow. For example, some users expressed that cohort-level data at a national level would be beneficial so that users could track cohort series through the years and evaluate their attainment levels.

1.28 In addition to the statistical bulletin and data tables, Scottish Government produces primary school and secondary school information dashboards which contain a wealth of school-level information drawn from different sources including some measures from the Achievement of CfE Levels statistics. Many users we spoke to were unaware of these dashboards or had experienced technical issues whilst using them. Based on our own research it was not clear what type of user the dashboards were aimed at, or how Scottish Government were engaging with these users to raise awareness of the dashboards and to seek feedback on potential improvements.

1.29 We consider that the dashboards have the potential to be an important source of school-level and comparator data to non-expert users such as parents and school staff. While the Achievement of CfE Levels statistics team does not maintain the dashboards, it would be beneficial for the team to work with the relevant teams across Scottish Government to identify the needs of the dashboards users and the extent to which these are being met, to promote the dashboards to wider users and to seek input on potential improvements.

1.30 The Code of Practice states that opportunities for data sharing and data linkage to enable the reuse of data should be taken wherever feasible. We understand that work is currently underway to make the Achievement of CfE levels data available on the Administrative Data Research Scotland data platform. We consider that making the data available for further use by approved external researchers would greatly increase the value of the data and the potential insights that can be drawn. We welcome Scottish Government’s ongoing work in this area and encourage the statistics team to consider additional ways that they could increase the reuse of these statistics.