Annex 1: Preliminary findings from the first phase of this assessment

Finding 1

1. Some users have reported difficulties in locating certain documents on the Census offices’ websites. The design and navigation through these websites vary notably between Census offices and may provide a barrier to the easy access of information for users. Census offices should consider the accessibility of research and other Census information on their websites and consider aligning website design and content where possible to provide a common user experience. 

1. Actions taken by NRS to address findings NRS has made changes and improvements to its Scotland’s Census website. NRS has made the link through to webpages on Scotland Census 2021 more prominent and added hyperlinks to content to make the user journey easier. New content has been added, some in response to other assessment findings such as highlevel information on the use of administrative data. NRS has used social media and other communications routes to highlight website changes and updates.  

NRS is undertaking development of its Census outputs website. The phased approach taken by NRS is in line with Government Digital Service (GDS) and Digital First Service Standards models, and subject to assessments by the Scottish Government’s Digital Assurance Office at Discovery, Alpha, Beta phases and prior to Live. In July 2020, NRS undertook usability testing as part of the Alpha phase of the project and has been active in seeking involvement from Census data users for example through its news pages and twitter.  

NRS has been working collaboratively with the Office for National Statistics (ONS) and Northern Ireland Statistics and Research Agency (NISRA) on both changes to its current Census webpages as well as on the development of its Census outputs website. 

1. OSR’s evaluation of evidence The evidence shared by NRS on both the current Census webpages and in the development of its output website shows it has taken on board this finding and has user experience embedded in its digital planning and developments.  

We would encourage NRS to share its experience of its Census digital developments – particularly in the context of developments being taken forward by the other Census offices. We consider NRS’s approach to have strong elements of best practice which would be valuable to share with colleagues across the GSS.  

NRS should continue to consider the accessibility of information published on its website as more content is released in future. 

Finding 2

1. There has been much public debate relating to proposals for Census questions and the question development process across Census offices. Census offices must ensure their actions and decisions are visibly taken with the honesty, integrity and independence necessary to deliver data and statistics that are of high quality and serve the public good. Census offices should be open and transparent on their decision-making processes and in their decisions on Census questions and guidance, particularly in relation to any areas of contention. 

1. Actions taken by NRS to address findings NRS has published a variety of materials on its development, processes and decision on questions for Census in 2021. These are collated and are accessible from a single webpage. 

NRS, in its evidence report to OSR, provides a summary of its development of the sex, sexual orientation and trans history questions and related guidance. It describes the wider context of the development, stakeholder engagement activity, and research undertaken. NRS has published the related materials from stakeholder events and research and recommendation reports on its website.  

Since the assessment report was published and as part of its planned process for question development, NRS has continued to appear before Scottish Parliament’s Culture, Tourism, Europe and External Affairs Committee. NRS addressed and gave evidence in relation to Committee questions, many of which focused on the Census sex question.  

NRS has faced much public scrutiny and criticism in its development work on the sex question and guidance. NRS has told the assessment team how it feels it has benefited from this and welcomed the challenges from different stakeholder groups.  

NRS has kept users informed of legislative processes through its newsletter and legislation pages of its website. 

1. OSR’s evaluation of evidence We consider the range of information published on its question development webpage offers users and stakeholders evidence of its decision making on Census questions and guidance. We note that NRS has now provided a link to the sex question recommendation report alongside previous research captured in topic reports. 

NRS’s approach to transparency is further reflected in its appearances at the Culture, Tourism, Europe and External Affairs Committee to discuss its decision making regarding Census questions. Evidence given at the Committee hearings would further support the published material available on NRS’s website. NRS should consider linking to Committee reports and minutes from its question development webpages.  

NRS should continue to engage with stakeholders particularly in relation to areas of contention, meet any commitments it has made, and seek to provide answers or explanations on areas of concern, in a transparent and open way. 

Finding 3i 

1. Census offices could benefit from working together more closely to better engage with users and stakeholders of UK Census data and statistics who have views and insights that are relevant across Census offices. UK Census users and stakeholders may be unnecessarily overburdened if they are required to submit multiple feedback or consultation responses on similar topic areas across Census offices. Census offices should consider how best to engage with users and stakeholders of UK Census data and statistics users and coordinate activities as appropriate. 

1. Actions taken by NRS to address findings Since the publication of the assessment report, NRS, working with the other Census offices, has:  

• established a UK Census Data working group. The working group is specifically looking at the assessment finding on UK Census data users. The group has been considering options such as UK wide events. 
• added a new UK Census data webpage, inviting UK Census data users to contact the Census team. The webpage should serve as a clear route to publish materials of particular interest to UK users. 
• attended cross-office user events. While this already happened, steps have been taken to be more visible at these events through ensuring people from other offices are introduced to attendees. 

In its evidence report NRS gives an example of balancing Scottish user needs and UK harmonisation in its consideration of developing UK harmonised outputs areas – as Census output areas in Scotland are smaller than those throughout the rest of the UK. 

1. OSR’s evaluation of evidence We consider the UK Census data working group an extremely positive initiative put in place to address this finding. We are encouraged by NRS’s engagement with this group and the steps taken so far.  

The new UK Census webpage is a positive step and having tailored content for this specific group of users is a valuable addition. 

We recognise the barriers that Census offices may face in identifying and engaging with this group of users. The assessment team is hopeful that the continuing efforts of the working group will help with this. The working group should look to increase the visibility of its activities and communications and draw on the expertise across the wider GSS in exploring new and creative ways to reach users groups that are harder to engage with. 

Finding 3ii

1. Published plans, research and topic reports from the individual Census offices are not always sufficiently clear on how country-specific proposals then relate to UK users’ needs and about the impact of these decisions on the availability of on harmonised UK data. Census offices should be clear about the impact of country-specific decision making for UK Census data and statistics and work together to provide greater transparency around their plans and decision making in meeting the needs of users interested in UK Census outputs. 

1. Actions taken by NRS to address findings NRS, with the other Census offices, has released a progress update on the Conduct of the 2021 Censuses in the UK in January 2020, reflecting the position as of November 2019. The statement of agreement on the conduct of the Censuses sets out the principles that the three Census offices will work together on to make sure the 2021 Censuses are successful. We understand there are plans to release a further updated position at the end of 2020. 

The UK Census Data working group, which NRS is a part of, is compiling a descriptive list of all the harmonisation working groups and activities contributed to by the UK Census offices to capture the variety of work that goes into UK harmonisation. The UK Census data working group has plans to publish this information and the Census offices are considering whether the most appropriate vehicle to do so is alongside the planned update on the conduct of the Censuses.  

Some high-level summary information on UK harmonisation is included on the new NRS UK Census Data webpage. 

1. OSR’s evaluation of evidence There is an array of groups in place which consider harmonisation as part of Census plans and development. Putting information about these groups in the public domain will offer greater transparency to users and stakeholders.  

When collating this information, if feasible, examples of how decision making has taken account of the needs of UK Census users should be included.  

NRS should also seek to engage with UK Census users on their response to how it has taken account of their needs.  

NRS should be transparent in how it meets the needs of users interested in UK Census outputs in future publications on harmonised questions, outputs and methods. 

Due to the decision by Scottish Ministers to delay the Census in Scotland, it is even more important NRS must work with the other Census offices to explain the impacts of this to UK Census data users. 

Finding 3iii

1. Proposals from Census offices on Census questions will be subject to relevant legislative processes in finalising arrangements for Census in 2021. Census offices should provide users, stakeholders and decision makers with information on harmonisation of Census questions and the impact on outputs at UK level to help inform users and support decision making. 

1. Actions taken by NRS to address findings In response to this finding, NRS has described the collaborative approach to question development between the three Census offices.  

In its evidence report, NRS has shared its plans to review and produce information on harmonisation of Census questions when legislative processes are finalised and will likely publish this in 2021. 

1. OSR’s evaluation of evidence The assessment team recognises the commitment of NRS and the other Census offices to publish information on harmonisation for users. 

Finding 4

1. Census offices have a work programme scheduled to establish and evaluate the data sources that will be used to support Census activities. In some cases, we are not fully assured that the necessary conversations with all the various data suppliers are taking place to inform the understanding of the nature, operational context and overall quality of the sources to be used. Census offices should build their awareness of the relative strengths and limitations of any administrative, commercial or other data sources used in the production of Census outputs, by regular engagement with suppliers. This should be undertaken on an ongoing basis and as part of a normal way of working. 

1. Actions taken by NRS to address findings Through its evidence report, NRS has more clearly articulated its use of administrative data in the Census and the further opportunities it is exploring to secure other data sources such as prisons data from Scottish Government. NRS describes how it maintains relationships with data suppliers and how it assures the quality of the data through comparing with other data sources.  

The assessment team held a data sources session with the Census team in March 2020 to hear more about the administrative data and how the NRS quality assure this data, for example NRS’s arrangements to use NHS Central Register (NHSCR) data. The assessment team also spoke with a member of the Geographies team that supplies data to the census administrative data team. NRS has practices in place to quality assure administrative data for Census use and were applying Quality Assurance of Administrative Data (QAAD) principles in its ways of working. 

OSR also ran a QAAD session with the three Census offices in May 2020, to respond to queries raised on our expectations on for quality assurance of administrative data as part of Census. NRS attended and engaged positively in that session.  

NRS has created a new webpage which gives a summary of the use of administrative data in the Census. 

1. OSR’s evaluation of evidence Through the information published in its evidence report and from the data source workshop held, the assessment team is more assured of NRS’s approaches to administrative data and its relationships with data suppliers.  

NRS should continue in its work to secure and build its knowledge and awareness of the strengths and limitations of data sources used for Census.  

Being transparent and providing users and stakeholders with assurances on its judgements is a key aspect of the Code of Practice for Statistics. We consider that information on data sources used in the Census and NRS’s judgement on quality and appropriateness for use should be made available in the public domain. The level of detail should be proportionate to level of assurance required, and NRS should consider publishing provision information and building on this as its research and understanding continues to develop. 

Finding 5

1. Census offices are developing their quality management approaches and developing various methods to support Census operations, data processing and outputs. This includes addressing potential biases between online and paper collection modes. However, the timing for publication of methodology documentation and quality assurance arrangements is unclear. Census offices should make information on the methodology and quality assurance arrangements available to users at the earliest opportunity. 

1. Actions taken by NRS to address findings NRS has set out its approach to methods development, detailing factors it considers when developing methods and method review and sign off procedures. It describes how methods papers are peer reviewed through an Internal Peer Review Group before being submitted to its External Methodology Assurance Panels which is made up of subject experts and academics.  

NRS held a methodology stakeholder event in February 2020, giving an overview of Census data processing. The slides from this event were published on the NRS website.  

NRS also published a Statistical Quality Assurance Strategy explaining how it will assess and measure quality throughout the collection, processing, production and dissemination of Census data. This was promoted through social media when released. 

1. OSR’s evaluation of evidence The assessment team considers the approach taken to develop Census methods is robust. We recognise the transparency to methods development achieved through NRS publishing the External Methodology Assurance Panel methodology papers and summary reports on its website.  

These papers, which should provide more detail for expert users, and the stakeholder events for the non-expert, provides good coverage and range of communication across the variety of Census users and stakeholders.  

We welcome the publication of the Statistical Quality Assurance Strategy. It is a detailed, well-written document which provides assurance of the processes and strategies in place to assess and measure levels of quality. 

The assessment team would like to explore methods being employed by NRS for Census in more detail to cover how methods have been developed, what methods have been selected and why NRS consider these suitable for use. We are planning to hold workshops on areas such as census coverage adjustment and statistical disclosure control in the coming months. 

Finding 6

1. Users told us that they were unclear at what points certain decisions will be made or when certain information will be made available. While planning documents are published, there is no easily accessible highlevel plan of key Census milestones. Census offices should provide users with an indication of future Census milestones – including future user engagement opportunities, publication of further research or reporting, and legislative milestones – to provide an added level of transparency and support trustworthiness and public confidence. 

1. Actions taken by NRS to address findings NRS has added information on Census milestones to its website. A description and timeframe are given for each milestone, as well links to supporting information.  

1. OSR’s evaluation of evidence The assessment team is pleased to see that this overview information is now available to users and interested parties.  

NRS should update this information to reflect changes to the timing of the Census in Scotland and continue to maintain this information providing further detail or updates on timings as appropriate. 

Finding 7

1. Use of internal review, audit and other independent measures to evaluate the effectiveness of processes is important across all three Census offices and provides additional safeguards. Census offices should be clear to users what assurance mechanisms are in place and be open about identified areas for improvement in a way that is a proportionate and accessible to users. 

1. Actions taken by NRS to address findings NRS has published information on its webpage on the assurance mechanisms in place to underpin Census delivery. Information provided describes the “three lines of defence” model – day-to-day management controls, Census Programme Management Office assurance processes, and external assurance for example through Scottish Government’s audit functions. 

NRS has also published its evaluation of the 2019 Census rehearsal which sets out key findings, areas for improvement, and next steps planned for further work. 

1. OSR’s evaluation of evidence NRS has published the information required which allows users to better understand the broader assurance and audit activity in place to evaluate processes. 

We note the publication of the Census rehearsal evaluation report; it is positive that this information is in the public domain. The scope of the Census rehearsal undertaken in Scotland in 2019 meant some aspects such as field force and Census Coverage Survey were not tested. The assessment team considers that NRS should supplement the report with information on how it is mitigating any risks associated with the reduced scope of the rehearsal.