Findings

Quality

Overall quality

2.1 The users that we spoke to were very positive about the quality of the Graduate Outcomes Data and Statistics, with some describing the statistics as ‘excellent’. One user noted that the systematic nature and the large size of the Graduate Outcomes data collection means that the statistical validity is far higher than if each Higher Education (HE) provider conducted their own research.

2.2 Jisc provides detailed information on the quality of the Graduate Outcomes Data and Statistics in the survey quality report section of the user guide, including commentary on the relevance, accuracy, timeliness, accessibility and coherence and comparability of the statistics. Jisc also provides some high-level information in the notes section of the summary statistics release as well an overview of historical issues on its data intelligence page.

2.3 Information on potential limitations, as well as useful contextual information, is included throughout the summary statistics release. One example is the Graduate Salaries and Work Locations section, which cautions against comparing salary data across years and contains expandable sections with additional information, such as ‘what are the upper and lower bounds for salary?’.

2.4 The amount of information that Jisc has published in the user guide is extremely positive. Sharing this information creates a strong sense of transparency and ensures that users have access to the information that they need to interpret the statistics and to support their own analysis. In addition, we commend the steps that Jisc has taken to draw all of this information together into a single user guide following feedback from our previous compliance check.

2.5 However, it may be difficult for some non-expert users to navigate the user guide given its extensive size. It would be helpful for Jisc to provide additional information in the notes section of the summary statistics release, such as the response rates and information on the potential limitations of the survey design, in order to support non-expert users.

2.6 The Graduate Outcomes Data and Statistics are accurate and reliable. These statistics are based on information provided directly by graduates through a well-designed survey (see the cognitive testing technical report and the cognitive testing outcomes report for more information). In addition, the large sample size and strong response rates ensure a high level of confidence in the estimates. Jisc has conducted considerable analysis on the various factors of quality, including measurement errors, mode effects, reliability of sensitive data, and processing error. Through undertaking this analysis and sharing the outcomes publicly, Jisc has demonstrated its commitment to producing high quality statistics.

2.7 The timeliness of the production of these statistics is impressive given the level of quality assurance that Jisc conducts on the data. Jisc aims to provide the individual-level data to HE providers around three months after the closure of the data collection, with the summary statistics and open data released after a further two months. Whilst a few users told us that they would like the data sooner, most acknowledged that this would limit the time available for the quality assurance.

2.8 The Graduate Outcomes Data and Statistics are a vital source of information on graduates in the UK that can be linked to other Jisc datasets, including the Student Record and other information on HE providers. The statistics also complement the Longitudinal Educational Outcomes (LEO) data produced by the Department for Education (DfE), as well as statistics on employment and labour market produced by the Office for National Statistics (ONS), which are based on the Labour Force Survey.

2.9 Jisc intends to explore how the Graduate Outcomes survey data compare with similar variables in LEO and the employment and labour market statistics. We welcome this planned work and consider that it will provide valuable information to users on how the datasets can be used together as well as an indication of the extent to which the information across the data sources align.

2.10 Jisc has robust quality assurance processes in place for the Graduate Outcomes data which are outlined in the user guide under data processing. These include automatic validation checks within the survey that prompt respondents to revisit their responses if they appear to be contradictory or inaccurate, as well as checks on the Standard Industrial Classification (SIC) and the Standard Occupational Classification (SOC) classifications. The data are also cleaned, for example, to remove outliers to the salary question and to cleanse the free text responses.

2.11 Throughout data collection, HE providers receive a live feed of the data which enables them to identify and flag potential errors to Jisc for review. This collaborative approach ensures that Jisc benefits from the sector-specific knowledge of the HE providers and that the HE providers are able to directly contribute to improving the quality of their own data. Jisc’s statutory customers also play a key role in quality assuring the statistics prior to publication.

2.12 Jisc has conducted analysis into the impact of COVID-19 on the data collection. Its analysis showed that response rates were fairly stable during COVID-19 and that the data were broadly in line with those from previous years. As such COVID-19 is not considered to have had a significant impact on the comparability of the survey data. Further information can be found in the survey quality report, as well as a standalone impact report, and is included in the notes section of the summary statistics release.

Data on income

2.13 The Graduate Outcomes survey asks a voluntary question on respondents’ annual pay for their main employment. Generally, there is a greater number of respondents who do not provide an answer to this question (known as item non-response) compared with other questions on the survey. This is especially the case for those respondents who complete the survey over the telephone compared to online.

2.14 In the most recent survey of the 2020/21 graduate cohort, the response rates for the income question were 82.7% for telephone interviews and 90.9% for online mobile responses. In comparison, the preceding question on the currency that respondents are paid in, achieved response rates of 97.5% and above across all modes. For more information, see reliability of sensitive data.

2.15 Jisc notes that the higher levels of item non-response are likely due to the sensitive nature of an income question and that this is not unique to the Graduate Outcomes survey. Jisc has made some improvements to the question design over time to help address this issue. These improvements include providing additional hover text to reassure graduates about the use of their data and changing the order of the questions to ask the mandatory question about the currency that they are paid in first.

2.16 In addition to item non-response, Jisc receives some responses to the annual income question that are considered to be inaccurate. In order to improve the quality of these data, Jisc removes responses that are implausibly low (below minimum wage) or high (the top 0.1% of salaries) from the dataset.

2.17 Some users that we spoke to told us that they do not use the income data due to concerns regarding quality and some suggested that the question be removed from the survey. However, the Graduate Outcomes survey provides the only source of comprehensive data on graduate earnings. The only other alternative data source on graduate earnings is the LEO dataset which does not capture information on self-employed individuals.

2.18 Whilst there may be some limitations to the income data, we consider that Jisc has taken steps to implement improvements and explain the nature of the data to users to support interpretation. For these reasons, we consider that users should have confidence in these statistics at an aggregate level.

2.19 We understand that Jisc is planning work to link the Graduate Outcomes data set to LEO in order to assess the accuracy of the income responses. This will be an extremely valuable piece of work and will provide a strong indication of the quality of the data on income and we welcome the outcome.

Response rates

2.20 Generally, the response rates for all graduate groups are around or above 50% and have remained relatively stable over the four years of the survey, with the exception of the non-EU domiciled graduates group which is discussed further in the ‘International Students’ section of this report. For the 2020/21 graduate cohort, the response rate for the largest group of graduates – UK domiciled, full-time students – achieved a response rate of 52% (see quality report – executive summary). We consider these response rates to be a strength of the Graduate Outcomes survey, especially when considered in comparison to other large social surveys, and that users should have a high level of confidence in the aggregated estimates.

2.21 Jisc undertakes a range of activities during the data collection period in order to ensure a high response rate. During the 13-week field period for each collection cohort, all non-responding graduates and those who have partially completed the survey receive up to six emails and seven SMS messages. The exact number and timing of these reminders vary slightly from one cohort to another and are communicated in the engagement plan which is published for each cohort on the HESA website.

2.22 Jisc uses its strong relationships with HE providers to ensure high survey response rates, whilst minimising additional costs. Jisc has a comprehensive suggested graduate contact plan that encourages HE providers to collect updated personal contact details and prepare graduates for the survey before they are contacted by Jisc. HE providers also have clear guidance on promotion of the Graduate Outcomes survey which includes best practice such as maintaining contact with graduates and links to the Graduate Outcomes survey communications materials such as promotional items and social media resources.

2.23 Some users that we spoke to raised concerns about the response rates and noted that the UK domiciled, full-time students had not met the 60% target. Whilst response rates are one measure of quality, it is important to consider other factors such as how representative the responses are of the wider graduate population. Jisc has provided detailed information on this in the Sampling error and non-response error section of the user guide.

2.24 Whilst the Graduate Outcome statistics are strong at an aggregate level, we recognise that many users, including HE providers, analyse the data at very disaggregated levels. For example, a university may be looking to examine the outcomes for graduates of a specific course with a relatively small number of students. Some users had concerns about the quality and reliability of such analysis where the estimates are based on small numbers of graduates as a result of the response rates. Whilst we are sympathetic to the needs of these users, we consider that it is not a realistic expectation that a survey of this nature will provide reliable estimates of all possible detailed breakdowns. Jisc is considering how response rates can be maintained and potentially improved, but notes that it considers it unlikely that response rates could be drastically increased without incurring significantly higher costs for HE providers.

2.25 One way in which Jisc is exploring increasing responses rates is through strengthening awareness of the Graduate Outcomes brand so that graduates are more familiar with it when they receive the invitation for the survey. Information on this work is set out in a blog. Jisc is also exploring using contact details from the Student Loans Company, which may hold more up to date information than the HE providers.

International students

2.26 In 2021, DfE and the then Department for Business, Energy and Industrial Strategy asked Jisc to reduce the financial burden on HE providers. As a result of this directive, Jisc decided to stop calling international students (also referred to as non-EU domiciled) during the data collection period, as communicated in a blog. International students can still complete the survey online, but they do not receive calls from Jisc. Since this change, the response rate for this group has dropped noticeably.

2.27 Some of the users we spoke to highlighted the reduction of information on international students as a key data gap. HE providers require data on the international graduate population in order to provide tailored support for them and to ensure that they achieve good outcomes from the UK HE system. However, we also heard from other users, including Jisc’s statutory customers, that currently they have little need for this information.

2.28 Jisc is considering other actions it can take to improve the response rates for international students without significantly increasing the costs involved and are working with the users of the statistics to undertake a review this year.

SIC and SOC coding

2.29 As part of the Graduate Outcomes survey, information is collected to allocate SIC and SOC codes to each respondent. These codes provide valuable information on economic activities and occupations and the coding is conducted by Oblong on behalf of Jisc.

2.30 Jisc undertakes extensive quality assurance exercises on the SIC and SOC codes. These exercises include a random stratified sampling strategy to assess the data, working closely with HE providers to investigate any concerns they have raised, identifying and reviewing discrepancies between key information, and reviewing distributions over time.

2.31 Jisc also works closely with Oblong, which conducts its own quality assurance. Jisc has undertaken investigations into the reliability of the coding conducted by Oblong in comparison with coding undertaken by ONS. These investigations have demonstrated a high level of inter-coding reliability.

2.32 Each year, Jisc provides a report to HE providers on the quality of the SIC and SOC coding and the checks that have been undertaken. In addition, Jisc provides regular feedback to ONS to help inform future developments of the SIC and SOC frameworks.

2.33 Some users reported that they felt that there had been some issues with the SIC and SOC coding in the first few years of the Graduate Outcomes survey but that the reliability has improved. Other users felt that there were still questions around the reliability of the coding and that the move to a central collection by Jisc had limited the ability of HE providers to use their institution-level knowledge as part of the quality assurance process. Jisc reassured us that each SIC and SOC query raised by HE providers is fully investigated but that it is not always able to provide detailed information on the outcomes back to the providers due to the volume of queries.

2.34 Some users also raised issues with us that related to the SIC and SOC frameworks more generally. Some users felt that the frameworks are somewhat out of date and do not take into account new roles or accurately reflect the skill level of certain jobs. Most users recognised that this was a broader issue with these frameworks rather than an issue unique to the Graduate Outcomes survey.

Value

Uses of the statistics

2.35 The Graduate Outcomes survey was designed to replace the DLHE survey and was developed in conjunction with users. The Graduate Outcomes Data and Statistics provide valuable information on activities and characteristics; activities by previous study characteristics; salaries and work locations; and reflections on activities.

2.36 The Graduate Outcomes Data and Statistics are used for a variety of purposes including information on the graduate labour market, to support careers and employability services in HE providers, and examining the effectiveness of specific courses. The statistics also feed into league tables for HE providers and publications by other organisations such as the What happens next? report published by the Association of Graduate Careers Advisory Services (AGCAS). In addition, the statistics are used by regulatory bodies such as the Office for Students to monitor the outcomes for all HE students. The majority of the users we spoke to use the full range of the statistics and many receive the raw data from Jisc, which they then use to conduct their own analysis.

2.37 The key Graduate Outcomes findings are communicated through the summary statistics release which includes well-presented graphs and interactive tables. In general, the users that we spoke to felt that the commentary in the summary statistics release is presented well and tells the story of the statistics by drawing out the key messages of the statistics.

2.38 In addition to the summary statistics release, Jisc provides a number of outputs tailored to different users. The publication of the statistics is accompanied by posts on the HESA X account and Graduate Outcomes X account as well as supporting blogs, such as The impact of the COVID-19 pandemic on Graduate Outcomes 2020/21. For releases prior to the 2020/21 graduate cohort, Jisc produced webinars to support understanding of the statistics.

2.39 Users are able to download a wide range of tables and data from the Graduate Outcomes open data repository, which enables them to conduct their own analysis. Jisc also share some high-level results with graduates on the dedicated Graduate Outcomes website through an infographic and work is taking place to consider how more information on the results of the survey can be communicated to graduates.

2.40 Generally, the users that we spoke to were satisfied with the accessibility of the Graduate Outcomes Data and Statistics through the HESA website and commented that the open data tables were easy to use once you were familiar with them. However, some users expressed concern that less-experienced users might struggle with navigating the website as well as the tables in the open data repository.

User Engagement

2.41 Jisc has strong working relationships with the main users of the Graduate Outcomes Data and Statistics. In general, the stakeholders that we spoke to reported that Jisc was easy to communicate with and was helpful and professional when dealing with queries.

2.42 Jisc has obligations to a range of statutory customers in all four UK nations, including the funding and regulatory bodies for HE in each nation including the Office for Students, the Higher Education Funding Council for Wales, Scottish Funding Council and the Department for the Economy Northern Ireland. These statutory obligations require Jisc to provide these customers with the data they need to carry out their public functions.

2.43 In addition to these statutory customers, key users of the Graduate Outcomes Data and Statistics include HE providers, local and national governmental agencies, graduate employers and academic researchers. Beyond these key users, Jisc is considering how it can increase awareness of the statistics among a wider range of users including the graduates themselves. An overview of Jisc’s approach to users can be found in its user engagement strategy for statistics.

2.44 One way in which Jisc is looking to increase awareness of the Graduate Outcomes Data and Statistics is by encouraging HE providers to promote the statistics and use the new Graduate Outcomes branding with graduates. Jisc is also looking to host more external events to help promote the statistics as well as sessions for HE providers to attend. Jisc has been working with the British Council (the UK’s international organisation for cultural relations and education opportunities) to co-brand communications aimed at international students in order to lend greater authenticity to the Graduate Outcomes brand.

2.45 During the first few years of the Graduate Outcomes survey, Jisc regularly engaged with key stakeholders through a dedicated steering group. This steering group was recently dissolved, reflecting the integration of the survey as ‘business as usual’. Some users expressed disappointment around the dissolving of this steering group. However, the Graduate Outcomes Data and Statistics now form part of Jisc’s broader provider forum. Regular forum attendees are nominated by HE bodies, and statutory customers can attend on a necessary basis. This provider forum provides a platform for discussion and feedback with relevant sector bodies on Jisc’s activities.

Innovation

2.46 As part of the 2020/21 cohort survey, several questionnaire changes were implemented following consultation with various sector representatives as part of a wider survey review project. These changes resulted in a noticeable reduction in survey length and streamlining of various questions. Further details on these changes and other aspects of the survey review are available at Graduate Outcomes survey review – our approach to continuous improvement.

2.47 The next phase of the Graduate Outcomes survey questions review is currently underway and is largely aimed at strengthening the questionnaire further by reducing the data collection burden, improving data quality and improving the alignment of the data collected with user requirements.

2.48 Jisc is in the process of making the Graduate Outcomes data available on the ONS Secure Research Service. Once on this platform, the data will be available for approved researchers which is an extremely positive development and will be of great benefit to academics working in this space.

2.49 We are aware that Jisc is investigating different areas for modernisation including RAP. We consider this work to be very positive and encourage Jisc to share updates on this work publicly when it is ready.

Trustworthiness

2.50 Jisc’s commitment to trustworthiness is apparent in its public communications and actions. Jisc is evidently aware of its responsibilities as an official statistics producer, demonstrating an overall commitment to transparency. Through our engagement with Jisc and with users of the Graduate Outcomes Data and Statistics, we have seen a dedication to building trust through transparent process management, orderly release, and professional capability.

2.51 Jisc publishes a detailed list of people with pre-release access (PRA) to the statistics, which is broken down by job role and department. Jisc has also published a historic record of PRA lists for each publication, which demonstrates a commitment to maintaining transparency. Jisc also ensures its principles for PRA align with best practice, for example, by requiring those who want PRA to request it specifically, rather than them retaining PRA by default. This ensures that PRA to the statistics is limited, which improves security. During our previous compliance check, we advised Jisc to reduce the number of individuals with PRA for Graduate Outcomes, which Jisc has since actioned.

2.52 Jisc’s online documentation includes a comprehensive list of its current user engagement practices and plans for the future, demonstrating to users and stakeholders that those with an interest in the public good served by the statistics are being consulted. The statistics team demonstrates a high degree of professionalism in its communication with stakeholders. According to the users we consulted, the statisticians are very responsive and proactive.

2.53 Jisc provides a comprehensive and public summary of relevant privacy information relating to the collection, use and storage of personal data. A detailed list of FAQs is provided with clear answers. The privacy information also explains how Jisc adheres to General Data Protection Regulation (GDPR). This privacy information can be found on the graduate outcomes website and so is directed towards the most appropriate demographic; those taking the survey. This shows that Jisc is conscious of users’ concerns around how personal information is handled.

2.54 Jisc operates to a high level of compliance on data protection and information security. Data governance and data protection procedures are regularly reviewed, and any related risks are assessed and treated. Staff receive regular refresher training on data protection and information security, and staff access to the Graduate Outcomes microdata is tightly controlled.

2.55 Jisc has published a comprehensive revisions policy that breaks down how and when changes to publications are made, in accessible language. Jisc commits to areas of good practice such as avoiding revisions unless necessary and ensuring that all revisions are communicated to users. It would be helpful if Jisc included clearer links to the revisions policy, for example, from the summary statistics release or the user guide, in order to ensure that users are able to find clear information about the revisions process to maintain their trust in the statistics.