Data Quality

There is a lack of robust and reliable data on mental health in Northern Ireland. This has had a profound impact on the provision of official statistics in this area, in particular for mental health data sourced from the community sector. NISRA have advised us that work is ongoing to develop experimental statistics in this area, however data quality issues are a significant barrier in reducing the current data gaps.

Inconsistent data collection and definitions across the trusts has led to a fragmented and potentially misleading regional picture

The responsibility for the delivery and provision of healthcare sits with each of the five Health and Social Care Trusts (HSCT) in Northern Ireland. Mental health services form part of this provision, with the trusts responsible for the delivery of mental health services on a localised and geographic basis. The autonomy of the Health and Social Care Trusts has led to inconsistent localised data collection methods and definitions, with no regional co-ordination to advocate for collaboration and standardisation across the trusts. The absence of regional harmonisation across the Trusts has created a barrier, meaning it is often impossible to carry out any comparisons between the trusts and an accurate picture of mental health service delivery for the region as a whole is missing.

We heard during our conversations, that terms such as ‘contact’, ‘new patient’, ‘review patient’ can be defined differently by the trusts. Equally the way in which trusts name appointments can vary, e.g. for waiting time statistics, ‘referral’, ‘assessment’ and ‘treatment’ appointments may differ and be counted differently. The lack of regional definitions, such as this, can lead to information being presented by the trusts in a misleading way and leads to an inability to harmonise across the trusts.

Data and statistics should be compiled using recognised standards, classifications and definitions. They should be harmonised to be consistent and coherent with related statistics and data where possible. The Department of Health (NI) need to develop consistent and clear operational definitions across the five trusts in collaboration with clinicians and practitioners. We are pleased to see that the development of a regional mental health service in the Mental Health Strategy 2021-2031 aims to improve this consistency across the five trusts.

The current IT infrastructure does not support robust mental health data collection – disparate localised systems and old inefficient ways of data collection are hindering progress and improvements to mental health data

Official statistics are underpinned by source data; in this case, collected during the delivery of health care services in hospitals and in the community. The foundations of good quality statistics start from the point of data collection.

At present, there is no high-quality data recording system that is used consistently across each of the five Health Social Care Trusts. Data is collected and stored within silos. The IT infrastructure is disjointed, with localised data held by the trusts for the management of service delivery within that area. Not only are IT systems or system versions different between trusts, but also within the same trust. We heard that data collection from the community is a particular challenge due to the number of disparate systems used. On the ground, clinicians are unable to view day-to-day caseloads due to the software used in community-based information systems. They told us that there are limitations with the software which hinders their ability to record information accurately. For example, in the Northern Trust in children and adolescent mental health services (CAMHS), clinicians cannot input multiple diagnoses or multiple clinicians for one individual due to a limited number of fields in the software. This has huge implications on the quality of the data – making the collection and collation of mental health data very difficult.

We heard that a lot of data collection within the trusts from the community is paper based, not codified and cannot be analysed. This is an old and inefficient way of recording data and does not support the production of statistics at an aggregate level. There is a distant relationship between individual practitioners inputting the data and service development. We understand from our user conversations that some practitioners are not convinced on the usefulness of the data and that it is not a priority for those working in crisis teams. Mental health clinical coding and recording is also affected by the multiple systems used across the trusts. Different versions of ICD-10[5] mean that any changes in clinical code are really difficult to implement. The HSCB are responsible for clinical coding training across Northern Ireland, funded by the Department of Health. They offer free training, yet we heard that uptake from the trusts is limited. Accurate coding at source is essential for providing consistent, accurate and reliable information.

For users we spoke to, the absence of a consistent high-quality data recording system across healthcare in Northern Ireland was the key contributor in stalling progress and improvements in the publication of robust mental health data and statistics.

Encouragingly, progress and development of a digital solution to improve the disparity in technology across the health and social care sector is underway. The encompass programme is a NI-wide initiative to introduce a digital integrated patient record across health and social care to improve patient safety and health outcomes. The aim is to work more efficiently and effectively. The proposed solution, Epic, will be a single system across both physical and mental health systems, replacing lots of the different systems currently in place. The new system is scheduled to be in use across all five trusts by 2025, with the first trust: South Eastern to go live in Spring 2023. Almost all users we spoke to as part of our review, mentioned the encompass programme and have real aspiration for the improvements it will make – we share those aspirations.

Statisticians from the Department of Health (NI) and clinicians must be involved in the development stages of the Encompass programme. Their involvement will maximise the benefits of a change in data recording of this scale and ensure that the data collected will be fit for statistical purposes. Epic will improve the consistency of data collection across NI and enable more accurate and robust mental health data and statistics.

Whilst full implementation of Epic across Northern Ireland is some time away, there are pockets of improvement and innovative data collection methods being trialled in mental health services. We heard that to improve understanding of patient outcomes, the Northern Trust were piloting an online system, Checkware, in Children and Adolescent Mental Health Services (CAMHS) to replace a cumbersome paper system, improve efficiency and insights. We also heard of a programme of work co-ordinated by the Health and Social Care Board (HSCB) to improve mental health data flows from the community.

The Business Services Organisation (BSO) delivers a range of regional business support functions and specialist services to the health and social care sector in Northern Ireland. One of those functions is to provide regional IT systems to support health and social care organisations. The Regional Data Warehouse (RDW) forms part of the current regional IT infrastructure for health data. Data supplied by the HSC Trusts is extracted, transformed and loaded into the warehouse. Currently, the data flows for mental health and learning disability are incomplete – there is no data flow for community mental health activity, and inpatient activity data has been disrupted due to some of the trusts changing their local IT systems around 3-4 years ago.

To tackle this issue, a mental health informatics user group was set-up to look at how this could be improved. We heard that the aspiration is to establish an agreed minimum dataset between each of the HSC Trusts, although progress to achieve this ambition has been slow. Changing priorities, the cost to the trusts for technical adjustments to the software and limited staffing resources have all been barriers. The mental health informatics user group meetings are currently paused. We were told that this was due to the re-prioritisation of staff to COVID-19 information duties.

To improve the collection of robust and accurate mental health information in Northern Ireland, it is essential to develop clear, regionally agreed data definitions and guidance to support an agreed minimum dataset across all settings. This should be seen as a priority, recognising that full implementation of Encompass is some time away. As an example, the Mental Health Services Data Set (MHSDS) is an information standard for mental health data derived from local information systems in England. The MHSDS statistics published by NHS Digital provide a national picture of the use of mental health, learning disabilities or autism services in England and is used by a wide range of users.

We are encouraged to see that as part of the Mental Health Strategy (2021-2031) under ‘Data and outcomes’ there is an outcome to establish a robust dataset which is comparable across Trusts to measure performance and to determine what works. As a starting point, we recommend that the mental health informatics group is re-started to reignite conversations in this area.  Prior to full implementation of Encompass, the Department of Health (NI) in liaison with the Health and Social Care Board (HSCB) and the five Health and Social Care Trusts (HSCT) must collaborate together to establish a minimum dataset to collect mental health information across all settings. We would advise the Department of Health (NI) to liaise with NHS Digital given the MHSDS information standard in England.

[5] ICD–10 is the 10th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD), a medical classification list by the World Health Organization (WHO)

Statistical quality guidance should support user interpretation, including the comparability of statistics within the UK and internationally where relevant

The key official statistic and lead measure for informing the Programme for Government’s indicator 6: improve mental health is the GHQ 12 scores from the annual Northern Ireland (NI) Health Survey published by the Department of Health (NI). The General Health Questionnaire (GHQ) is a screening tool designed to detect the possibility of psychiatric morbidity in the general population. The questionnaire contains 12 questions about recent general levels of happiness, depression, anxiety and sleep disturbance. An overall score of between zero and twelve is constructed, with a score of 4 or more being classified as a respondent with a possible psychiatric disorder and referred to as a ‘high GHQ12 score’. The Northern Ireland (NI) Health Survey covers a range of topics and it includes a number of modules of questions around different policy areas, including indicator 6,  GHQ12.

Users raised concerns with us about the potential use of these data as a prevalence indicator, given that the Health Survey is a screening tool and not a diagnostic tool. We heard misleading comparisons are sometimes made, particularly by the media when comparing the prevalence of mental illness across the UK. From our review of mental health statistics in England, we know that the Adult Psychiatric Morbidity Survey (APMS) was considered the best source of information on the prevalence of mental health conditions in England. The APMS survey is a diagnostic tool, therefore any comparisons of prevalence between England using the APMS and the NI Health Survey; a screening tool, would be misleading. In order to prevent misleading comparisons such as this, the Information Analysis Directorate (IAD) in the Department of Health (NI) should consider whether additional guidance and advice on appropriate use of the Northern Ireland (NI) Health Survey is required.

We know from our wider regulatory work that comparisons of health care data across the UK are challenging, and mental health data is no exception to this. For Northern Ireland, comparisons are also commonly made by users with the Republic of Ireland (RoI), which again is often tricky given the different health systems and data reporting.  Comparisons can be misleading and users voiced their concerns about this to us. Statistical guidance must be clear and support user interpretation of the statistics and data. Separate to our review, its worth noting that the Government Statistical Service (GSS) Harmonisation Team recently found inconsistency across surveys in the UK around mental health questions. The mental health harmonisation guidance published in 2021 aims to assist statistic producers to improve the consistency and accessibility of their statistics for users. It provides a useful list of available data sources for recent adult mental health data (since 2010) across the UK, including definitions, legislation, clinical measures and relevant reports and is a useful resource for users and statistic producers alike.

Users have confidence in official statistics published by NISRA.  Users told us that the data quality guidance which accompanied published statistics is produced by NISRA in a transparent way. However they felt there was room for improvement. An example of where guidance could be improved for users is the compulsory admissions under the Mental Health (NI) Order 1986 in mental health and learning disability hospital statistics. These statistics are badged as National Statistics and should meet the highest standards of trustworthiness, quality and value. We heard from expert users that limitations of the data were known through experience and familiarity with the data. The figures include those admitted to hospital voluntarily and excludes patients who are admitted to hospital and later detained under the Mental Health (NI) Order 1986 – users reported that this was less obvious to non-expert users and the guidance should be clearer and more detailed.

Quality information for existing mental health statistics should be reviewed by the Information Analysis Directorate (IAD) at the Department of Health (NI) in order to ensure that it is clear, relevant and meets the needs of a broad range of users.