In our latest guest blog, Oscar Donnelly, Lead for the Mental Health Outcomes Framework, discusses the outcomes-based approach to collect standardised mental health data and statistics using the new patient record system ‘Encompass’…
In 2021, OSR’s Review of Mental Health Statistics in Northern Ireland highlighted a scarcity of robust mental health data for Northern Ireland, revealing significant data gaps. The lack of data on the outcomes of mental health services, in particular, made it impossible to answer questions people may have, such as whether services are effective in the treatment and care they provide.
These concerns were amplified in a 2023 Northern Ireland Audit Office (NIAO) report that pointed to a need for improved data around mental health services. NIAO emphasised that establishing an appropriate framework for measuring service outcomes would help support decision-making and the monitoring of services’ effectiveness. It was clear to us that addressing these concerns would be an ambitious but necessary undertaking.
Our Ambition to Improve Mental Health Data
Northern Ireland’s Mental Health Strategy 2021-2031 recognises the deficit in information on mental health care outcomes, similarly highlighting a need to “Develop a regional Outcomes Framework in collaboration with service users and professionals, to underpin and drive service development and delivery.” The strategy indicates that a Mental Health Outcomes Framework (MHOF) would support the use of evidence as the foundation for decision-making. Further, it should establish a comparable data set across the five Health and Social Care Trusts to allow us to measure performance and determine how best to improve health and social care services for the Northern Ireland public. As a result, in 2021, I was asked by the Department of Health (DoH) to lead on the development of a Mental Health Outcomes Framework for Northern Ireland. This is the story of how we developed and are implementing the framework and reflects on our progress and priorities so far.
Developing the Framework
During early engagement, I was advised by a psychiatrist colleague that for a Mental Health Outcomes Framework to be accepted and used in services, it would need to be clinically relevant to staff and meaningful to service users. So, it became clear to me that the primary focus of such a framework should be to improve outcomes for service users by supporting evidence-based clinical practice. Collecting data on overall service activity and performance, although important, is secondary to this objective.
A regional steering group comprising a range of stakeholders from across professions, mental health services, organisations and sectors was established to oversee the work. This included people with lived experience as service users and carers and representatives of both the DoH Information and Analysis Directorate and the Planning and Performance Group regional commissioner. Further input and review were provided by academics from Queens University Belfast and a consultant clinical psychologist with expertise in mental health data and statistics, outcome measurement and analysis and, most importantly, the practical use of outcomes measures within services.
The steering group met monthly, with a smaller working group drawn from the steering group meeting weekly to drive the work forward.
Three stakeholder workshop events were held in producing the framework: two for mental health service users and carers and one for mental health professionals drawn regionally from across service types and providers. A reference group of professional staff was also established to advise on the selection of appropriate measures for different mental health services.
The Outcomes-based Accountability approach
The Northern Ireland Government uses an outcomes-based accountability (OBA) approach to outcomes measurement, which we adopted for the framework. OBA measures are built around three core questions, which we used to inform the basic structure:
How much did we do?
We completed an audit that identified the poor quality of currently available data on community mental health services for adults and older adults. We found that the deficiency in these data relates both to the data being generated through a range of independently developed Trust IT systems and, more problematically, the variability of basic mental health service structures across Trusts. As a result, the routine collection of regionally consistent data is highly challenging. To address these issues, we developed a “basic metrics” template to improve the quality and consistency of data on mental health activity, applying a regional reporting template to structure the information inputted by each Trust.
How well did we do it?
We considered both service users’ and carers’ experience and reflected on what is meaningful to them when they access mental health services. These measures were co-produced with service users and carers and then tested in services. Separate questionnaires were developed for service users and for carers, with adapted versions for each of the Child and Adolescent Mental Health Services (CAMHS) and perinatal mental health services.
Is anyone better off?
Working with clinicians from across services and Trusts, we identified a range of appropriate and validated tools for measuring clinical outcomes. These included universal measures of emotional wellness to be used across all mental health services and a portfolio of condition and service specific measures. This portfolio reflected that people access mental health care for many reasons, so services should measure outcomes across a wide range of individuals presenting with varying needs. We identified a total of 47 measures of clinical outcomes for the MHOF.
Implementing the Framework
We engaged widely with international mental health outcomes programmes, particularly in Australia and Scotland, to learn lessons and maximise the success of the framework. This helped us identify that the implementation phase is where outcome measurement initiatives are most at risk of either failing or underachieving. The plans for implementation and operation need to inform each step of the development of a mental health outcomes framework.
Most importantly, implementation must be seen as more than just a technical exercise; one of the biggest challenges to success is ensuring that mental health practitioners and services value and routinely use the outcome measures in their day-to-day practice.
Ensuring that outcomes measures are accessible and easy to use in busy clinical settings is therefore critical. It is hugely opportune, then, that Northern Ireland is currently investing in a new regional electronic patient record system: Encompass. This system will replace the various electronic patient record systems across Trusts with one that creates a single digital care record for every person in Northern Ireland receiving care. In this way, Encompass will be an important enabler to ensure successful implementation.
Where are we now?
The co-produced regional MHOF was approved for implementation by the local health minister in October 2022.
We are currently in Stage 1 of implementing the framework. Our aim, during this phase, is to embed the framework measures during the build of the Encompass system and establish the capacity to report robust regional data on mental health service activity. We plan to complete this work by early 2026.
Work is progressing, with approximately 60% of measures having now been quality assured, license arrangements sorted, and formats digitised onto the Encompass system. We are also engaging with mental health and informatics staff across Trusts to implement basic activity metrics.
A challenge around capacity has also emerged. As the Encompass system starts to go live in each Trust, Encompass systems staff will be confronted with competing priorities and naturally, they prioritise supporting the clinical functionality of the new system over other development work.
What’s next?
Stage 2, which we hope to start in 2025, will involve supporting mental health professionals and services in using the outcomes measures embedded in the Encompass system during Stage 1. This will require a phased programme of engagement with teams and clinicians and regional clinical networks to select appropriate measures for similar services across Trusts and to regionally operationalise these in a robust and consistent way. It will involve training and supporting teams and services across Trusts to integrate the use of outcomes measures within their clinical practice. It will also require engagement with service users to test and evidence its acceptability and to determine how best to support them.
In Stage 2, we will explore international examples of services that have successfully implemented outcome measurement into their clinical practices. Our aim is to embed the framework with selected services to support outcomes reporting by late 2026. The second phase will also include the data collection and reporting of service user and carer experience, alongside the development of a patient portal on Encompass.
Implementing and operating an ambitious framework such as this requires dedicated resourcing. As such, a dedicated regional programme lead will be recruited, with this role expected to be filled by winter 2024/25. It is also hoped that MHOF champions will be appointed in each of the five Health and Social Care Trusts. Though these roles are still awaiting funding, they will likely be critical to the success of the framework and its adoption by services.
Funding has been secured for a Regional Business Unit, who will work with the MHOF programme lead to ensure the development of the OBA reporting capacity in the Encompass system. Dashboards will report routine measurements as standard. They will also have the capability to further analyse MHOF data against other demographic and clinical information, increasing our understanding of how we can best respond to the needs of service users.
The future of outcomes measurement
Implementing outcomes measurement in mental health services is a long-term strategic commitment that requires ongoing engagement, support and practice development.
The framework’s multi-faceted approach to improving our information on mental health services will enable services to develop and adapt in a changing landscape where resources are limited and demand is growing. It will help us assess the effectiveness of services so that we can maximise their benefits, address issues such as long waiting lists and improve the quality of care and outcomes.
We expect to begin collecting statistics from the Encompass system from Spring 2026, which should address the issues around comparable data.
For further information on this work, please contact Oscar Donnelly, Chair of the Mental Health Outcomes Framework: Oscar.Donnelly11@outlook.com.