Blogs

Authored by Mustafa Al-Haboubi, Josephine Exley and Nicholas Mays

At the outset of the pandemic, digital contact tracing (or exposure notification) apps were widely promoted as a key weapon in the fight against COVID-19. Early modelling suggested that apps could effectively suppress transmission rates without the need for stringent “lockdown” measures if they were widely used by the public. Our latest research suggests we can draw three key lessons from the experience of using the app in England and Wales. These insights should be highly relevant when responding to future pandemics.

High levels of support

Before the roll-out of the apps, surveys in the UK and other Western countries suggested high levels of support for, and willingness to, download and use such technology to control the COVID-19 pandemic. People appeared to be more concerned about the virus and less so about any potential intrusion on their privacy.

The NHS COVID-19 app was launched in England and Wales in September 2020, six months later than originally planned, and following highly publicised technical issues resulting in the switch from a centralised design to a decentralised model based on technology developed by Apple and Google.

Our study

We recruited two groups of online study participants to track the changing views of the public in England and Wales on, and their use of, the contact tracing app. One was a representative sample of 2,032 smartphone users aged 18 to 79, the other a sample of 684 smartphone users aged 18 to 79 from six of the largest minority ethnic groups, who were purposively selected for the study. We surveyed the two groups seven times between October 2020 and September 2021.

Motivated by civic, public and social responsibility

Despite changes in government policy and case numbers, installation of the app remained stable at around 50% of smart phone users, with the majority of those who had ever installed the app doing so soon after it was launched. People’s desire to contribute to the common good was the main reason given for uptake among early adopters. Uptake was also higher amongst those who considered themselves vulnerable to COVID-19 or were concerned about the risk the virus posed to themselves and the country. A perceived change in the severity of the crisis was a trigger for installing the app in later months.

How to overcome the "intention-action gap”

Results from our study highlight a potential “intention-action gap”, with installation rates lower than might have been expected based on surveys conducted before the app was rolled out. The 50% level of uptake was well below the level initially estimated as required to suppress the virus. Increased focus would have been needed to encourage those who had not installed the app initially to do so later. Requirements for use of the app, such as the need to scan QR codes to gain entry to venues, could have encouraged greater uptake. There was only a limited window in which QR codes were required. Indeed, displaying QR codes at venues was not mandated in Wales during the pandemic. A key lesson for the future is that to increase uptake, people to have a day-to-day personal need to use the app in order to carry out their normal activities.

Allaying privacy concerns

We also identified a need to improve trust in the information provided by government and allay privacy concerns, since both factors were associated with lower installation rates in England and Wales. The highly publicised delays to roll-out associated both with privacy and false alert concerns potentially undermined support for the technology. The government did little to counter untrue claims regarding the so called ‘Pingdemic’ in the summer of 2021. Amongst those who uninstalled the app in this period, we found an increase in the perception that the app was sending false alerts. In future, governments must address privacy concerns head on, from the outset, by demonstrating how the technology works and robustly refute misleading media coverage.

Reaching groups with lower uptake

There were systematic differences in uptake in England and Wales by area, ethnicity, education and employment status. The lesson here is that future efforts to promote app use need to rapidly identify and find ways of reaching groups with lower uptake. These disparities cannot be explained in terms of the “digital divide” since all respondents to our survey were smartphone users. The reasons behind these differences also need to be explored in future research before the next pandemic.

The publication can be accessed here

Funding

This research was funded by the NIHR Policy Research Programme through the Policy Innovation and Evaluation Research Unit (PR-PRU-1217-20602). The views expressed are those of the author(s) and are not necessarily those of the NIHR or the Department of Health and Social Care.

Authored by Josie Dixon, Josephine Exley, Gerald Wistow, Raphael Wittenberg, Martin Knapp and Nicholas Mays

Traditionally, research has found that people would prefer social care for older people to be funded by taxes or national insurance, like the NHS. However, more recent surveys appear to show increased support for shared funding. Shared funding is where services are paid for using a combination of Government funding and people contributing some of their own money ("out-of-pocket" payments).

We conducted 12 focus groups across England and used a framing approach to explore people’s reasons for supporting shared funding. The findings have been published in Health and Social Care in the Community. 

We found that support for shared funding was dominated by concerns about scarcity and affordability. Out-of-pocket payments were seen as necessary to support a fragile social care system. Sharing costs was also thought to be the fairest approach where neither individuals nor Government can afford to pay for care alone – described using phrases such as "splitting the difference", "fair on both sides" and "a mutual thing".

Given this scarcity, there was a limited vision for what social care could achieve, with a focus on funds being used to meet what were perceived as medically-related needs. This, however, was accompanied by associated anxieties about care quality, cliff-edge costs - whereby people have to find significant sums of money to pay for care very suddenly - and abandonment.

The private market for social care was viewed as potentially a way for individuals to obtain better care for themselves, but there were doubts about the availability of good quality care and concerns about what would happen to an individual when their money ran out. There was also a strong view that good, basic care should not be dependent on people’s ability to pay.

Despite concerns about scarcity and, often reluctant, acceptance that it may not be possible for social care to be funded by national tax or insurance contributions alone, there was also strong support for government making a much larger contribution than currently.

The full report can be accessed here >>>  

Authored by Bob Erens, Mary Alison Durand, Nicholas Mays

Lack of integration between health and social care can lead to inefficiencies, poorer patient experiences and outcomes, and less value for money. In England, there have been a number of initiatives over the past 50 years which have attempted to integrate health and social care services, but with little evidence they have had much impact on, for example, levels of emergency admissions or cost savings. Against this background, 14 local ‘Integrated Care and Support Pioneers’ were announced by then Department of Health in November 2013. Each Pioneer was to: encompass whole system integration involving health and social care; drive change at ‘scale and pace’; deliver improved patient-centred care and outcomes; and realise financial efficiencies. An additional 11 Pioneers were announced in early 2015.

PIRU has been evaluating the Pioneers since 2014. The Pioneer programme came to an end in March 2018. Since the end of the programme, we have continued to examine integration activities taking place in the 25 geographical areas that were designated as Pioneers. Our overall aim is to assess the extent to which the Pioneers, and subsequent integration initiatives, have been successful in making progress towards providing ‘person-centred coordinated care’, including improved outcomes and quality of care, in a cost-effective way. The evaluation consists of several elements, one of which is to understand the experiences of those implementing service change in the 25 Pioneer sites. One method for achieving this has been to collect data over time by carrying out annual surveys among ‘key informants’ (mainly senior managers) identified as being involved in the development or implementation of health and social care services integration initiatives at Pioneer level. Our fifth, and final, key informant survey was carried out in autumn 2020, about six months after the first national ‘lockdown’ was imposed in England on 23^rd March, due to the Covid-19 pandemic. Since the pandemic had significant impacts on England’s health and care system at all levels, and quickly placed substantial additional pressures on many health and care organisations and both managerial and clinical staff, our survey aimed to collect key informants’ perceptions on the effect of the pandemic on local integration activities.

Not surprisingly, the Covid-19 pandemic had a profound impact on the individuals and organisations involved in the planning, management and delivery of integrated health and social care initiatives. Two-thirds of the survey key informants said their own role had changed due to the pandemic, usually leading to the individual taking on additional pandemic-related roles and responsibilities.

In terms of the integration of health and social care services, the pandemic was said to have had both positive and negative impacts. Some of the positive impacts included:

• reinforcing the importance of providing ‘joined-up’ services, such as hospital discharge and transfer of care
• strengthening both strategic and operational relationships between local health and social care services, such as improved communication and trust between organisations, improved team working across organisations and lowering barriers between services such as increased information sharing
• much faster implementation of initiatives that would have taken far longer to put in place in normal times (e.g. especially the shift to virtual consultations)
• progress in reducing unplanned hospital admissions and helping patients to better manage their own care.

On the other hand, key informants identified some negative impacts arising from the pandemic including:

• generating considerable pressures on health and care managerial and clinical staff and services, which required significant extra staff time and resources to deal with
• a drastically increased workload leading to increased stress and reduced wellbeing among staff
• working from home leading to the breakdown of some working relationships
• the inability to visit patients’ homes during lockdown resulting in less insight into patients’ circumstances and social connections
• hindering progress in improving quality of patient care, and in making services more accessible and cost effective.

The overall impression gained from most key informants was that the positive impacts of the pandemic on health and social care integration at a local level outweighed the negative ones, at least at the relatively early stage of the pandemic. Moreover, many thought that the ongoing pandemic would continue to help strengthen local collaborations, partnership working, etc. over the next 12 months. However, the disparity in funding between health and social care, as well as between different sectors of the NHS, was identified as a potential challenge to working relationships. Several key informants thought some of the improvements (e.g. in working relationships) might dissipate over time, as pent-up demand and the impacts of the pandemic on patients and staff continued to take their toll. In the extreme, a few key informants thought the pandemic could even deter future collaborations and lead to a withdrawal into organizational silos.

Results from our survey appear to suggest that an extreme jolt to the system, such as that triggered by the Covid-19 pandemic, can lead to a number of developments that promote the integration of local health and social care services. Different services and organisations had to pull together to deal with the considerable challenges created by the pandemic and to find their own ways to overcome some of the regulatory, financial and other barriers to joint working. Whether local NHS and social care services are able to maintain these integration-related gains over time remains to be seen. Support from national policy-makers – e.g. for the continued development of local integrated care systems and primary care networks, and providing equitable and sufficient financial resources for both the NHS and local authority social services – will be important to enable the continued development of sustainable integrated services.

The full report of the key informant survey results can be accessed here >>

Results from the fifth survey (autumn 2020) of Pioneer Key Informants: perceived effects of the Covid-19 pandemic on integration activities. (July 2021) Bob Erens, Mary Alison Durand, Gerald Wistow, Agata Pacho, Mustafa Al-Haboubi, Lavanya Thana, Nicholas Mays.

In the midst of a coronavirus pandemic, the awareness of planning and preparedness for crises has never been more important. Images of local communities working together to support care home residents trapped by the floods of Storm Dennis contrast with more recent images of care home workers struggling to keep residents and themselves safe from Covid-19.The year 2020 has become a testing time for public health services and communities. The risk of heatwaves adds another layer.

Heatwaves are often referred to as ‘silent killers’. They may not have the same visibility as other emergencies such as storms or floods, nor be as deadly to a population as a virus pandemic.They are, nevertheless, a cause of much premature death and ill health; a severe heatwave occurring in the summer of 2003 accounted for over two thousand excess deaths in the UK. National heatwave plans go some way to safeguard people who are most at risk, especially those in health and care facilities, but there remain a number of groups who are still unaware of their heat-health risks and who do not follow public health advice.

As we enter the summer period in England, the risk of increasingly severe hot weather as a result of global warming presents an additional threat to people’s health, and provides a rationale for having clearly defined national and local strategies for managing heatwaves. This is especially important as heatwaves have been shown to have a significant impact on the health of older and more socially isolated people, and those with chronic respiratory and cardiovascular diseases.

Severe heat places an extra strain on the body to regulate temperature, and this risk increases with age, leading to heat-related conditions such as dehydration, heat cramps or, more seriously, heatstroke. It also accelerates other chronic conditions and can result in hospitalisation or death, even within the first day or two of a heatwave. It is agreed that many heat-related deaths and illnesses could be prevented with good preparation, responsive care, and individual behavioural change. However, despite England having a national plan to reduce the adverse health impacts from heatwaves, significant excess deaths and illness still occur, placing an extra burden on health and social care systems. For example, in 2019, the deaths of almost 900 people were attributed to high summer temperatures.

A heatwave plan (HWP) has been in place in England since 2004. We identified gaps in the system in how some potentially vulnerable groups were identified and protected during the heat-health alert period, particularly older people with health problems who were not in regular contact with health or social care services, many of whom may be living on their own. Many health and care managers relied on general public health information, through national or local media, to reach these groups, and assumed that recipients would act on that advice.

Most people had positive views of hot weather, and for many it was something they looked forward to as it invoked feelings of good health and well-being. In addition, most adults did not feel that hot weather posed a risk to themselves. As a result, protective measures were often not taken, even those that people felt were effective, such as staying out of the heat and in the shade. Younger adults were most likely to underestimate their risks and ignore public health advice on sunbathing and drinking alcohol in hot weather.

Similar results were found among people who were most vulnerable to the effects of heat, particularly in their perceptions of risk. For example, only 40% of participants aged 75 or older saw themselves personally at risk of hot weather. According to many of those interviewed, risk was associated with physical and mental frailty, not age itself, and there was some resistance, even resentment, to any ‘vulnerable’ label that may be applied to them in this context, as they did not consider themselves ‘frail’. Participants felt that hot weather increased their risk of sunburn and skin cancer, however there was much less awareness of the thermoregulatory risk of hot weather, and how this risk increased with age. Security and cost, in addition to self-perception of risk, were key considerations for this group. For example, many did not open windows at night for fear of intruders, or use an electric fan due to the cost of running one. The expectation that any period of hot weather would likely be over quickly was often weighed against any discomfort experienced.

As we move towards more frequent hotter summers, developing systems of identifying, reaching, and supporting our most at-risk communities will increase in importance. As a first step we recommend that Public Health England revise their advice and publicity to improve public awareness of the risks of hot weather. Messages should be tailored to the information needs and media usages of different population groups, and take account for the finding that many people do not self-identify as vulnerable, and therefore should focus more on the risks of hot weather, and help people self-assess their own risk more realistically without the label of ‘vulnerability’.

The published report can be accessed here>>

The situation of older people in care homes and the difficulty of protecting them during the current coronavirus pandemic attracts much attention these days, and rightly so. The devastating effect of the pandemic on care home populations highlights a number of shortcomings in residential care in England, which for years has neither received adequate levels of funding nor much policy attention. Yet over 350,000 older people currently live in care homes, because they require high levels of care during most of the day and night. By ‘care home’ we mean organisations that provide long-term accommodation and services to people who need 24-hour care and attention, including help with personal care (in a residential care home) and medical care (in a nursing home). A care home is usually a person’s last home, but it is nonetheless their ‘home’. So it is important that they are able to live their lives as fully and as independently as possible.

In England, consecutive governments have emphasised the importance of personalisation for people receiving long-term care services. Personalisation refers to the process of tailoring care services to the specific needs and preferences of service users. In the past, this has mostly involved community care for individuals who receive care in their own homes, with policy-makers giving less attention to personalisation in care homes. This is regrettable, as it reflects the way many people associate a care home with an ‘institution’ or ‘place of last resort’, rather than a destination of choice in later life. Most initiatives to make care more personalised have been driven by the charitable sector (e.g. through initiatives such as ‘My Home Life’) or by care homes themselves.

Our study has sought to investigate how personalisation in residential care is conceptualised, which approaches are likely to succeed in promoting personalised care, and which barriers need to be overcome to provide an effective service to people living in a care home.

There is a difference of view of what personalisation means in the context of a care home. Essentially there are two sets of ideas, pulling in different directions. The first emphasises residents’ choice and control over the services they receive. This idea was influenced by attempts to promote ‘voice and choice’ among patients receiving health services in the NHS. Choice and control have been given prominence in domiciliary services for service users eligible for local authority support, who are given a choice between accepting care services organised by the local authority, and accepting a budget to organise their own care in the form of a direct payment. In this context, choice and control are often used interchangeably with ‘self-directed support’, which has been a key aim first set out by the disability movement. This movement has sought to increase the inclusion of people with disabilities in society, and to improve the services they receive. Its success has led to a major shift in awareness of what people with disabilities are able to achieve and the quality of the care and support received. It has been argued that this is due to the effects of individuals holding the purse strings for their care on the balance of power between them and the service provider. However, research has shown that many older people in residential care are less likely to benefit from choice and control in this way, given the extent of their care needs and the nature of how care homes are financed.

The second set of ideas developed out of concern for the quality of care for people with dementia. Typically referred to as ‘person-centred care’, it emphasises the importance of care and caring, and the role of the professional carer in promoting the wellbeing of the person receiving care. This concept gives the carer and the care home the responsibility of ensuring that people with dementia are supported in maintaining their identity and to articulate their wishes and preferences. A central aim is here to help people with dementia to maintain a sense of continuity between their lives before and after the onset of dementia, and their experience before and after moving into a care home.

Clearly, these two sets of ideas are not mutually exclusive. Proponents of person-centred care would always emphasise the importance of giving residents choices and control over their daily activities and to support them to lead as fulfilled lives as possible. Likewise, the idea of choice and control does not preclude people from enjoying caring relationships: indeed, most direct payments in domiciliary care are used by service users to select and employ their own carers.

However, these two sets of ideas differ in how they conceptualise the approach to personalisation and make different assumptions about the person receiving care. While person-centred care is rooted in a philosophy that accepts that the need for care and support is part of the human condition and a universal experience, choice and control casts the individual as a consumer in a market for care. They therefore differ in their assumptions about the relationship between the individual and the provider of care, the ability of the person to exercise choice and make their own decisions, and by extension, the person’s cognitive capacity. They also make assumptions about the level of support required for individuals to derive benefit from such choices and decisions and how this support is organised and funded. Both versions of personalisation have cost implications. Both necessitate adequate levels of staffing and training. Yet while the investment in staff is associated with the provision of customer care in one model, the other model requires an adequate number of skilled care staff to be able to build the relationships that underpin the provision of person-centred care.

In our study, we found both sets of ideas reflected in the care homes. When interviewing managers in care homes, almost all recognised the concept of person-centred care, noting the importance of trust between staff, residents and their family members; a cornerstone of personalised care. Some evoked images of the ‘family’ of care home residents and staff to describe these close, trusting relationships. In contrast, others used the image of the care home as a ‘hotel’ to express their aspiration to provide comfort and convenience to their customers, as if they were ‘on an expensive holiday’. Such customer care was reflected in practices such as providing menus on the tables of their ‘restaurant’ (rather than dining rooms); and it resonates with practices in certain types of more upmarket homes.

What are the implications of different concepts of personalisation in care homes? In principle, the availability of different models of personalised care enhances choice. If someone wishes to live in a ‘hotel-style’ care home they should be able to do so, even if the type of care such homes provide may be less focused on personal relationships. In practice, however, people with fewer resources have considerably less choice of care home than those with greater resources. People with low incomes, few savings and no relatives able to contribute to the costs of their care face a choice limited to those care homes which accept residents at local authority rates. Those with relatives able to top up local authority rates have more choice of care home. People with high incomes or substantial savings, while not eligible for local authority support, have the widest choice of care homes. This means that there is socioeconomic inequality in choice of care home. In addition, there is also the question as to whether the model of personalisation they offer, perhaps especially for ‘hotel-style’ homes, will be suitable for people whose care needs are substantial and who may be better served by a different care model such as one modelled on small ‘family’ units.

To date, the diversity of approaches to personalisation in the care home market and their implications for service users are not well understood.When the care home sector emerges from the pandemic, time should be taken to reflect on the funding and policy support needed to develop a sector that is not only more resilient to emergencies, but is also more equitable. It is time to reimagine the future of residential care to create homes that people would like to live in as a form of supported, communal living, rather than a place of ‘last resort’.

Ettelt S, Damant J, Perkins M, Williams L, Wittenberg R (2020): Personalisation in care homes for older people. London, Policy Innovation and Evaluation Research Unit.

A summary of the report can be found here >> and the full report can be found here >>

Advocacy of co-produced research

Many, if not most applied research projects, are undertaken with some degree of collaboration between researchers and ‘stakeholders’, who include research funders, policymakers or practitioners, members of the public and civil society, or other actors such as patients in health studies. This collaboration takes various forms (e.g. consulting on research topics, or working iteratively throughout the research process), and in general, people feel positively about it. Funders often require it, or at least support it, and researchers have argued that it is the most effective way to ensure that evidence is used or translated into practice. Today, there are many forms of collaborative research practice, including co-production, co-design, co-creation, stakeholder and public engagement, participation/involvement and integrated knowledge translation which sit under this umbrella. They reflect a very diverse set of motivations, activities, identities and discourses about how research interacts with the rest of society. Here we use ‘co-production’ as shorthand to discuss these practices, but acknowledge the diversity and differences in the above.

People argue that collaborative and co-productive research

• Helps with better-quality research, through improving our understanding of issues, mechanisms, and, of course, the research questions of most importance
• Makes it more likely that this research will be acted on
• Is the right way to do research (especially if publicly funded), being more inclusive and less elitist
• Makes potential users feel more empowered, trusted and more persuaded of the importance and veracity of the research findings.

Facing up to the challenges and costs of co-production

Undoubtedly, making research more relevant and used is the aim for many applied researchers, and their attempts to do so and document this are laudable. There is a need to help uncover if, when, and how collaboration is the best way to achieve this (normative) aim.

Firstly, there is very limited empirical evidence about whether collaborative research does lead to improved uptake of findings – even if we could agree what that looks like in reality.

Secondly, collaborative research brings significant challenges. Collaborative research may be more uncertain, slower, or less innovative than non-collaborative research. These challenges and related costs may be experienced throughout the research process, as the lists below indicates.

Challenges of coproduction

Developing mixed research teams

• Stakeholders not homogeneous, and can disagree
• ‘Usual suspects’ can take over, where coproductive discussions are dominated by certain individuals

Framing research questions

• Stakeholders and researchers may have different priorities and values
• Useful research can lack originality
• Research can be co-opted by partners, for example, to justify status quo or historical decisions

Collecting data

• Researchers may pressure stakeholders to allow their organisational resources to be used to facilitate data collection – e.g. using staff time or applying pressure for site access

Analysing and interpreting data

• Stakeholders may want to know which participant agreed to participate or what they contributed to the dataset
• Stakeholders may want to help analyse the data

Formulating recommendations

• May be little agreement about the importance of research
• Researchers may be pressed to frame findings in particular ways

Disseminating research

• Researchers or stakeholders may be prevented from sharing unwanted findings
• Stakeholders may want to share findings before researchers are ready

Implementing change

• Tension between advocating for research, or advocating for policy/practice changes
• Researchers show little interest in providing assistance with implementation efforts

Costs of coproduction

Developing mixed research teams

• The research process may take more time compared to a traditional research process
• Shared decision-making is threatened when process dominated by certain voices or interests

Framing research questions

• Damage to interpersonal or organisational relationships
• Damage to research careers
• Damage to researcher independence and credibility

Collecting data

• Damage to interpersonal or organisational relationships, particularly with more powerful stakeholders

Analysing and interpreting data

• Violation of research ethics obligations
• Researcher needs to train stakeholders and format data in an appropriate way to conform with research ethics obligations

Formulating recommendations

• Findings are misrepresented
• Damage to researcher independence and credibility

Disseminating research

• Damage to researcher independence and credibility
• Damage to the credibility of the research process

Implementing change

• Can damage relationship with practice or policy colleagues
• Implementation of research findings fail

Finally, there are other very significant potential costs of co-production, which, in our experience are often unequally borne by junior, untenured, female members of staff.

• Practical and administrative resources such as the time required to arrange meetings with busy people for whom research is not their primary activity. Managing group dynamics in such meetings can require interpersonal skills with which researchers may not be endowed.
• Stressful interpersonal interactions: these can be dismissed as mere relational difficulties. We must take the potential for disagreement, conflict, reputational risk and power imbalances seriously as part of the research process. The consequences of mismanaging these are severe.
• Individual researchers already have to balance their teaching, research and administrative responsibilities. Developing another set of professional skills and networks to create collaborative research projects with real world impact is simply too great a leap for some. The implication of this is that some or all of the co-production activities related to a research project could be led by a specialist in knowledge transfer and exchange rather than by members of the research team. This approach has been advocated and implemented by some.
• Some feel that taking part in applied, highly collaborative research can lead to researchers becoming, or seeming partisan and biased, or as academic “lightweights” producing little of substance
• The research outputs themselves may be co-opted to serve the political agendas of others. Co-produced research findings (possibly no different from any research) can be appropriated and used to serve the self-interest of more powerful groups. Some groups lack the skills to engage in the use and promotion of research findings so lose out to more skilled and better connected groups. There is also the risk that co-produced research is more likely than other forms of research to produce findings biased in favour of prevailing norms of what is ‘correct’. This last type of research leads to repetitious, ‘safe’ research

What do we need to do differently?

We may be able to avoid some of these costs, or they may be an intrinsic part of collaborative processes, in which case we need to work out the best balance between costs and benefits.

Armed with a better understanding of the costs and benefits of co-production, those planning new projects should be much better placed consistently to ask themselves:

1. What is everyone bringing to the table? For example, policy-makers and funders bring money, knowledge of the political context, pressure for answers; researchers bring topic and methodological expertise; the public and patients bring their experiences.
2. Under which circumstances are these resources needed, for what purpose, and at which stages of the research process? For example, when is it better to have patient representatives articulate the user perspective rather than derive understanding from a systematic review of patient experiences?
3. What are the costs, and how will they be borne and defrayed by those involved?
4. How will decisions about the direction of the research be taken, and how will responsibility and accountability for decisions be shared? Will group dynamics, market forces, formal authority or some other basis be used? In turn, how will this be governed?

In parallel, research organisations and funders also need to consider:

1. How to create (co-create) and support the infrastructure and leadership for co-production
2. How to provide training in co-production, and help interested researchers and funders take this seriously as a necessary skill
3. How to reward good practice, and to recognise the work co-production may take even if it does not lead to research impact
4. How to evaluate the potential impact(s) of co-production
5. How to ensure that co-production supports diversity and quality in research and policy

Conclusion

Co-production is an exciting approach to research that can, with care, generate truly novel, unexpected findings and impacts. Yet it takes time and investment, and there is still little evidence about how co-production changes research, policy or practice, or how it compares to alternatives. We think more reflection about how, why and when we do co-production would be helpful, as would more discussion about how co-production influences the process of research, and the roles and responsibilities of everyone involved in collaborations. Ultimately, we need to be clear about how and why exactly we work with each other, to transform evidence for society.

This blog grows from a paper by the authors published in Health Research Policy and Systems 2019: 17: 33 https://doi.org/10.1186/s12961-019-0432-3

Kathryn Oliver is Associate Professor of Sociology and Public Health, Department of Public Health, Environments and Society, London School of Hygiene and Tropical Medicine, UK

Anita Kothari is Associate Professor, School of Health Studies, University of Western Ontario, Canada

Nicholas Mays is Director of the Policy Innovation and Evaluation Research Unit (PIRU) and Professor of Health Policy, Department of Health Services Research & Policy, London School of Hygiene and Tropical Medicine, UK

Public service reform can benefit from outcome-focussed approaches to commissioning. However, we should avoid assuming that Social Impact Bonds are likely to have wide application, explain PIRU’s research team.

Social Impact Bonds (SIBs) are likely to become just one part of the search among policy makers, practitioners and researchers for viable approaches to outcomes-based commissioning of public services. That’s what we conclude in our assessment of the SIBs Trailblazers in health and social care in England, set out in our recent report. Here we reflect on a number of issues raised in our own research and by previous blogs in the PIRU series on SIBs.

Testing, as yet unproven, outcome-based models of commissioning is important for public services, which are often clear about their goals but hamstrung by inadequate implementation including unsupportive funding mechanisms. In health policy, for example, there is a goal to develop healthier citizens – not just to treat more sick people. Reaching that goal could reconcile the ambition to provide better health care with the need to control budgets. However, how can health and social care – and the other parts of the public sector – work together to achieve such goals? We lack a tried and trusted mechanism.

Focus on outcomes-based funding

In this light, the focus that SIBs provide on developing outcomes-based approaches to commissioning services is welcome. SIBs in health and social care have the potential to break through institutional inertia and generate innovations in public service delivery. At best, they can free and motivate front-line service provider staff to focus strongly on the needs and desired outcomes of their clients.

One of the issues holding back public service innovation, particularly in the last decade, has been a lack of funding for prevention as the demands of those currently in need of support have become ever more pressing. Amid austerity and lack of time for contract management, commissioners have found it difficult to think strategically. In this context, SIBs have been welcomed by policy-makers, commissioners and providers, drawn by the promise of upfront funds to develop services designed to improve outcomes in the longer term.

Issues with SIB upfront funds

What’s not to like about that promise? We have two caveats. Some Third Sector organisations lack the capacity to monitor user outcomes well and thus to demonstrate accountability for the upfront funding that they receive. Thus, upfront funding needs to be used not just for service delivery, but also for developing capacity at the provider level in areas such as outcomes measurement and data management.

A second - and perhaps more fundamental - concern about SIBs’ provision of upfront funds is whether this is an efficient way to generate extra cash. The injection of more funds can be seen as a good thing per se, but, it brings with it two further problems.

First, investors are often risk-averse, which can get in the way of innovation, a raison d’être for SIBs and for outcomes-based funding in general. There may also be clashes between the values of private investors and those of the ultimate payer for results – the public commissioner on behalf of the public/tax payer. Additionally, there may be adverse impacts on Third Sector staff’s motivation if the pursuit of specific outcomes is prioritised ahead of meeting the specific needs of individual clients.

Second, many SIBs are set up with the help of intermediary organisations which can add to their cost. As a result, these costs have to be set against the value of any client outcome improvements. The activities of such intermediaries may also inhibit capacity building at the provider level if they assume long-term project management and monitoring roles.

Need for experiments to stretch the SIB model

Ultimately, the public purse foots the bill for SIBs, even if this only takes place after the successful delivery of agreed outcome improvements. Yet, there may well be other more efficient approaches to outcomes funding. We may need to have experiments that stretch the already flexible SIB model and compare it with other approaches. So, for example, Big Society Capital, the Big Lottery Fund, or other socially minded investors could experiment with conditional grants, rooted in outcomes-based contracts, to spur innovation in service delivery, and compare these with SIBs. In her contribution to this blog series, Katy Pillai, Investment Director at Big Issue Invest, signalled that investors are keen to support research into what works best in this regard.

In our experience, most up-front SIB finance does not come from business investors. It comes from philanthropic investors, alongside central government sources such as the Life Chances Fund. For example, the eventual outcomes payments made to investors in the Trailblazer SIBs which we studied were mostly paid for by tax-funded public commissioners upon delivery of the service. Upfront philanthropic funds (investors) were useful and the Big Lottery (which provides a quasi-governmental subsidy for outcome payments) helped to fund the additional effort needed for commissioners to consider and experiment with outcomes payments.

Given how SIBs are largely working this way in practice, would it be wise to experiment with a different SIB model that removes the commercial imperative altogether? This SIB model is, in ways, a variant on the experiments suggested above for socially-minded investors such as Big Society Capital and the Big Lottery Fund. It could amount essentially to organising forgivable loans to Third Sector providers from philanthropic agents, alongside dedicated contract and performance management support. Such an approach would remove the financial architecture of ‘investment’ that surrounds SIBs at the moment but could reduce their transaction costs and accelerate the innovation process.

Clarify values of outcome-based contracting

A big issue when discussing SIBs – or any form of outcome-based funding – is to define their purpose. Are they simply about driving efficiency and making savings to the public purse or are they about transforming the social value of services? Are they a tool to create better monitoring and evaluation cultures in non-profit organisations, so improving accountability and value for money? What are the values being prioritised in such contracts? Thinking and practice in the Asia Pacific region, discussed in the blog by Chih Hoong Sin and Ichiro Tsukamoto, focussed, for example, on a further role for SIBs in transforming the relationship between civic society and the State.

What PIRU and other research has found

So far at least, it has not generally been possible to show that SIBs have produced cashable savings to the public purse. Globally, the main demonstrable success of SIB projects has been in helping marginalised groups that had previously been badly neglected by public services. Here, there was often a baseline of little or no provision, so it was highly likely that anything would have helped in such circumstances. In areas with some previous provision, it is less clear that SIB-funded services, for example, for people with chronic health conditions, have led to superior improvements in health.

Next Steps

The general argument, in principle, for developing outcome-based commissioning and contracting is strong, given the difficulties that policy makers have in matching their goals with the limited funding mechanisms currently available to them. However, we are some distance from resolving all the difficulties in making an outcomes-focus work in public service commissioning and in establishing the case for applying these approaches at scale. So, it is wise to experiment widely and to keep learning.

In this light, policy makers should focus on the components within SIBs that show promise in strengthening outcome-based contracting, while avoiding the notion that the whole package offered by a SIB is necessarily the way forward. SIBs, as currently conceived, may have some dedicated roles, but these are likely to be in particular circumstances (for instance, when outcomes are relatively easy to measure and relatively uncontroversial). The greater value of SIBs and related thinking, in the long run, may be in informing the wider development of other models of outcome-based contracting for mainstream service delivery rather than in replacing these approaches.

Stefanie Tan and Alec Fraser are Research Fellows at the London School of Hygiene & Tropical Medicine and part of the Policy Innovation Research Unit (PIRU) at LSHTM.

Nicholas Mays is Professor of Health Policy at LSHTM and Director of PIRU.

SIBs are emerging as a way to reconfigure relationships between the state and civil society in ways that their Western inventors may not have anticipated but may wish to replicate.

Japan’s experience of Social Impact Bonds (SIBs) offers some important insights. First, it challenges the notion - held by some - that SIBs should be seen chiefly as a way to cut costs: in Japan we see a typically Asia-Pacific framing of SIBs that’s more focussed on developing civic society. It’s an approach that may catch on more widely around the world.

Secondly, the Japanese experience demonstrates the importance of institutional infrastructure and culture – and not only availability of finance - to make SIBs work well. SIBs require a complicated culture in which to thrive: Japan shows how it can take time for the various elements to be established.

Japan expressed an early interest in SIBs, becoming part of the OECD Social Impact Investment Taskforce. A five year nationally-funded research project, based at Meiji University in Tokyo, looking into developing SIBs for Japan is almost complete.

On the face of it, Japan might look to be in the same boat as, for example, the UK. The country’s national debt at the end of the last fiscal year was 254% of GDP, compared with the UK’s 88%. So, one might think, Japan would use SIBs to drive down spending and increase efficiency.

In fact, although Japanese rhetoric initially focussed on fiscal savings, interest swiftly shifted to SIBs steering improvements in societal wellbeing that might require more, rather than less, public spending. Japan has also been much more like other Asia-Pacific countries where SIBs have been seen as a means to reconfigure relationships between the state and civil society and focus on long-term issues of public health.

Past and existing projects have looked at, for example, how to create learning supports in deprived areas for NEET young people – those not in education, employment or training. A nationwide health SIB has just been launched focussing on better managing or preventing long-term conditions such as dementia, diabetes, cancer, cardio vascular disease.

Japan’s government has shown a high level of interest in SIBs. It’s seen its role largely as being to signal interest and legislating to make money available. In December 2016 Japan harnessed dormant bank accounts, like the UK has done, to set up the equivalent of Big Society capital, which is envisaged to go live from 2019. In addition, the Ministry of Economy, Trade and Industry (METI) and the Ministry of Health, Labour and Welfare have provided pilots around the country with funding for operations.

So pilot programmes have tended to suffer not from shortages of capital. Their problems have, rather, concerned lack of measurement or been because service providers are short on thinking or capacity to deliver outcomes-based contracts.

Institutional weaknesses in Japan

Japan has been institutionally handicapped in developing SIBs in a number of ways. The country has a weak public management culture: transparency, a focus on efficiency and on prioritising clients’ views of outcomes are less developed than in the UK.

The country is only slowly creating the building blocks for outcomes-based markets. The UK government has supported the creation of outcome payers: SIB investment funds such as the Innovation Fund, the Fair Chance Fund, Commissioning Better Outcomes Fund, and the Life Chances Fund. Britain has invested in developing an evidence base that can support outcomes-based contracting. There is also legislation that directly or indirectly supports SIBs, including the Public Services Social Value Act and provision of social investment tax relief. Japan has had a much less developed basic infrastructure.

In comparison, the UK’s voluntary – not-for-profit – sector is also more robust, diverse and heterogeneous, including organisations of different sizes. There are strong umbrella bodies such as the National Council for Voluntary Organisations. Japan has a long history of social sector organisations but the flowering of the not-for-profit sector is more recent, encouraged in the past 20 years by legislation liberalising regulation of the sector.

Many Japanese organisations remain small and hyper-local, which raises questions about their capacity to engage with SIBs. The country’s not-for-profit umbrella bodies are also relatively weak in terms of their lobbying capacities and the extent of their networks across the sector.

There are fewer charitable foundations than in the UK. So, social sector organisations in Japan largely rely largely on government funding, whereas UK social sector organisations tend to have more diverse funding. This financial dependence makes it harder for social sector organisations to question government.

Problems measuring impact

These days, the UK social sector is familiar with measuring and demonstrating impact. Japan’s social sector organisations are less experienced in measurement and, where they are, their focus tends largely to be on activities and inputs.

Lack of diversity in markets - in terms of service providers, investors and intermediaries – also hampers the growth of SIBs in Japan. Until 2017, when the Yokohama SIB was developed, all pilots were driven by the Nippon Foundation or its subsidiaries. So a single, large player was trying to nudge the market along.

There are significant differences between the UK and Japan in the maturity of intermediary markets. The UK has moved from a virtual monopoly among a few intermediaries in the early days of SIBs to a growing, diverse market, bringing in many different skills. Intermediaries no longer offer only financial backgrounds. There are people with commissioning support experience as well as good provider engagement experience. The same cannot be said quite of Japan, where the emergent intermediary market remains in its infancy.

From the commissioner side in Japan there remains a poor understanding of SIBs so it is difficult to collaborate with providers about constructing SIB models. As a result, commissioners may not know what to specify. They tend to identify general outcomes which may lack meaning and be unachievable within the timescales.

Building underdeveloped civil society

Nonetheless, Japan’s vision for SIBs is innovative as it tries to use them to place individuals at the heart of public services that have tended to be dominated by centralised bureaucracies. As in China, SIBs offer Japan a model that facilitates an underdeveloped civil society sector to grow and play a more visible role. SIBs allow responsibility to be devolved downwards while retaining central control through outcome-based funding. SIBs help square the circle for centralised governmental structures that wish to devolve responsibility while retaining policy direction.

Western countries such as the UK, where SIBs originated, would be wise to take stock of this evolving emphasis. Just as football played in Brazil and Spain has turned out to be more exciting than the game played by its inventers, SIBs, in their cultural adaptation, may prove to be a more transformative model than originally envisaged in Britain.

Attractions of Asia Pacific model

This adapted model could turn out to be an attractive export even to its inventors. Austerity – the ground into which SIBs were originally planted – will not last forever. SIBs may, in the longer run, offer opportunities for a more ambitious and hopeful agenda of improved well-being that may be politically realistic in more prosperous times. Such reinvention and rebranding of SIBs might also make them more attractive to their Western critics. The Left in Britain and US has tended to dismiss SIBs, suspicious of them as vehicles for commodisation and privatisation of public services. A revisioning of SIBs might suit a more left-wing agenda.

This SIBs model with a different emphasis might also offer an approach to reform in Britain where big providers such as Serco and Capita - focussed on driving down costs rather than on quality delivery to the person – are edging out smaller local players. Since 2010, many small civil society organisations have collapsed in the UK and others have had to merge as size has become the key to survival. Might SIBs eventually offer a way to address this trend and recognise financially the value of local deliverers with smaller caseloads but higher quality?

As the SIBs story unfolds, the purposes and future of this approach to service delivery remain in flux. Analysing the Japanese – and wider Asia Pacific - approach is a reminder that we may not have scratched the surface of SIB’s potential for achieving a wide range of agenda.

Dr Chih Hoong Sin is Director of Innovation at the Office of Public Management, London, UK

Ichiro Tsukamoto is Professor of Public Management at the School of Business Administration, Meiji University, in Tokyo, Japan.

If you wish to receive our weekly blog on SIBs, please email Paula.Fry@lshtm.ac.uk and we will add you to our subscriber list.

Evaluation should test key SIB promises to government: greater collaboration, prevention and innovation, argue two researchers from Oxford University’s Government Outcomes Lab.

To discover whether Social Impact Bonds work, we would be wise, first of all, to ask: “Work for whom?” Our response is: “For public service commissioners.” This answer acknowledges a reality: if SIBs don’t make sense to government, they are unlikely to be more than a passing fashion.

This standpoint cuts through a myriad of promises that typically surround SIBs and other payment by results programmes. It also helps focus how to evaluate them. We should measure SIBs by looking at what, if anything, they can deliver that public service commissioners seek.

That’s what we’re doing in the Government Outcomes Lab – GO Lab – at the Blavatnik School of Government, where our work is currently co-sponsored by the School and the UK government.

The dominant narrative around SIBs, born in an age of austerity, is that they can deliver cashable savings to government. Of course, every government seeks such dividends. However, this claim, far from proven - at least at scale - may or may not be true. Either way, we believe SIBs go well beyond promising the efficiency and entrepreneurialism claimed by conventional outsourcing. SIBs make a more fundamental offer that should be rigorously tested: that they provide an opportunity to remake the public sector itself.

Public sector reform has sat at the heart of every government’s agenda – Conservative and Labour – for nearly 40 years, since the election of Margaret Thatcher in 1979. Its goals – aside from making savings – have included three principles that SIBs directly claim to address.

SIBs address 3 principles of public sector reform

The first principle is to improve collaboration in the commissioning and provision of services so that the collective effort is better focused on the needs of the service user.

The second principle is to shift activity to a more preventive footing, so that service provision is moved upstream of social challenges and towards earlier identification of risk.

The third principle is to make innovation less risky for commissioners so they move to evidence-based procurement, decommissioning underperforming services and becoming better at reaching those, often experiencing the most complex mix of social issues, who may be poorest served by existing provision.

We should, therefore, question, in our evaluations, whether SIB models are improving collaboration, shifting services towards prevention and creating spaces in which public service commissioners and delivery organizations can innovate.

Principles largely untested by evaluations

Our review of evaluations suggests that they rarely explicitly or rigorously test any these three principles by comparing a SIB commissioning approach with a grant, fee-for-service or even in-house delivery for a given population. Evaluations also face the “doughnut challenge”. That’s the difficulty, particularly with impact evaluations, in teasing apart the effects of the commissioning architecture (the dough) from the effects of frontline services operating in different ways (the jam). Indeed, most evaluation material is quite descriptive and exploratory. It doesn’t typically offer or test a mechanism by which a particular SIB is expected to work in practice.

Collaboration is seen

However, where we have tried to read these three principles into the evaluations that have been produced, we see glimmers of each logic being present in SIBs. So Peterborough’s SIB – focused on reducing ex-offender recidivism – saw front-line organisations pull together in cohesive ways, wrapping around the service user.

We’ve also seen multiple commissioners joining forces as, for example, where the Department for Education, Cabinet Office and the Department for Work and Pensions came together for a fund-based approach to support young people vulnerable to disengaging from school and work. We see top-up funding approaches from the Cabinet Office to acknowledge that there might be wider public service benefits springing from a SIB that are not captured within the outcome payer’s domain.

“Step-down services”, more than prevention

As far as prevention is concerned, very early intervention programmes are rare in the SIB field, but there is a focus on “step-down” services. The Birmingham Foster Care SIB is a good example that is already demonstrating cashable savings.

A growing number of SIBs try to de-escalate situations of family conflict or address unstable conditions for children and young people. Typically, for example, the Department for Work and Pensions has worked with people only once they are employed. Now, through SIB programmes, the Department, through the Young Engagement Fund, is becoming involved in programmes that try to understand and anticipate future needs. It is working with school children to prepare them for a smooth transition into work.

Space for innovation

There are also examples of SIBs supporting innovation and helping to dissipate cultural inertia. Typically commissioning has a cycle – there’s the planning stage, the doing phase and then there’s the evaluation. A SIB can help reduce risk at any or all of these stages, allowing SIB commissioners flexibility to think differently about their work. So, for example, with an innovative programme such as the GLA Rough Sleeping project, a SIB allowed commissioners to ask of themselves: ‘Do I know how to be the best contract holder, or would I benefit from bringing in a third party that might have a better appreciation or knowledge base about that side of things?’

SIBs can also enable a certain amount of experimentation by providing quicker feedback through performance management. It’s this feedback that allows frontline organisations to change the way they are doing things to better meet the needs of their clients. A well-designed SIB also creates impetus for high quality impact evaluation within a payment mechanism. That inclusion can, in the long run, only help to improve the quality of future commissioning.

Time to test public sector reform potential

In conclusion, we cannot demonstrate, on the basis of existing evaluation evidence, that SIBs fulfil the many and varied promises that their champions make for them. But observing SIBs through the lens of public sector reform – and their possible role in it – we offer an approach to evaluating whether they might be a tool for remaking the public sector.

Eleanor Carter and Clare FitzGerald are research fellows at the Government Outcomes Lab (GO Lab)at Oxford University’s Blavatnik School of Government.

If you wish to receive our weekly blog on SIBs, please email Paula.Fry@lshtm.ac.uk and we will add you to our subscriber list.

A major investor highlights the vital role that research and evaluation should play in developing this form of outcomes funding.

Evaluation and research into Social Impact Bonds (SIBs) is a hot topic for Big Issue Invest. We are one of the UK’s leading social impact investment firms, having invested in approximately 350 charities and social enterprises since 2005, in our quest to dismantle poverty and create opportunity.

Our investments support areas such as access to housing, financial and social inclusion, mental and physical health and well-being, and employment, education and training.

We have made several SIB investments since 2012 and have watched the market develop. Our first-hand experience is that there is more work to be done to refine the model, but there have been impressive outcomes for programme participants and the charities delivering the contracts in which we have invested.

How we use research and evaluation in SIBs

We aim to address the structural challenges of SIBs and maximise their individual and collective social impact. Research and evaluation can help in this goal. Many commissioners and service delivery providers are unfamiliar with SIBs and I often direct them to impartial, well-informed research to build awareness and understanding.

Discussing Big Issue Invest’s learnings and experiences from individual programmes with evaluators helps us to contextualise the situation and identify emerging trends in this rapidly-developing field. These partners can develop the tools and models to test and critique our theories and insights ‘from the field’ and evaluate the wider market, whereas our frame of reference is often limited to our specific investments.

Research in social investment and SIBs is at a nascent stage. It is crucial that the right foundations are laid today to enable good-quality ‘market’ level analysis in the future. Consistency and rigour of approach will pave the way for systematic reviews and meta-analyses, essential if outcomes-based approaches are to become commonplace in the commissioner and policymaker toolkit.

We also welcome more quantitative approaches and ambitious evaluations that compare SIBs to traditional fee-for-service mechanisms or payment by results more broadly. SIBs are often conflated with outcomes-based approaches in general, which makes it very hard for investors to assess – and improve – their social impact.

Funding is, of course, needed to allow this work to take place. At the moment, evaluations are too often the balancing item in a very limited budget, constraining their ambition. We wholeheartedly support the calls for a ring-fenced fund for SIB evaluations, recognising the value of the output to commissioners, central government and potentially also philanthropic funders who might seed the fund.

Understanding what, why, how

We are an impact investor: outcomes are our reason for being, not just a by-product of our investment activity. Social impact due diligence underpins every investment decision we make.

We look closely at the theory of change for each SIB. There needs to be a coherent and credible hypothesis for how outcomes will be improved for the programme participants and - beyond that - how the programme could help to tackle the underlying issue through, for example, earlier intervention. We interrogate potential perverse incentives in order to mitigate them. Research and evaluation from previous programmes helps us to do this: we rely on it to validate the causal link between inputs, outputs and outcomes and complete our due diligence of the intervention.

It’s important that everyone involved reviews the theory of change periodically after the contract launches. One of the strengths of SIBs is that they shine a light on what works and what doesn’t, enabling real-time improvements and sharing of learnings for future contracts. If we scrimp on monitoring and evaluation, we undermine the programme and indeed the SIB model.

Evaluations therefore need to be robust, relevant and timely. We want to understand not only the results achieved but also why they are (or are not) achieved and how we can replicate and improve on them. That might be a programme evaluation or an impact evaluation, or qualitative or quantitative approach, depending on the context but we certainly need more than outcomes verification.

We seek insights into the drivers of success so they can be reflected in future projects. The GLA Rough Sleeping SIB in 2012 was divided into two lots awarded to separate providers, one funded by Big Issue Invest. We know the absolute outcomes achieved by the programme but would like to dig deeper into whether different operational or investment approaches had a bearing on success.

We are keen to work with the evaluator community to design the evaluations and contribute to them. It’s important to be confident that Big Issue Invest’s loan has achieved its social objectives – and those of our investors in turn – so we are a consumer of evaluations as well as contributors to them.

We are one of very few organisations that has worked across several SIBs in different regions and policy areas. We can contribute data, insight and practical experience and welcome the opportunity to do so. At a practical level, we can coordinate with the evaluator to minimise the data collection burden on the service provider’s staff and the programme participants. If we can bring the evaluator in to the design phase early, we can also incorporate their evaluation into the delivery model early to avoid duplication or complication down the line.

Using data and analysis to target interventions

Reliable data and analysis is essential to high-quality SIB design. For example, we are involved in an ‘edge of care’ SIB for young children where it is unfeasible to roll out an intensive (in other words, expensive) intervention to all children on social services’ radar. Rather than only work with children on the very cusp of care – when it is often too late to reverse their trajectory or the trauma they have suffered - a researcher is working with commissioner data to identify early risk factors that increase the child’s propensity to enter care. The programme will be targeted towards these high risk children as well as those on the very cusp of care. This allows the commissioner to fund an early intervention service that is also cost-effective, often challenging in outcomes-based commissioning. There is huge potential to harness data and analysis in this way to design preventative services.

The value of timely feedback

Speed of evaluations is a challenge. Evaluations are valuable to SIB stakeholders when developing follow-up programmes and carrying out due diligence. If investors have a good level of confidence in the achievable outcomes, the cost of their risk capital should be lower. That is in everyone’s interest. It doesn’t encourage evidence-based commissioning if the evidence is released after the next programme is launched!

Midline and end-line reviews as part of a formal evaluation are, of course, extremely important but they are not enough. Outcomes-based programmes also need shorter, informal feedback loops, preferably involving the evaluator. Early results and findings can be used to improve programme delivery - but not if they are shared in an end-of-programme evaluation that takes a year to publish. Ideally, we’d like a quarterly or a six monthly check-in with the evaluators that can identify and unpick performance and its drivers.

We recognise the tension between this approach and concerns that the evaluation will influence the programme outcomes. A balance needs to be struck. SIBs support people with complex needs who deserve the best possible chance of better life outcomes, so although evaluation rigour is crucial, we owe it to them to make the intervention as effective as it can be.

Importance of counterfactuals

Three inputs are usually needed to assign a value to a SIB outcome: (1) the projected costs to deliver a programme (preferably validated through a competitive procurement process); (2) the costs per outcome achieved under comparable programmes, if known; and (3) the savings case (the projected benefits for the commissioner). If you don’t have an understanding of what would have happened anyway, at least one of these calculations will be flawed. That’s why we can’t afford to disregard the counterfactual.

That doesn’t mean that every SIB needs to link payments to performance compared to a counterfactual, measured by an RCT. There are lots of factors to consider when designing the payment mechanism and there is no single ‘right’ approach. However the counterfactual can always be taken into account. Under a rate card approach, the rates should be set after considering deadweight – even if the assessment is imperfect, it is better than ignoring it completely. The counterfactual can then be assessed in the programme evaluation and used to inform the pricing of future contracts.

I am not saying SIBs should be commissioned only if there is perfect data to value the counterfactual. Rather, I am emphasising the need for new approaches that measure outcomes and cost-per-outcome to allow commissioners to make evidence-based decisions in future. Big Issue Invest is trialling approaches that allow an outcomes-based contract to be launched with imperfect information, while ensuring checks and balances limit windfall gains and losses and include mechanisms to tackle the information gap.

One option is to run an initial ‘discovery phase’ of the contract for one to two years. The discovery phase outcomes pricing estimates the counterfactual, but sets parameters to ensure that no party makes excess gains or losses. In this way, the partners have the opportunity to implement the SIB delivery model. During this time, outcomes and the counterfactual are measured rigorously. The data and analysis is then incorporated into a revised payment mechanism for the rest of the contract, after which point it operates like a ‘standard’ SIB. This approach bridges the knowledge gap without delaying a potentially high-impact programme or risking inequitable risk and return.
Where next?

SIBs bring together different worlds. The success of SIBs is dependent on partnerships where the whole is greater than the sum of the parts. They require new ways of working for everyone involved – for investors, providers and commissioners. I expect they can seem strange to evaluators as well. Forging new links and understanding the perspectives of others is crucial.

We are starting to see these worlds come together and collaborate for better outcomes. There is more interaction and understanding between researchers and evaluators, policymakers and budget holders, delivery organisations, and investors. It is early days but the outlook is promising.

Katy Pillai is Investment Director, Big Issue Invest: www.bigissueinvest.com @katyjones | @bigissueinvest

If you wish to receive our weekly blog on SIBs, please email Paula.Fry@lshtm.ac.uk and we will add you to our subscriber list.