Performance indicators and public reporting in health care. A review of the scientific literature and an ethical analysis

Social pressure to get more information about the quality of healthcare is increasing. This has also become a major priority for Government policy in recent years. The idea is for ’performance indicators’ to be used to demonstrate the quality of care and allow comparisons to be drawn, so that all parties concerned can immediately see where care is up to scratch and where it is deficient. This process has two aims: transparency, and improving quality. Society must be able to see how public funds invested in healthcare are spent. This should offer choice to patients and insurance firms, and encourage healthcare providers and professionals to improve the quality of the care they provide.

It is clear that a fresh stimulus for improving the quality of healthcare was needed. This is not to say that nothing was done in this area in the past. The original design of the Healthcare Facilities Quality Act expected healthcare providers and professionals to put their quality systems in order themselves. In practice, this did not seem to work properly. The new approach (highlighting differences in quality and introducing external stimuli) should change this situation. It is expected that healthcare providers and professionals whose performance is poor will want to avoid damage to their reputation or a loss of market share. They will hope to achieve better results in a subsequent assessment, even if they may lack the internal motivation to improve quality. This effect could be enhanced even further by the use of (financial) rewards and penalties for high and low scores.

The question of whether this commercial approach, which has been introduced into other areas of public service over the past few years, is an effective means of achieving both aims (transparency and quality improvement) is a matter for often strongly-polarised discussion. Those in favour of this approach seem to dismiss the possibility of unwanted effects, while its opponents are already convinced that no good can come of it. An objective debate is vital if policy in this area is to be developed rationally. This report therefore hopes to make a contribution in the form of scientific data and an ethical analysis.

Performance indicators

Performance indicators are measurable elements of healthcare provision that act as a possible pointer to the quality of care. They act as signals: low scores signal that something might be going wrong, and that further analysis is needed to find out whether this is indeed the case. Indicators have to be reliable and valid if they are to fulfil this signal function adequately. By ’valid’ we mean that they must actually measure what they are supposed to measure.

Findings of literature review

First, the facts. What can scientific research tell us about the effects and possible side-effects of the public use of performance indicators in healthcare? We have performed a systematic literature review to answer this question in our report. This review concentrates purely on clinical healthcare. We looked for publications addressing the effects of revealing performance scores on the behaviour of institutions and healthcare providers (whether or not quality improved, other behavioural effects); the effects of forms of (financial) rewards and penalties attached to these scores; the effects on the choice patterns of patients, insurance companies and referring doctors; effects on costs and bureaucracy.

The findings of the literature review show that the assumptions underlying the public use of performance indicators, and the expectations as to the effects of this approach, are not at present based on sound scientific evidence. With regard to encouraging better quality, we did find that this approach had some effect on the performance levels of institutions. Quality improvement initiatives were introduced after publication of performance scores that could reflect badly on the institution’s image. But we found no evidence of this effect in terms of individual healthcare providers. When we found public reporting to have an effect on behaviour in this group, it seems that the effect does not take the form of quality improvement but of strategic behaviour, such as refusing high-risk patients or manipulating data. It does appear that the introduction of reward systems can have a beneficial effect on the behaviour of healthcare providers, but more research is needed to find out how this instrument can best be designed.

We have also found little evidence to support the expectation that patients, insurance companies and referring doctors would be guided by a comparison of performance scores when making choices. Patients do seem to be interested in the results of performance measurements, but they make only limited use of them when choosing among various healthcare options. They seem to give greater weight to the experience of people they know. Little research has been done into the choice behaviour of other parties that make choices. Insurance companies and referring doctors appear to have little faith in public performance figures, and so make little or no use of them.

We have little information as to the costs associated with public reporting and the implications of this approach for the administrative burden (’bureaucracy’). However, it is likely that there would be an increase in this burden in the early stages. It may well be that more efficient methods of data collection would reverse this trend eventually, but that is by no means certain at present.

Ethical aspects

The second part of the report looks into relevant ethical aspects of the public use of performance indicators. We start by clearing up the misconception that the requirement for public accountability is in conflict with the idea of professional autonomy. ’Autonomy’ is not a green light to professionals allowing them to disclaim all responsibility for their actions. Professional autonomy and professional accountability are in fact two sides of the same coin. The room for manoeuvre in terms of autonomy is that which allows healthcare providers to offer their patients the best possible care. Society is entitled to demand that this is reflected in healthcare outcomes.

The principle underlying the new Government policy is that self-regulation in this area is too lax. The requirement to publicly report performance scores that can be compared with one another should bring into play a mechanism in which external stimuli (reputation, market share, rewards) also encourage less strongly-motivated institutions and professionals to make a serious effort to improve their quality. From an ethical point of view it is important that professionals are addressed differently. The approach should focus not primarily on their intrinsic motivation but principally on their sensitivity in terms of their image and economic benefits. The use of such stimuli can certainly be sensible if it has a positive stimulating effect. But our literature review also found indications of a negative effect on the motivation and behaviour of professionals. The moral implications of the relationship between the positive and negative consequences of the use of external stimuli are far-reaching. Healthcare professionals are not required to be saints; however, the core of their professional identity is determined by their focus on the well-being of other people. Undermining this would cause damage to society that would be difficult to repair.

The new role that patients would play in the new Government policy, i.e. ’healthcare consumers exercising choice’, is also associated with a moral risk. It raises a contemporary moral ideal, that of self-determination. Another important aspect is that a patient’s choices can help encourage healthcare institutions and providers to compete on the basis of quality. Policy documents refer to this as ’horizontal supervision’. However, the literature we have reviewed for this report indicates that patients do not as yet behave in this way. They are not guided by performance indicators when making choices. Further investigation is required to ascertain what choice options in healthcare patients really need. But even more important is the question of what this consumer role means for the relationship between healthcare providers and recipients, a relationship which is based on trust. Does this role still leave room for recognising that each party needs the other: healthcare providers with their specialist skills and expertise, patients with their specific needs and priorities, so that they can jointly decide what treatment or care is the most appropriate response to the patient’s symptoms? And in broader terms, how will public confidence in healthcare be affected if people are continually reading reports that emphasise failing healthcare providers?

Furthermore, the political principle that ’quality must be measurable’ can easily lead to a situation in which the relative importance of different performance indicators is obscured. Injustice and imbalance could result. ’Injustice’ where healthcare providers are addressed and judged by outcomes which fail to take sufficient account of contextual factors over which they have no influence, for example the exact make-up of the patient population. ’Imbalance’ can result from the choice of indicators that are perhaps insufficiently representative. All the energy and attention might then be brought to bear on improving a measurable aspect of care, without at the same time improving the quality of what underlies it. Another question is what the emphasis on measurability means for other parts of healthcare, such as nursing and personal care, or other aspects such as the way patients are treated by healthcare professionals and how they perceive their healthcare experience, for which it may be harder to develop good performance indicators.

Finally, another important moral question is whether the costs and bureaucracy associated with the public use of performance scores is proportional to the rewards in terms of better quality and accountability. The results of the literature review indicate that this is still unclear. The social experiment which this undoubtedly is requires a cautious, gradual approach combined with ongoing assessment and investigation of the effects.

What next?

The debate on the public use of performance indicators has so far been conducted from firmly entrenched positions, and is often couched more in absolute terms, i.e. that this approach is either entirely rational or complete nonsense, than in examining specific opportunities and risks. This examination is vital if policies in this area are to be developed sensibly. All participants in the debate need to have an open mind, and the discussion must be held against a background of scientific data and ethical analysis. We hope that this report will contribute to the future debate.

It is vital for support to be generated and distrust overcome. This can be achieved by working on two fronts. The Government can help by turning away from imposing external stimuli as the prime, or only, method of improving quality and appealing instead to the intrinsic motivation of professionals, enabling them to express this motivation in their work. It must be made quite clear that performance indicators will be used in such a way that no-one needs to fear being exposed without justification, or suffering professional damage without good reason. Another important condition is avoiding unnecessary bureaucracy.

As far as healthcare providers are concerned, the main challenge is to experience the demand for systematic quality improvement in a different way. Not as something imposed from outside, as a chore to be undertaken to avoid penalties or image problems, but as a core element of professional responsibility. Quality improvement should be an intrinsic part of the healthcare sector: a vital basic skill which professionals at all levels must develop. This aspect must be addressed in training, which is not sufficiently the case at present.

The approach we are arguing for would mean professionals in the medical sector, particularly scientific associations, taking ownership of the initiative. This would take the form not only of developing performance indicators for use in the context of internal quality systems but also in assessing which indicators are suitable for external use.

Further research is needed into the effects and side-effects of public use of performance indicators in the healthcare sector, the effects of reward systems, and patients’ need for performance information when making healthcare choices. The representativeness of performance indicators is a key issue as they come to be developed. It is also important to ensure that aspects for which it is more difficult to develop good performance indicators, such as personal attention, the way patients are treated by healthcare professionals and other relationship factors, are not neglected.