Success and failure in the simulation of an Accident and Emergency department

Abstract Healthcare simulation has the potential to offer many benefits but the implementation is often problematic. This paper describes the development of a simulation of an Accident and Emergency (A&E) Department in an NHS hospital. The early experience of the client provoked great enthusiasm but ultimately the simulation failed to meet all expectations. The simulation delivered a number of benefits, notably in terms of stimulating constructive debate and helping the stakeholders appreciate the complete A&E system. The project produced a technically proficient tool that was delivered too late to have the desired impact. This mixed record of success appears typical of many simulations. Important lessons were learned, both technically and in the management of client expectations, which have contributed to subsequent successful implementation in other departments of the hospital. The experience suggests that both potential clients and analysts need to establish realistic expectations and appreciate the particular challenges of simulation in a healthcare environment.


Introduction
The technique of simulation has been extensively employed in analysing and designing healthcare systems (Jun et al, 1999;Fone et al, 2003). In particular it has been used in various studies of A&E departments (Gunal and Pidd, 2006;Fletcher et al, 2007). This paper examines the experience of using simulation in redesigning activities in an A&E department. The study was part of a larger, 3-year Knowledge Transfer Partnership (KTP) between the University of Stirling and NHS Fife Health Board. The aim of the KTP project was to explore a variety of modelling methodologies in support of the redesign of healthcare services. The development of the simulation is described and the paper offers reflections on the success of the study, highlighting the failure to meet all expectations. This mismatch between potential and reality in healthcare simulation appears has been identified in other studies. Some have suggested that the problems lie in the diversity of clinical and organisational practice, and the particular characteristics of the relationship between the clinical and management staff (Kuljis et al, 2007). Given the critical human element in healthcare management, successful simulation may require special care in managing the relationship between analysts and client staff. The iterative nature of the modelling and the involvement of all interested staff has been identified as particularly important in healthcare simulation (Baldwin et al, 2004). In many health studies, publication occurs before the work is fully implemented and there is little opportunity to assess the impact of the simulation (Fone et al, 2003). However, this was not the case in this project, which was undertaken at the start of a 3-year programme. The longer term relationship with the client provided a good opportunity to gain honest feedback and to redress some of the problems encountered in the implementation of this study in subsequent simulations.

Redesign of A&E in the NHS
The NHS in the United Kingdom launched a widespread programme of redesign as outlined in the NHS Plan (Department of Health, 2000). The redesign programme has deployed various tools and techniques that have been found to be effective in industry, with the intention of transferring good practice into the NHS. However, some refinement in their implementation is often necessary to retain a focus on the needs of the patient (Bell et al, 2006). The redesign programme has been driven by an ambitious series of targets for health trusts and boards across the United Kingdom. One such target is that patients should not wait for more than 4 h in an A&E department.
In July 2005, NHS Scotland formed the Unscheduled Care Collaborative Programme (UCCP) (Scottish Executive, 2005) involving all 14 Health Boards in Scotland. This provided the forum for a nationwide redesign initiative to deliver improved services and faster treatment for patients; in particular the 4 h target for treatment in A&E had to be met by December 2007. This required patients to spend no longer than 4 h between arrival in the A&E department and either being discharged or admitted to a ward. The UCCP recommended that five Flow Groups be formed to manage the redesign of unscheduled care, supported by a series of national events which aimed to disseminate good practice. The objective was to make systematic and sustainable changes and deliver patient focussed improvements across the five patient flows: Group 1-Minor Injury and Illness Group 2-Acute Assessment (covering A&E and Acute Receiving Units) Group 3-Medical Admissions Group 4-Surgical Admissions Group 5-Out of hospital care In NHS Fife, the first task for the KTP project team was to assist in the redesign of Unscheduled Care. Although the project had some involvement with all the Flow Groups, initially most attention was focused on Flow Group 2, which had the responsibility for redesigning A&E services to meet the 4 h target. A group of staff was established for Flow Group 2 comprising of managers and clinicians from A&E and associated services, including radiology, diagnostic laboratories and patient transport. One member of the group was an analyst providing a variety of skills including simulation. The group was chaired by a consultant in A&E. The use of such multidisciplinary teams had previously been found to be an important element in transforming patient processes (Newell et al, 2003) and the genuine involvement of a range of front line staff was recognised as critical (Locock, 2003). The group met regularly to review progress against the 4-hour target and discuss proposals for improving the flow of patients. The Flow Group provided a real opportunity for staff to make proposals, plan implementation of new actions and review the effectiveness of changes. This created a genuine sense of ownership of the changes, and an excellent environment for exploring the use of simulation as an embedded support tool in healthcare redesign.
The initial work of Flow Group 2 in NHS Fife concentrated on three areas of study: Patient flow through A&E. It was necessary to understand all the stages in the patient journey, to identify the bottlenecks in the system, how the treatment varied for different categories of patient, and identify the resources that were required at each stage in the patient journey. Matching capacity with demand. An information system had been established which recorded the admission and discharge times of the patients in A&E. These demand data were compared with the main capacity determinant, staff shift patterns. Identification of the major causes of breaches (patients who spend longer than 4 h in A&E). The common problems identified were waiting for assessment by a doctor, waiting for a bed or waiting for transport. This analysis highlighted the dependency of the A&E department on the rest of the hospital's activities. These problems were investigated in detail and discussed with representatives from the relevant areas of the hospital.

Initial analyses
A major objective of the larger redesign programme was the development and implementation of process mapping tools. Initially the staff produced hand drawn process maps and these were transferred to Microsoft Visio. The Visio maps included picture icons similar to those used in the simulation. It was found that the icons enhanced clarity and the Visio maps could be saved in html format exploiting the hyperlink facility. This enabled several levels of mapping detail, from a high level map showing all potential routes through the service, to very detailed maps of specific facilities, for example the x-ray process as shown in Figure 1. A further advantage of such Visio maps was the ability to share the understanding on the intranet, providing an additional route for feedback and dissemination. The maps were developed and used with considerable success, helping the staff appreciate the whole system of A&E and facilitating discussions about improvements.
A simple analysis of historic capacity and demand produced some noticeable discrepancies between staffing levels and A&E activity. Rapid action was taken to adjust shift patterns and provide additional staff to redress the balance. Further analysis of the breach data confirmed that the vast majority of breaches of the 4 h target were with major patients, that is, those who would be admitted to hospital. The data confirmed staff perceptions that many of the breaches occurred in the afternoon but analysis of hospital admission and discharge times showed that most beds became available in the late afternoon and early evening whereas the peak demand for beds for A&E patients was in early to mid afternoon. Addressing these problems which involved interactions with services in the rest of the hospital proved more difficult. One solution proposed by a medical consultant was the establishment of a medical assessment unit. This would be a ward with focussed diagnostic services for acute medical patients who may require admission to a more specialist ward. The unit would allow these patients to be located together facilitating regular assessment by physicians, who could decide who needed to be admitted to a specialist ward and who could safely be discharged. This arrangement of services had proved successful elsewhere (Fox et al, 2003). However, it was not known how many beds were required for the unit.
The Flow Group asked that a prototype a simulation model should be constructed to help answer this question. The simulation modelling excited much interest, with many staff, including clinicians, becoming enthusiastic about the possible role of simulation. During the regular meetings with A&E staff, the debate inevitably moved to other questions about the magnitudes of flows and the capacities of the different components of the A&E system. Staff wanted to develop a better understanding of the uncertainties and the appropriate contingencies necessary to deliver the 4 h target. Such questions are well answered by simulation and further demonstration A&E simulation models were constructed. It was soon decided that the purpose of the model was to model was to understand the flows and capacities within A&E as well as the medical assessment unit.

Model development
The A&E department receives patients from a variety of sources. The patients are assessed and emergency treatment may be provided, with the majority being discharged with the possibility of follow-up care. A proportion of patients will require inpatient care and the A&E department must interact with other emergency receiving and treatment services in the hospital. For example, a smooth flow of patients through A&E requires a reasonable supply of beds on emergency medical and surgical wards. This in turn requires that these wards have effective discharge practices, avoiding unnecessary delays. Ideally, discharged patients should vacate their beds in the morning to ensure sufficient capacity to meet the peak in admissions from A&E, which usually occurs in the afternoon. A&E also requires the cooperation of diagnostic services so that tests and diagnostic radiology can be provided on demand, 24 h per day. One of the modelling challenges was to capture the critical details of these interactions without attempting to model the whole hospital. For example, the activities on the wards were modelled at a high level, accepting the historic patterns of discharge excluding detailed modelling of inpatients' transport requirements.
A good understanding of patient flows through the department had been obtained from the pathway mapping exercise and a simplified version of this map formed the basis of the simulation model. In addition, a limited patient tracking exercise was undertaken to collect specific detailed data, for example details of the time required for the return of the results of diagnostic tests. This, combined with the routine A&E information system data, provided the key information required for the simulation. These sources of data provided sufficient data to model the arrival of patients into A&E, the flows and timings of various activities in the A&E department and the exit, either as a discharge or an admission as an inpatient. Figure 2 illustrates the high level flows of patients through the A&E system. A series of individual meetings were held with clinicians and A&E staff to determine the logic of the system in detail. As development continued prototype models were presented to Flow Group meetings, providing valuable feedback.
The staff were most enthusiastic. Simulation had only been included in the KTP project proposal as an exploratory option and it was not anticipated to be a significant contributor to the redesign study. However, it was agreed to amend the programme of work and devote more effort to constructing a detailed simulation of A&E. As noted in other healthcare simulation studies (van der Meer et al, 2005), the approach and objectives were well suited to the needs of the NHS in redesign, with a strong emphasis on understanding the whole system, clinical involvement and sustainability of the approach. The enthusiasm and co-operation of the clients persuaded the modelling team that simulation was a realistic method and could deliver a practical tool to support the Flow Group's activities. This enthusiasm was illustrated by the suggestion from some client staff that the simulation might provide real time forecasts, warning of possible congestion and breaches of the 4 h target. The modelling team explained that such forecasts would require much more reliable data about individual patients' likely requirements. It was agreed that real time forecasting was beyond the scope of the current model but it was accepted that the model would be designed such that it could be used by staff with minimal training in simulation. This would provide a degree of sustainability with the model being used routinely in future reviews of A&E organisation in NHS Fife without having to depend on the availability of specialist simulation skills.

Interface design
The model was developed with the intention of it being used by two very different groups of users. Firstly, it was designed to be used by analysts with simulation expertise who might explore a wide range of options, and would have the ability to make changes to the detailed logic of the simulation. The model was also intended for more routine use by staff without simulation expertise; the model was designed to support decisions arising from adjustments in the A&E environment such as the staff allocation required to meet future targets given changing patterns of demand. Satisfying this second group of users required a sound appreciation of decision-makers and their needs. This understanding evolved through the close relationship that developed through the Flow Group meetings. The user interface aimed to highlight the key input parameters that were within the control of local staff and to provide clear outputs so that they might appreciate the consequences of proposed actions. A significant additional advantage of the graphical simulation was the animation of the model, with users able to view the arrivals of different types of patients and the changing queues throughout A&E. Examining these movements at slow speed enabled staff to provide feedback on the model's logic, contributing to the open box validation and helping to increase the model's acceptance.

Outputs and diagnosing the causes of the 4 h breaches
The focus of the activity of the Flow Group was the reduction in the number of patients spending more than 4 h in the A&E department. Weekly reports of the breaches of the 4 h target were circulated and staff were most concerned to identify the possible causes and resolve the problems. The simulation output was designed to reflect this key objective. In addition to statistics such as the proportions of minor and major patients being treated within the target time, outputs were provided to help staff diagnose the sources of delay. The objective was that staff could experiment with a proposal, using the simulation to gain a deeper insight and refining the proposal until a solution was identified. One such output is illustrated in Figure 3. It depicts the mean proportion of patients meeting the 4 h target as a function of the time of arrival at A&E, based on repeated trials of a simulation of A&E activity for January-March. In the particular example of Figure 3, the main difficulties are associated with patients arriving in the afternoons, with the proportion being treated within 4 h falling to just 50% on Mondays and 60-70% on other weekday afternoons. Various reasons were suggested for such patterns and experiments performed to test the hypotheses and explore possible mitigating options. This output was also used in validation, confirming that the particular problem times suggested by the simulation corresponded to those indicated by the records and staff experience. The output also includes estimates of the proportions being treated within 3-2 h to help staff appreciate the challenges of meeting tougher targets in the future. One particular management option was the reallocation of staff over the week such that there was a better match with demand. Outputs, such as Figure 4, described the utilisation of key resources over the week and help identify the causes of the pattern of delays. In this example the mean utilisation of doctors is typically 90-100% during the weekday afternoons suggesting an overload which could well lead to congestion in the system. The nurse utilisation is also high but there does not appear to be such a strong correlation with the instances of breaches of the 4 h target of Figure 3. Considering these outputs the users were prompted into experimenting with alternative shift patterns, using the simulation to test whether the major source of the problem has been identified and resolved by the proposed action. A sophisticated optimising model could be developed, such as goal programming explicitly weighting the various objectives that managers have to consider (Oddoye et al, 2007). Such an approach would involve estimating weightings for staff time and patients' time in A&E, reflecting the non-linear nature of the 4 h target. Eliciting these weights can be a constructive exercise in its own right. However, a simpler approach was adopted in this study with the emphasis on supporting the decision-makers by providing appropriate outputs. This approach was intended to help the staff develop a deeper appreciation of their own A&E system and feel a greater ownership of the proposed solution, consistent with the underlying concept of the Flow Groups. A fuller study might compare the shift patterns derived from a goal programming approach with the solutions obtained by staff judgement, supported by the simulation. Ideally, the two approaches might be combined, with the goal programming offering a suggested shift pattern and the staff then using the simulation to experiment with alternatives that reflect some of the tacit constraints that can be difficult to quantify.

Inputs and decision-makers
Input screens were designed to enable staff to adjust key inputs describing demand, resource allocation and organisational practice. Although numerous inputs could be refined to tailor the model to suit individual A&E departments, the key inputs were highlighted as those which provided   sufficient control for most users. For example, although the detailed hourly and seasonal arrival pattern at A&E could be revised, the common requirement was for control of the annual volume and the division between minor and major patients. The resource availabilities could also be defined in great detail but the practical management options were more limited and these were reflected in the parameters highlighted for the decision-makers, as illustrated in Figure 5. In the analysis illustrated by Figures 3 and 4, this facility allows alternative shift patterns to be specified, ensuring a better match to demand and hence reducing the afternoon 4 h breaches.
The organisation of A&E departments varies throughout the NHS, reflecting the substantial degree of clinical independence. The simulation attempted to capture some of this variation by allowing users to specify selected organisational options. One major concern was the interaction of the A&E department and the whole hospital system. A particular issue was the transfer of patients requiring a bed on a ward: patients need to be 'clerked in', with information being collected and a doctor confirming the transfer on to the ward. The issue of the location of this 'clerking in' can be critical. The simulation experiments confirmed staff experience that 'clerking in' in A&E could impose a significant delay as the patient has to wait for the appropriate doctor to visit the A&E department. Similar variation in practice in the use of discharge lounges, and hence releasing beds for early use by A&E patients requiring admission, was also identified as critical and requiring a specific input.

Simplification and distinguishing the critical details
A simple simulation is highly desirable but often it is only possible to distinguish and eliminate the unnecessary complexities retrospectively. It was only through developing a sophisticated A&E simulation model that sufficient understanding was gained to enable the analysts to distinguish between the irrelevant feature and the critical detail. This process was reflected in the selection of the highlighted input parameters for the routine users. The development of this A&E simulation followed a sequence of: a simple prototype simulation; an elaboration of the model incorporating a wide range of details; simplification in which critical features were emphasised and the extraneous details removed, or at least hidden. Other studies have also observed that the common recommendation to keep the simulation simple is not always easy in practice and the reality involves a degree of iteration (Chick, 2006).

Validation
The black box validation (Pidd, 2003) focussed on the key outputs, notably the proportion of patients treated within the 4 h target. This was supplemented with open box validation, with members of the Flow Group judging the Figure 5 Screen display illustrating input windows. reality of the model's logic and the weekly/daily patterns of activity and the consequent bottlenecks. Given the close working relationship, several iterations were undertaken, refining the model to achieve a good match with staff's perceptions, and hopefully reality. Other studies (Fletcher et al, 2007) have also found that a combination of black and open box validation is appropriate in healthcare simulation.

The successes and failures
Technically the development of the A&E simulation was a success: the final product had all of the required features specified by managers and clinicians. The simulation was eventually delivered as a user-friendly tool incorporating clear interfaces with parameters enabling users to specify the major features characterising their A&E department.
The model was validated and demonstrated to staff who agreed that it provided an accurate representation of their A&E system. During the development, the value of the model was demonstrated in assessing options for improving the flow of patients in A&E.
The vision had been of clinical and management staff regularly using the model to explore options as their environment, notably demand, changed. However, the ambition of delivering an operational tool with users assessing options without recourse to simulation expertise, was not realised. The main reason for this failure was that the development took longer than anticipated, partly due to the numerous iterations required as the understanding of both the modelling team and the client's staff developed. The simulation development included the many participants but this resulted in additional iterations. The complete model, with a user interface enabling decision-maker input, was delivered too late in the Flow Group's programme of change: the A&E system had been thoroughly investigated, change implemented and the four hour target had been met by the time that the simulation was completed.
The major success of the simulation model was its contribution to the understanding of the Flow Group of the problems and the possible actions to enhance A&E performance. Particular examples were its use in matching of staff to demand and the investigation of the interactions between A&E and the rest of the hospital. The development of the simulation prompted focussed data collection and preliminary analyses that resulted in effective change, well before the final, validated model was delivered. The A&E simulation experience also contributed to success in other studies in NHS Fife: a simulation of orthopaedics' outpatients was constructed rapidly and implemented as an operational tool. This was a further bespoke simulation with only a small degree of component reuse from the A&E model (Robinson et al, 2004). However, much of the design was similar with lessons from the A&E development being transferred to the orthopaedics simulation. The A&E experience enabled a rapid and timely development of the orthopaedics simulation resulting in it having a greater impact on the redesign of the outpatients' processes.

Conclusions
The development of the simulation model for A&E in NHS Fife was a mixed experience. The close working relationship between the simulation modelling team and the NHS staff assisted in developing a shared understanding of the current A&E systems and options for improvement. However, most of these benefits were realised during the development phase which initiated focussed data collection and a series of valuable preliminary analyses. By the time the model was properly validated and ready to provide rigours assessment of proposals, the debate had moved on. The early successes contributed to unrealistic expectations and an eventual sense of anti-climax. Simulation can provoke great enthusiasm among clients but care is needed to manage expectations.