No gains from use of safe surgery checklist in Ont., study suggests

TORONTO – A new study suggests Ontario hospitals saw no measurable improvements in their surgical safety records six months after they implemented a World Health Organization-designed safe surgery checklist program.

The lead author of the study said the findings may suggest earlier claims about the patient outcome improvements hospitals could achieve through use of the surgical checklist were overstated.

But other experts, including the leader of the WHO effort, said the new study shows Ontario and its hospitals fumbled the introduction of the surgical safety checklist program.

“The lesson is that it hasn’t been implemented properly, not that the checklist is garbage,” said Dr. Jason Leitch, clinical director for Scotland’s National Health Service, which has achieved a 23 per cent reduction in surgery-related deaths over the past six years using a set of interventions including surgical checklists.

The new study was published Wednesday in the New England Journal of Medicine, which also published the 2009 study that reported large improvements in surgical safety associated with use of a standardized checklist the WHO issued in 2008.

The checklist requires operating teams to introduce themselves and outline their roles before surgery begins, confirm the procedure that is to be done, ensure that all necessary equipment is in place and discuss any known concerns. Before the surgery ends, staff confirm all instruments and sponges are accounted for and the team discusses how the patient’s ongoing care needs to be managed.

The 2009 study reported a 47 per cent reduction in deaths associated with high-risk surgeries when the checklist was used. The rate of surgical site infections was almost cut in half and there were also significant declines in all complications and unplanned returns to the operating room – code for surgeries to fix problems created by the first surgery.

The buzz generated by that study prompted many jurisdictions to move to adopt the checklist approach. In Ontario, the Ministry of Health told hospitals they would need to begin reporting publicly whether they were using a surgical checklist beginning in July 2010. Some hospitals were already using surgical checklists; others quickly got onboard.

So several researchers from University of Toronto hospitals decided to see whether surgical checklists were delivering results.

Using hospital records compiled by the Institute for Clinical Evaluative Sciences, they looked at surgery-related deaths and complications – things like surgical site infections and unplanned secondary surgeries – for 92 hospitals around the province.

They compared how many of these events happened in the individual hospitals for a three-month period before the hospital introduced its checklist program to the number of events that happened in a three-month period after the checklist was in place.

The before and after periods studied were not the three months immediately leading to or following the adoption date, but rather the quarter before and after that. So if a hospital started using the checklist on July 1 of a given year, their before period was from Jan. 1 to March 31 and their after period from Oct. 1 through Dec. 31.

They found no statistically significant differences in surgery-related deaths or complications when they compared the before and after pictures.

Lead author Dr. David Urbach, a general surgeon at Toronto’s University Health Network, suggested the impact of the checklist may have been “oversold.”

“For a study to say that 50 per cent (of surgery-related deaths) not only are preventable, but are actually prevented by the use of a … very brief, inexpensive, straightforward intervention like adhering to a checklist – it seems like a bit of an extraordinary claim,” he said.

“What we found was making extraordinary improvements in patient safety is probably going to take a lot more work than the introduction of something like these safety checklists.”

Proponents of the checklists agreed that merely instructing surgical teams to use a checklist won’t achieve the improvements seen in the 2009 study. And they suggested the Ontario results are proof of that.

Still, they argued that the fault was not in the checklist, but in not doing enough promoting of the system and training on it. They also suggested looking for big results so soon after implementation was unrealistic.

Dr. Atul Gawande, who led the WHO program to develop the surgical checklist, said it takes time and effort to effect the culture change required to allow the checklist program to be successful.

“I think the (Ontario) study is premature and incomplete,” said Gawande, a surgeon at Brigham and Women’s Hospital in Boston and a professor at Harvard School of Public Health.

“If you rolled out a seatbelt change … you wouldn’t measure three months later and say: ‘Oops, didn’t work.”‘

He said studying the system requires getting a clear picture on whether hospitals and individual surgical teams are actually following the checklist. When asked, most will say they are, he said. But studies of whether hospitals comply with other safety protocols suggest there can be a gap between what a hospital says it does, and what it actually does.

“If you asked every hospital ‘Do your staff wash their hands?’ 100 per cent will say ‘Yes. Absolutely,”‘ Gawande said.

“But if you look at how much resistant infection is spreading through our hospitals and whether people are actually doing it, it’s a whole different question.”

Leitch said surgical checklists are required in every operating room in Scotland. He said for the program to work properly, every member of a surgical team, even the most junior, should feel empowered to raise a concern or warn before a mistake is made – something that hasn’t been standard operating room protocol.

“That cultural change doesn’t happen if you do it as a tick-box exercise,” he said.

While Urbach looks at his study’s results through a different lens than Gawande and Leitch, he isn’t suggesting surgical checklists be dropped.

“I think the checklists are here to stay,” he said, adding they are helpful for improving operating team dynamics and engaging all members of a team.

Still, he thinks the Ontario results should raise some questions.

“One thing I’m hopeful (about) is that it will serve as a bit of a reality check to say that ‘You know it’s not this easy to create major improvements in what really is a large and complex problem,”‘ Urbach said.

“And I think it will hopefully continue to prompt people to explore new ways to improve safety and not sort of identify this as the be all and end all of patient safety interventions, which we’re at risk of doing. Because now checklists are proliferating and there’s a checklist for everything.”