When we announced we were publishing Surgeon Scorecard, a debate erupted over how and whether Medicare data can be used to compare surgeons. That conversation is certain to intensify as researchers, practitioners, and the public dig into our findings. We have created this page to facilitate that conversation.
The initial entries include comments from some of the patient safety experts who reviewed our methodology. Now that the scorecard is public, any expert who would like weigh in on our approach should send comments to: [email protected]. We may edit responses for length.
We look forward to hearing from you.
This post was updated on Oct. 8, 2015.
ProPublica’s methodology is one of several new methods to improve the detection of harm at the local level – in this case, across surgeons – and they all represent the future of patient safety. Recognizing that all of these methods have advantages and disadvantages, progress will be iterative and incremental. What each of these new methods reveals is that we are markedly underestimating the occurrence of harm with existing methods and that there is significant variation in the occurrence of harm locally, both at the unit and practitioner levels. This safety improvement opportunity – enabled by improved measurement – has never been more timely, as new studies suggest that inpatient safety problems may result in the death of over 400,000 patients a year and may result in more than 6 million injuries per year.
—Dr. David Classen, associate professor of medicine, University of Utah
CMIO PascalMetrics, a patient safety organization
I think the methodology was rigorous and conservative. I would be surprised if any experienced clinician challenged the basic finding, which is that there is real variation among surgeons. A critical step toward improving care is to recognize that there are opportunities to improve. I think transparency on quality is a powerful tool, and, frankly, I prefer that to financial incentives as a way to drive competition and improvement on quality.”
—Dr. Thomas Lee, a professor at the Harvard School of Public Health
It’s long overdue. It’s wonderful to have the information out in the public because then we can act on it. I think there’s going to be an immediate effect for patients. It will give them a tool to gather important information about quality. They will be able to make an informed choice about surgery. A larger benefit is going to be to help shatter the code of silence that surrounds complications and the quality of care that we provide. Hopefully, it will be a step toward a culture where transparency and open discussion of mistakes complications and errors will be the norm and not something that’s hidden.
In order to improve you first need to know where you started and know that there’s a problem. You need to have data. We’re beginning to collect the data, and now we need to know what to do with all of this data. The first thing to do is make everyone aware that there are problems.
—Dr. Charles Mick, former president of the North American Spine Society
This is an exciting milestone towards ending medical harm because for the first time we are seeing that there is significant variation in the quality and safety outcomes of individual surgeons – a variation that can have devastating effects on surgical patients. These variations must be identified in order to prevent poor outcomes. ProPublica has taken a first big step toward that goal.
This report dramatically illustrates the benefits to the public when Medicare makes its health care information public.That has allowed ProPublica to inform us about the quality and safety of surgeons’ care and can help consumers make better choices for elective surgeries.
Consumers have virtually no comparative information about the safety and quality outcomes of individual surgeons – until now. We have only been able to use hospital outcomes as a proxy for how surgeons perform. ProPublica’s in-depth report reveals significant variations among surgeons - even within a particular hospital – and demonstrates the compelling need for more physician-focused data and analysis.
Based on the findings in this report, consumers should demand more information for physician-focused data and analysis. Until now, consumers had virtually no comparative information about the safety and quality of surgeon’s performance.
—Lisa McGiffert, Consumers Union Safe Patient Project
The ProPublica measure is not valid. Though the methodology does account for some of the potential biases that might unjustly influence findings, it fails to account for another significant bias. For the ProPublica method to be a valid measure of surgical quality, all patients facing a potential readmission should have the same probability of being readmitted. Only then could readmission rates serve as a surrogate for complication rates and thus surgeon quality.
But patient factors such as their social support system, physician factors such as willingness to accept risk, and factors effecting access to care such as the presence of observation units or care in the emergency department, all impact whether a patient will be readmitted. Indeed, CMS has stopped reimbursing hospitals for admissions lasting less than two days because they recognize that the decision to admit a patient is arbitrary and that many of the same patients could be managed under observation.
The Methodology Needs Improvement: Even with these adjustments to the model, like any new quality measure, this would need to be tested and validated before it should be presented as a valid tool intended to assist consumers in their medical decision-making. In summary: the model uses an indirect measure of complications that fails to properly account for the variation in the reasons for a readmission.
—Dr. Peter Pronovost, senior vice president for patient safety and quality, director of the Armstrong Institute for Patient Safety and Quality, Johns Hopkins Medicine
Since the start of the modern patient safety movement, one of the key insights has been the powerful effect of transparency. Not only is transparency ethically correct, but it also creates conditions that make the health-care system better and safer. In this groundbreaking study, ProPublica has rigorously assessed the safety of eight common elective surgical operations. Their findings – particularly the surprisingly large differences in complication rates among individual surgeons – will influence health policy is meaningful ways. Moreover, if I were a patient who needed one of these surgeries, these results would carry substantial weight in my decision about which surgeon and hospital to use."
—Dr. Robert Wachter, Professor and interim chairman, department of medicine, University of California, San Francisco
Author of “The Digital Doctor” and “Understanding Patient Safety”
Consumers have wanted to compare doctors by name for a long time. ProPublica’s Surgeon Scorecard is a courageous database that identifies differences among physicians. While most surgeons are good to great in America, a small percentage lags expected performance. In addition to helping consumers, the easy-to-use Surgeon Scorecard empowers hospital boards of directors. Board members who want to make sure their hospital gives high quality care to the public, can use Scorecard data to simply ask: Does each surgeon meet our own hospital performance expectations?
Consumers will be delighted to use the Scorecard to compare orthopedic surgeons for hip and knee replacements. After seeing what is possible, consumers will demand even more. They already want to see complication rates for all patients, not just Medicare. They want to compare doctors for elective outpatient surgeries which outnumber inpatients. They want to be able to compare surgeons for tonsillectomies, hernia repair, cataracts, endoscopies and outpatient gall bladder surgery. Expanded transparency is the wave of the future.
ProPublica’s gutsy move may stir up a hornet’s nest, in part because the methods of risk adjustment are not perfect. Professional medical societies and coding experts will need to be the ones to judge the validity of the risk adjustment methodology. But for consumers, the new Scorecard is a welcomed contribution to help make our healthcare system safer, to spare anguish and save lives.
—Gretchen Dahlen, President of ConsumerHealthRatings.com
The North American Spine Society (NASS) applauds efforts of ProPublica journalists to bring transparency to health care, from its ongoing “Dollars to Doctors” feature to the newly-released “Surgeon Scorecard” database that offers complication rates for more than 17,000 surgeons throughout the United States.
NASS leadership worked with ProPublica reporters to identify appropriate elective spine procedures, define “complications” and offer ongoing risk-adjustment guidance and research for the “Surgeon Scorecard” project. “When ProPublica approached NASS for assistance on this effort in 2013, there was no doubt that we would actively participate and offer what assistance we could,” said Charles Mick, MD, an orthopedic surgeon and former NASS president. “It is our hope that the data released will help patients ask questions and make informed choices when surgery is required and help physicians and hospitals to identify and offer the highest quality care to patients.”
Dr. Mick counseled that while this database offers helpful information, patients should consider it as only one piece of data in their health care decision-making. They should be consulting with their primary care physician, spine specialists and the people who know them well—their family and friends—to make important health care decisions. To help patients get information on spine care conditions and treatments, NASS offers www.KnowYourBack.org, a free web site that provides credible, trustworthy, evidence-based information to the public, without any advertiser or promotional influence.
—The North American Spine Society
You and your colleagues are to be widely congratulated on your recently published Surgeon Scorecard. I am the Associate Medical Director Quality, Surgical Services at the Palo Alto Medical Foundation, a 1000+ physician multi-specialty group in California. I also direct our Outcomes Information Program (OIP) which is aimed at gathering meaningful safety (aka complication) and efficacy outcomes data following surgical intervention. Our goals are challenged by the fact that the bulk of our high impact surgical procedures are done at over a half dozen “non-owned” (by us) community and academic facilities. Thus the simple first step of data acquisition has been hugely problematic. Yet we’ve had some recent success and now are in the process of optimizing our reporting format and delivery. You have set a very high bar.
I’m impressed greatly by so much of what you’ve done with the Scorecard from reporting physician level outcomes to risk adjustment to the analytics used and the wonderful user interface and graphics. I am currently in the process of sharing your website and report with all my surgical colleagues as well as our leadership. I am hoping to leverage what you have to help us support our ongoing patient safety and quality improvement efforts as well as to help guide the ongoing development of our own data gathering and reporting process (OIP).
—John Cooper, M.D., the associate medical director quality, surgical services, at the Palo Alto Medical Foundation
What (ProPublica) created is, to our knowledge, the first purely objective public report card of health care quality at the individual physician level. While the result may be rough, the future should and will feature more polish. The nation needs more report cards like theirs and will get them, one way or another.
—Geoff Dougherty and Ben Harder, U.S. News & World Report
Disruptive innovation, a phrase coined by Clay Christensen, is usually a new product that, to experts, looks inadequate. Because it is. These innovations are not, initially, as good as what the experts use (in this case, their network of surgeons). They initially dismiss the disrupter as being of poor quality. But disruptive innovation takes hold because, for a large chunk of consumers (i.e. patients looking for surgeons), the innovation is both affordable and better than the alternative. And once it takes hold, it starts to get better. And as it does, its unintended consequences will become dwarfed by its intended consequences: making the system better. That’s what ProPublica has produced. And that’s worth celebrating.
—Dr. Ashish Jha, Harvard School of Public Health
A number of experts and other folks have criticized the methodology used by ProPublica to indicate the relative rate of complications for surgeons across America.
Here’s the issue in a nutshell, as I see it. There is a rigorous methodology available for evaluating surgical outcomes. It is from the American College of Surgeons, and it is called NSQIP. It is indeed the “leading nationally validated, risk-adjusted, outcomes-based program to measure and improve the quality of surgical care in the private sector.”… Ideally, ProPublica or others could publish the NSQIP results, except for one thing. Under the ACS rules, the evaluations must be held confidential.
So there we have it. We could all have a rigorously derived comparison tool, but since the profession chooses not to make it available, we must have a surrogate of the sort that ProPublica used in its article. Or nothing at all. What would be your choice?
—Paul Levy, former CEO of Beth Israel Deaconess Medical Center
The real danger of scorecards like ProPublica’s lies not in their failure to objectively capture quality but in their pretense that they succeed. Though some observers see such scorecards as superior to online ratings like those on Yelp, I disagree. Few people know how to interpret overlapping confidence intervals like ProPublica’s, whereas Yelp fully embraces its inherent subjectivity. … The irony in hailing the scorecard as a victory for transparency is that its purported objectivity obscures its methodologic limitations and the complexity of quality itself. No amount of transparency can overcome the fact that, when it comes to what we value, we don’t all see eye to eye. The real promise of transparency, then, lies in finding better ways to let our patients see what we see.
—Dr. Lisa Rosenbaum, cardiologist at Brigham and Women’s Hospital, in a commentary in the New England Journal of Medicine
ProPublica, in a seminal article, Making the Cut, via data, shows us the power of transparency in complications rates during surgery ... What Making the Cut elucidates is a new world order in healthcare. Everyone on a surgical team is now part of the statistical modeling paradigm; for better or worse. Was the surgeon responsible? Was it the nurse, the anesthesiologist, the post-surgical care, the patient, the follow up care coordination process? – who is ultimately responsible for a bad outcome that is not clear-cut. In many cases, they may never be clear … The Centers for Medicare & Medicaid Services data that powers the ProPublica article is a blunt instrument; like the first scalpel design – not sharp, not precise, but effective in making it’s point known. A growing chorus of physicians argue that they see a ‘sicker population,’ that their patients are ‘more complex,’ and while this may be true, the data scalpels will become more sophisticated over time and physicians should be designing these tools with every major stakeholder for the sole purpose of getting the best outcomes. After all, patients want the best outcome, and they are the whole point of medicine.
—Dr. Jordan Shlain, internist and founder of Healthloop, which automates follow-up care
ProPublica’s stated goals in producing the Surgeon Scorecard are laudable: ‘to provide patients, and the health care community, with reliable and actionable data points, at both the level of the surgeon and the hospital, in the form of a publicly available online searchable database.’ However, as with any performance report, the Scorecard’s ability to achieve these goals is limited by the rigor of the methods and the adequacy of the underlying data. Our critique of the ProPublica Surgeon Scorecard has identified substantial opportunities for improvement. Until these opportunities are addressed, we would advise users of the Scorecard—most notably, patients who might be choosing their surgeons—not to consider the Scorecard a valid or reliable predictor of the health outcomes any individual surgeon is likely to provide.
—Mark W. Friedberg, Peter J. Pronovost, David M. Shahian, Dana Gelb Safran, Karl Y. Bilimoria, Marc N. Elliott, Cheryl L. Damberg, Justin B. Dimick, Alan M. Zaslavsky in “A Methodological Critique of the ProPublica Surgeon Scorecard”, a Rand Corp. Perspective, Sept. 25, 2015
A final point about the Scorecard — and maybe the most important: This is fundamentally hard stuff, and ProPublica deserves credit for starting the process. The RAND report outlines a series of potential deficiencies, each of which is worth considering — and to the extent that it’s reasonable, ProPublica should address them in the next iteration. That said — a key value of the ProPublica effort is that it has launched an important debate about how we assess and report surgical quality. The old way — where all the information was privileged and known only among physicians — is gone. And it is not coming back.
—Dr. Ashish Jha, Harvard School of Public Health, in “Misunderstanding ProPublica: transparency, confidence intervals, and the value of data”, Oct. 8, 2015
Maybe our discomfort with the ProPublica Surgeon Scorecard comes from the fact that we were not involved in its methodology and we understand the limitations of its data source. But its publication should stimulate us to move our own reporting and safety mechanisms into high gear. That’s one reason the AAOS Now Forum next year will focus on risk assessment. Sure, ProPublica’s report card has its limits. But, if my cousin in California needed a cholecystectomy, I might suggest she look up her surgeon before scheduling the procedure.
—Dr. Eeric Truumees, editor-in-chief, AAOS Now, Nov. 2, 2015
The power of ProPublica’s site was demonstrated to me this morning when my brother sent me a text message that said, “Kathy (his wife) is having her knee replacement this morning.” I immediately asked for the surgeon’s name and then looked up his stats on ProPublica’s new site. In less than 5 minutes, I had information that would have previously taken hours, if not days, of phone calls to colleagues and professional acquaintances for recommendations and best guesses about who would be best for my sister-in-law to see.
—Dr. John Byrnes, former Chief Quality Officer of Spectrum Health, Nov. 2, 2015
