But randomized controlled trials (RCTs) rarely happen in surgery. Now, a new report argues that these trials don’t come to fruition because the organization and support to help trialists plan and carry them out is lacking.
In a study published in the September issue of the American Journal of Surgery, researchers from Houston’s Methodist Hospital show that the processes in place to help surgical clinical trialists are haphazard and inefficient, failings that likely contribute to the low numbers of RCTs in surgery (2012;204:339-346.e5).
“Our findings show a remarkably unsystematic process by which highly motivated individuals struggled to find the information and resources necessary to develop competence as trialists and to mount scientifically and ethically sound surgical trials,” concluded lead author Anna F. Jarman, BA, then a graduate student at The Methodist Hospital, and colleagues.
That lack of support and resources translates into a poor showing for surgeons in clinical trials, said the researchers. Only 7.6% of studies evaluating surgical procedures use a comparative clinical trial design, according to another study published online this summer in the Journal of the American College of Surgeons (2012 [Epub ahead of print]).
“Our results show that the methodologic quality of surgical trials, and their reporting, are in need of significant improvement,” said the authors.
The researchers acknowledged that unique hurdles to clinical trials exist in the field of surgery, but they argued that high-quality surgical research can be completed, even trials that include randomization and blinding, if a better support system existed.
The real failing is inadequate training of surgeons and researchers in clinical trial methods, which results in trials that end with “findings of questionable scientific value,” said the authors. They cautioned that such failings have significant ethical implications for patient care.
Senior authors Carol Ashton, MD, MPH, an internist, and Nelda Wray, MD, MPH, a pulmonologist, have worked together on quality assessment research for the past 30 years. Since 2008, they’ve focused their efforts on surgical trials, looking at the quality—or lack thereof—in surgical trials.
In their most recent project, the investigators interviewed 11 accomplished surgical trialists, each of whom played a role in one or more highly cited surgical trials. They also interviewed an additional four trialists who were attempting to launch their first trial. The group included 11 surgeons, two physicians, a research ethicist and a biostatistician. (Of the 11 surgeons, nine currently hold a leadership role such as a deanship or chairmanship of an academic department.) All were asked about their use of clinical research resources and their perspectives on the practical utility and educational value of these resources.
Almost all interviewees said they learned how to conduct trials as they went through the process of setting up a trial and not from their educational training. They said they gained knowledge by actually performing trials, and to a lesser extent working as reviewers or as members of an institutional review board. Said one: “Through that trial I just learned so much about clinical trials. … I became a more seasoned investigator throughout the trial.”
The respondents made statements such as the experience was “painful,” or they were “learning by making mistakes” and “had to do things on my own.”
“What we found was a true sadness among those we talked to—sadness and frustration at the [inability] to get guidance, training and support when they had, as surgeons, a cutting-edge important idea,” said Dr. Ashton, the John F. and Carolyn Bookout Professor of Surgical Quality and Outcomes Science at The Methodist Hospital, Houston, and a research professor of medicine in surgery, Weill Cornell Medical Center, New York City.
What makes the findings even more remarkable is that the surgeons surveyed represent a highly educated group in terms of clinical trial training, a group expected to have a smooth experience organizing a clinical trial. Nearly half of those surveyed had obtained some kind of formal education in clinical trials beyond their doctoral degrees. Nine of the 11 surgeons interviewed undertook a year or two of research as part of their training (although their research years were spent mostly in laboratories and not in clinical trials). And six of the physicians interviewed had voluntarily sought out formal training in clinical research leading to a master’s degree or graduate coursework. Four surgeons had participated in seminars in clinical trial methods offered by organizations such as the American College of Surgeons or the Veterans Affairs Cooperative Studies Program.
Most respondents said their didactic instruction prior to starting the trial was inadequate. Instead, they reported that informal training from other people such as mentors, peers or collaborators provided “some of the most important education on how to develop and execute a trial.” Even anonymous reviewers who provided
critiques on protocols, applications and manuscripts helped trialists to learn more about
Experienced surgical trialists said the report paints an accurate picture of the current state of surgical research.
Heidi Nelson, MD, the Fred C. Andersen Professor of Surgery at Mayo Clinic, in Rochester, Minn., is one of the country’s leading surgical trialists. She co-chaired the American College of Surgeons Oncology Group for seven years and led the landmark COST trial, a multi-institutional Phase III prospective randomized trial comparing laparoscopic and open colon cancer surgery.
Dr. Nelson agreed that the trial process is fraught with frustration, but said that the study underplays one element of the process: the immense sense of reward that comes with completing a trial. “I certainly share some of the frustrations, but I found clinical trials vastly rewarding—challenging but vastly rewarding.”
Clinical trials have not historically been a part of surgical culture, said Dr. Nelson. Randomized trials do not integrate easily into surgical practice, with issues like how to reimburse, how to blind, and whether sham operations are ethical. And surgeons are not always comfortable with the idea of equipoise, she said.
Dr. Nelson compared surgical trials with medical trials. “Medical trials are often formulaic, whether it’s a new chemotherapy agent that goes through Phase I, II and II trials or a diabetes medicine that goes through a development testing model.”
Surgery lacks a similar well-built paradigm, she said. “In large part, that’s because surgical innovations and surgical technologies do not have the same straightforward path. The pipeline for surgical trials is not as well established or linear as for medical trials.”
The study authors pointed out that a significant body of published literature and guidelines designed for surgical trialists exists and is growing annually. Part of the problem, they said, is that surgeons do not use it.
To date, more than 1,000 documents have been published that provide guidance on clinical trials, including 302 from the United States alone. Three-fourths of the guidance documents were released or updated in the past decade.
According to the survey, surgical researchers generally do not use the documents and guidance published by regulatory and professional organizations. Only three out of 15 interviewees mentioned published guidelines for trialists.
“This seems unfortunate, given the wealth of information these documents contain on how to perform ethically sound and methodological rigorous clinical trials and how to report them,” said co-author Dr. Wray, also a senior member of The Methodist Hospital Research Institute. Dr. Wray served as the lead scientist on a landmark trial of knee arthroscopy published in 2002 that included a sham procedure control group (N Engl J Med 2002;347:81-88).
Barbara Bass, MD, the John F. and Carolyn Bookout Distinguished Endowed Chair of Surgery at The Methodist Hospital, said the study hits on “an important theme, long bemoaned, but rarely studied effectively in the surgical community.”
Dr. Bass, who was not a study author, said there are not enough resources within academic departments today to thoroughly train residents and fellows to be clinical trialists in surgery.
“Our surgeons in academic departments are expected to operate, teach, support institutional programs, and on top of that, be academically productive. Most chairs believe that training a cohort of surgeons to be accomplished surgical trialists is an important goal. Unfortunately, gathering the resources to support that mission is a tough challenge, particularly as return on investment, extramural research funding, for that type of research in surgery remains very lean.”
In their report, the study authors laid out some suggestions for trialists and would-be trialists, based on their interviews with surgical trialists. They urged aspiring trialists to obtain formal training in the methods and issues of clinical trials (both the programs from the ACS and VA were recommended); to seek out mentors experienced in clinical trials; and to establish a strong network of collaborators like statisticians and psychometricians. Finally, they recommended that surgical trialists seek out positions that get them involved in various aspects of clinical trials. As an example, joining a hospital’s IRB or participation in a trial can be helpful, they said.
In an interview, the investigators also made a broad recommendation to the surgical community on how to raise the bar on surgical trials. They called on the surgical community to establish a network of surgical trials that has consistent funding and includes surgeons and experts from other scientific fields.
“We’d love to see an effort across the country that ensures stable funding for a surgical cooperative randomized trial network so that new interventions can be tested thoroughly before they widely disseminate,” said Dr. Wray.
Many barriers to such a program exist, the top being a lack of funding.
But the investigators said that they expect more funds will become available for surgical trials as a result of the Patient-Centered Outcomes Research Institute, created by the 2010 Accountable Care Act. “I see this as a great opportunity now. Traditionally it’s been very difficult to get funding for non-industry surgical trials,” said Dr. Wray.
Without more organization, the authors said, surgeons and surgical trialists aren’t likely to attract funding dollars. The slapdash structure for surgical trials also reduces the chances of gaining external support, financial or otherwise, when up against “highly trained competitors from fields outside surgery,” the investigators said.