Clinical Researcher—April 2020 (Volume 34, Issue 4)

PEER REVIEWED

Kara Lorduy, PhD, CCRP; Victoria Brown, PhD, MBA, CIP, CCRP; Suzanne J. Rose, MS, PhD, CCRC

 

[Editor’s Note: To view the Figures noted in this paper, please access this PDF version.]

According to a recent report on “U.S. Investments in Medical and Health Research and Development,” the biopharmaceutical industry spent approximately $15 billion in direct costs in the establishment of roughly 4,500 clinical trials in the United States in 2017. These sponsored trials included more than 920,000 participants.{1} Considering that 86% of trials in the United States fail to enroll before the contracted period,{2–4} a research program’s stewardship of qualified and well-trained study coordinators, who are tasked with balancing ever-increasing regulatory demands and protocol complexity,{5} is paramount to its success.

This paper surveys the current state of the field and compares two similar adaptations of the Ontario Protocol Assessment Level (OPAL) in tracking productivity at their respective research programs.{6} The findings provide a compelling case for improved efficiency and productivity, increased job satisfaction and retention, and higher levels of funding over prolonged use of adaptive productivity metrics.

Background

It is well known that coordinating a successful research program has become very challenging in today’s clinical research environment, due in part to greater protocol complexity, fewer available studies, decreasing site budgets, high levels of staff burn-out, and an increased regulatory burden.{7} In addition, research sites are often expected to project staffing needs in order to  bring on new trials, maintain existing ones, and stay abreast of the regulatory demands for multiple studies. In order to manage the workload of these studies, there is a need to better understand the time, personnel, and financial resources needed to conduct clinical trials. The benefits of this focus include increased enrollment success, funding, efficiency, quality, and job satisfaction and retention of study coordinators.

Historically, federally funded research programs have been guided by the 1992 National Cancer Institute (NCI) Cancer Clinical Investigations Review Committee algorithm of 1.0 full-time equivalents per 40 enrollments.{8} However, meta analyses across 51 research programs alongside many other pivotal developments in this arena,{4} have led to a growing consensus that productivity models should incorporate complexity, or acuity, as well as the regulatory and administrative tasks in their metrics.{9,10}

Recent attempts have been made by various groups and sites to develop workload tools that adequately address the true workload of clinical research coordinators (CRCs). The tools formulate the workload effort through various mathematical calculations.{9,11–17} In an effort to portray the evolution of efficiency in clinical research practice, we depict the replication of the OPAL{6}  metrics on  improved efficiency at one site (Children’s Health System of Texas) following one year of implementation productivity metrics, while another site (Stamford Health) presents the compounding benefits of adapting the OPAL protocol acuity rating metrics to comprehensively include additional workload factors, coordinator roles, and phases of the study cycle in the Clinical Research Workload Tool (CRWT) across eight years.

OPAL and the CRWT overlap in terms of using an eight-point protocol score, or multiplier toward the total enrollment number, with simple studies (e.g., registries) rating “1” and complex studies (e.g., Phase I) rating “8.” Comprehensively, the CRWT model adjusts the protocol score considering the study role contributions (i.e., data, regulatory, nurse, and coordinator) and adds weights to the score for each additional workload factor (e.g., industry trials, duration or number of visits){18} to comprise a CRWT score or multiplier toward total number of active enrollments before adjusting for the phase of the study (e.g., start-up, enrolling, follow-up).

Importantly, our findings support the growing body of knowledge regarding the adaptation of metrics originally explored in oncology research programs for use in these two non-oncology research programs at two distinct phases in their development of adapting OPAL-based metrics. The non-oncology programs aimed to develop a common currency of productivity that could be benchmarked and leveraged to improve efficiency and progress in their clinical research practice.

Methods

To reiterate, we present two research programs utilizing an adaptation of the OPAL productivity metrics. The first site, Stamford Health, collects data relational to use of the CRWT model across eight years. The second site, Children’s Health System of Texas, collects data relational to use of another adaptation of the OPAL metrics in the first year of implementation.

Stamford Health

The CRWT was developed based on the OPAL workload planning tool.{6} Stamford’s model also allowed for an “other” category, which was determined at the site level for an extenuating circumstance that added protocol complexity. With the addition of the complexity modifications, the total CRWT score could amount to as high as 12.

In addition, another novelty the CRWT accounts for is that, at Stamford, regulatory coordination and data management tasks are assigned to other personnel than CRCs. To account for the regulatory and data management burden not being part of the typical CRC’s daily workload, the CRWT offers additional reductions in workload by 25% per additional resource based on a previous workload study in which coordinators recorded their time spent on protocol management, eligibility, and entry, treatment, follow-up, and final stage. Approximately 25% of clinical research associate (study monitor from a sponsor or contract research organization) time recorded was spent on protocol management (regulatory coordinator responsibilities) and 25% of time on follow-up and final stage (data manager responsibilities).{19}

Data were collected over an eight-year period. CRWT scores for 14 coordinators were recorded on a monthly basis. A total of 606 CRWT scores were calculated in this time period (M=117, SD=57.47).

Children’s Health System of Texas

In a similar adaptation of OPAL, the enrollment-derived productivity of a separate, non-oncology clinical research program in the first year of implementation (T1=January 2017; T2= January 2018). Consistent with the methodology defined previously,{6} all protocols included in the research program’s portfolio were scored. Children’s Health then examined the workload of 10 experienced CRCs. The total workload (the DEVO score) for each coordinator was constituted by two components: 1) enrollment derived productivity using the OPAL (i.e., the OPAL score) apprised workload method{9} and 2) all contributions made to the developmental department initiatives (the DEV score) including, but not limited to, writing standard operating procedures, internal quality reviews, and training (see Figures 6 and 7).

The DEV scores were computed by adding additional points to overall workload score by counting the total number of hours spent across four primary categories (regulatory, training, developmental, and patient care) reported in a time tracking system and dividing them by a factor of two. Additionally, coordinators completed a five-point Likert scale assessing their perceived fairness of the metrics system, as well as how likely they were to “still be working at the department two years from now” and their overall job satisfaction.

Results

Study Conducted at Stamford Health

Figure 1 shows characteristic CRWT monthly score profiles over a four-year period for two full-time oncology (CRC 2 and CRC 3) and three non-oncology (CRC 1, CRC 4, and CRC 5) coordinators who worked four years contiguously.

[Figure 1: Workload is highly variable among full-time oncology and non-oncology coordinators. Oncology coordinators average a higher workload (M=193.5, SD=17.78) than non-oncology coordinators (M=104.5, SD=36.13). The difference in means was highly significant by a t-test for two independent groups, t(45)=-10.13, p<.001. This can be attributed in part to the higher number of trials that oncology coordinators (M=19.25, SD=1.86) conduct versus non-oncology coordinators (M=8.14, SD=4.06). The difference in means was highly significant by a t-test for two independent groups, t(45)=-11.34, p<.001.]

Based on frequent coordinator assessment of their workload (too light, moderate, heavy, or unbearable) and capability of conducting assigned studies (yes, somewhat, no), four categories were assigned (see Table 1).

Assigned Study Coordinator Ranges
Range		Zone
<100–125		light green
126–150		green
151–175		orange
175–200+		red

Table 1: Coordinators in the light green zone were usually new research coordinators. Coordinators in the green and orange zones were more confident with workload and most capable of taking on new studies. Coordinators in the red zone often felt overworked.

In addition to calculating monthly CRWT numbers, annual revenue productivity per coordinator was calculated and adjusted according to whether the coordinator left the institution. A scatter plot of annual revenue and average annual workload was created, and Pearson’s Correlational analyses were used to examine the relationship between the annual revenue and average annual workload (see Figure 2).

[Figure 2: For non-oncology coordinators, a higher workload correlated to greater revenue generation. There was a strong statistically significant positive correlation between average annual workload (M=104, SD=36.13) and annual revenue (M=$121,697; SD=$63,681), r=.62, p<.001, n=27.]

The positive correlation does not appear to be related to variation in study characteristics, as they did not vary greatly during the eight-year period, as shown in Table 2. Studies are categorized as Device (Pre- and Post-Market as well as total Device), Drug (Phases I–IV) and Registry trials not categorized as Phase IV trials. As evident in the data, Stamford focuses primarily (in order) on Phase III drug trials, Pre-Market device studies, and Registry-type trials.

Table 2: Study Characteristics for Non-Oncology Enrolling Trials
Study Type	n	Mean	SD
Total Device	8	6.50	1.60
Pre-Market	8	5.13	0.83
Post-Market	8	1.88	0.64
Total Drug	8	9.88	4.76
Phase I	8	0.63	0.74
Phase II	8	1.38	1.51
Phase III	8	7.88	3.64
Phase IV	8	0	0
Registry	8	3.25	1.83

For oncology coordinators, a scatter plot of annual revenue and average annual workload was created, and Pearson’s Correlational analyses were used to examine the relationship between the annual revenue and average annual workload (see Figure 3). This was despite a dramatic increase in the number of subjects accrued to industry-sponsored trials in the last two years (see Figure 4).

[Figure 3: For oncology coordinators, a higher workload negatively correlated to revenue. There was a strong statistically significant negative correlation between average annual workload (M=196, SD=15.4) and annual revenue (M=$106,094; SD=$49,588), r=-.71, p<.001, n=20.]

[Figure 4: Enrollment for Oncology Coordinators by Study Type]

For all coordinators, no correlation was found between average annual workload (M=142, SD=53.36) and annual revenue (M=$115,057; SD=$58,039), r=.065, p=.666, n=47 (data not shown).

Study Conducted at Children’s Health System of Texas

Two-sample, paired t-tests were conducted to test the hypothesis that tracking productivity would improve the efficiency of a clinical research program’s enrollment-derived productivity over the course of a year (T1=January 2017; T2=January 2018) holding total coordinator hours worked constant (see Figure 5). Additionally, we tested the effectiveness of our productivity tracking metrics to improve study coordinator efficiency, using two-sample, paired t-tests to compare mean percentages of study coordinator time spent for non-enrollment derived activities  (regulatory, training, departmental initiatives, patient-facing) at two distinct time points (T1; T2) holding total coordinator hours worked constant (see Figure 6).

[Figure 5: Total cumulative enrollment was significantly increased from T1(M=295.30, SE=135.79) to T2(M=350.90, SE=151.49), t(9)=-2.80, p<.01.]

[Figure 6: Mean monthly productivity scores significantly increased in enrollment-derived productivity (OPAL) from T1(M=52.23, SE=10.60) to T2(M=103.10, SE=24.98), t(9)=-2.35, p=.02, demonstrating increased productivity over time as expected. As expected, overall productivity (DEVO), including enrollment-derived productivity (OPAL) and developmental activities (DEV), significantly increased from T1(M=87.86, SE=11.49) to T2(M=141.70, SE=22.79), t(9)=-2.37, p=.02.]

Together, these results demonstrate that increases in enrollment-derived activity coincided with significant increases in the mean percentage of time study coordinators spent in patient-facing activities in the first year (see Figure 7).

[Figure 7: Mean percentage of time spent toward patient-facing activities was significantly increased from T1(M=.30, SE=.05) to T2(M=.51, SE=.08), t(9 =-3.19, p<.01, demonstrating increased efficiency over time as expected. Additionally, the increase in patient-facing activities coincided with a significant decrease in mean percentage of time spent toward regulatory from T1(M=.45, SE=.07) to T2(M=.29, SE=.07), t(9)=2.02, p<.05.]

Additionally, the retention rate of our coordinator staff increased from 40% (2016–17) to 86% (2017–18) post-implementation of the aforementioned productivity metrics system. Pearson correlations of the self-reported perceived fairness of the procedures for measuring and administering study workload assignments to the self-reported job commitment and satisfaction levels are presented in Figure 8.

[Figure 8: Relationship Between Self-Reported View Toward Procedures for Fairness of Workload Distribution and Self-Reported Job Commitment and Satisfaction]

Discussion

This paper focused on two adaptations of the OPAL productivity model considering protocol complexity, the number of procedures, and additional regulatory and administrative tasks to build a more comprehensive model and representation of coordinator workload.

Where the reproducible results of OPAL to improve efficiency and productivity in clinical research practice can be gathered from the data presented concerning two study sites, this paper elucidates how the benefits of efficiency are fostered through adapting certain metrics. Specifically, these metrics capture the redistribution of workload across study roles to optimize and specialize staff, and more centrally, the value of encompassing additional workload factors for a more comprehensive and accurate model.

Moreover, a robust relationship between perceived procedural fairness of study assignments and workload distribution to self-reports of job commitment and satisfaction was presented, and is further bolstered by the 46% increase in retention following the introduction of adaptive productivity metrics to a novel site (Children’s Health System of Texas). It is confirmed that understanding what additional workload factors the coordinators are managing and incorporating them in your adaptive metrics helps gain their buy-in and perception of fairness to improve their work-life balance, job commitment, and satisfaction to retain them as an asset to your program.

Additionally, the maturation of benefits that can be expected after several years using an adaptive productivity metric system (CRWT) includes increased funding support, as evident from the eight years of data presented from Stamford Health.

Conclusion

The model shown here offers great flexibility in both oncology and non-oncology settings, as it allows for continuity of care amongst study coordinators for their study participants. These metrics can be used to justify new and existing employees for research programs running clinical trials predominately on the high end of the acuity continuum. In addition, the metrics can be used to increase funding for additional coordinator and support staffing while improving study coordinator job satisfaction and retention at sites. This allows sites to achieve their enrollment goals and promote progress in their clinical research practice. In the wake of such improvements, research programs can expect more funding opportunities and greater success.

References

1. https://www.phrma.org/-/media/TEConomy_PhRMA-Clinical-Trials-Impacts.pdf
2. Anderson D. 2001. A Guide to Patient Recruitment: Today’s Best Practices and Proven Strategies. Boston, Mass.: CenterWatch/Thomas Healthcare.
3. appliedclinicaltrialsonline.com/subject-recruitment-and-retention-barriers-success
4. Treweek S, Lockhart P, Pitkethly M, Cook JA, Kjeldstrøm M, Johansen M, Taskila TK, et al. 2013. Methods to Improve Recruitment to Randomised Controlled Trials: Cochrane Systematic Review and Meta-Analysis. BMJ Open 3(2). https://doi.org/10.1136/bmjopen-2012-002360
5. Getz KA, Campo RA. 2018. New Benchmarks Characterizing Growth in Protocol Design Complexity. Therap Innov Reg Sci 52(1): 22–8. https://doi.org/10.1177/2168479017713039
6. Smuck B, Bettello P, Berghout K, Hanna T, Kowaleski B, Phippard L, Au D, Friel K. 2011. Ontario Protocol Assessment Level: Clinical Trial Complexity Rating Tool for Workload Planning in Oncology Clinical Trials. J Oncol Pract 7(2): 80–4. https://doi.org/10.1200/JOP.2010.000051
7. Getz KA, Wenger J, Campo RA, Seguine ES, Kaitin KI. 2008. Assessing the Impact of Protocol Design Changes on Clinical Trial Performance. Am J Therap 15(5):450–7. https://doi.org/10.1097/MJT.0b013e31816b9027
8. Gwede CK, Johnson D, Trotti A. 2000. Tools to Study Workload Issues. Appl Clin Trials 9(1):40.
9. Good MJ, Hurley P, Woo KM, Szczepanek C, Stewart T, Robert N, Lyss A, Gönen M, Lilenbaum R. 2016. Assessing Clinical Trial-Associated Workload in Community-Based Research Programs Using the ASCO Clinical Trial Workload Assessment Tool. J Oncol Pract 12(5):e536–47. https://doi.org/10.1200/JOP.2015.008920
10. Morin DJ. 2020. Harmonizing Protocol Complexity with Resource Management and Capacity Planning at Clinical Research Sites. Therap Innov Reg Sci. https://doi.org/10.1007/s43441-020-00120-8
11. Fowler DR, Thomas CJ. 2003. Issues in Research Management: Protocol Acuity Scoring as a Rational Approach to Clinical Research Management. Res Pract 4(2):64–71.
12. National Cancer Institute. U.S. National Institutes of Health. 2009. NCI Trial Complexity Elements & Scoring Model. http://ctep.cancer.gov/protocolDevelopment/docs/trial_complexity_elements_scoring.doc
13. National Cancer Institute. 2018. Assessing Clinical Trial Associated Workload (Webinar). dcpaquip.com/Documents/Media/Assessing%20Clinical%20Trial%20Workload_Full%20Slide%20View.pdf
14. Richie A, Gamble D, Tavlarides A, Griffin C. 2019. Trial Complexity and Coordinator Capacity: The Development of a Complexity Tool. Clinical Researcher 33(5). https://acrpnet.org/2019/05/14/trial-complexity-and-coordinator-capacity-the-development-of-a-complexity-tool/
15. Richie A, Gamble D, Tavlarides A, Strok K, Griffin C. 2020. Establishing the Link Between Trial Complexity and Coordinator Capacity. Clinical Researcher 34(2). https://acrpnet.org/2020/02/11/establishing-the-link-between-trial-complexity-and-coordinator-capacity/
16. James P, Bebee P, Beekman L, Browning D, Innes M, Kain J, Royce-Westcott T, Waldinger M. 2011. Creating an Effort Tracking Tool to Improve Therapeutic Cancer Clinical Trials Workload Management and Budgeting. J Nat Compr Cancer Network 9(11):1228–33.
17. Good MJ, Lubejko B, Humphries K, Medders A. 2013. Measuring Clinical Trial–Associated Workload in a Community Clinical Oncology Program. J Oncol Pract 9(4): 211–5. https://doi.org/10.1200/JOP.2012.000797
18. Rose S. 2018. Assess Clinical Trial Staff Workload and Maximize Site Performance (Webinar). https://go.forteresearch.com/webinar-assess-clinical-trial-staff-workload-and-maximize-site-performance?_ga=2.67391190.652342446.1581515325-121221525.1581515325
19. Roche K, Paul N, Smuck B, Whitehead M, Zee B, Pater J, Hiatt M-A, Walker H. 2002. Factors Affecting Workload of Cancer Clinical Trials: Results of a Multicenter Study of the National Cancer Institute of Canada Clinical Trials Group. J Clin Oncol 20(2):545–56.

Kara Lorduy, PhD, CCRP, is a Clinical Research Coordinator with Children’s Medical Center of Dallas.

Victoria Brown, PhD, MBA, CIP, CCRP, is Director of Research Administration with Children’s Health System of Texas.

Suzanne J. Rose, MS, PhD, CCRC, is Director of the Office of Research at Stamford Hospital in Connecticut.