This case study examines using centralized manual data review with statistical approaches to compare value and fit.
Risk based monitoring (RBM) has been a widely discussed topic in clinical research in recent years but currently there is limited literature providing examples of practical application of techniques, in particular where an organization has limited resources for implementation of new approaches to trial oversight. The biopharma industry currently relies on extensive feet-on-the-ground monitoring to provide an assessment of site risk. Many organizations have been supplementing this with centralized manual (by eye) data reviews to identify patterns which may indicate where additional resources and effort are best placed, however this involves the sourcing and combination of multiple data sources, often found in hard copy or on traditional spreadsheets. This ad-hoc labor-intensive process is both prone to error and time consuming.
This exploratory case study illustrates how RBM can be applied using a specific technology solution from Algorics, called Acuity, and examines both data-driven models and centralized manual data review. The case study was performed in collaboration with Neuroscience Trials, a not-for-profit clinical research organization that specializes in neuroscience clinical research located in Melbourne, Australia. Since this was an exploratory exercise to compare the value and fit of different RBM approaches, data from a completed study in stroke recovery was used. The key objectives of this study were:
Methodology
The clinical trial selected had the following profile:
The following steps were carried out once the study was selected:
Risk parameter
Observation score
Deviation - Stroke onset to clot buster
Less than 4.5 hrs
0
More than 4.5 hrs
2
Deviation - Stroke onset to groin puncture
Less than 6 hrs
0
More than 6 hrs
2
Unreported AEs based on 90 day conmed
No unreported issues
0
1 AE unreported
1
> 1 AE unreported
2
Non-congruence between NIHSS, mRS, Home time & AEs
No issues
0
Non-congruence issues
2
Death
No
0
Yes
1
Two approaches were used to review and investigate the data analysis:
Centralized manual data review
Data review: Data visualizations of risk parameters for each subject were reviewed from Acuity and output of the review was generated in a standardized data review sheet using the risk scoring guide. The risk score for each subject was obtained after adding up the observations for each risk parameter. This process ensured that observations were objectively and quantitatively captured at a subject level.
Data driven analysis
Need for a data-driven exploratory study: The authors further explored if there is an opportunity to apply a data-driven model, which in combination with the output of site risk classification based on manual review could help in improving risk characterization of a site. In order to do so, key study data points were chosen that could provide insight into functioning of the site. In this clinical trial, the subjects who were enrolled had suffered ischemic stroke. From a treatment management perspective, it was important that they be treated as early as possible from onset of stroke. Given the various factors that can impact the time from stroke onset to hospital admission, the clinical management team advised that delay of treatment from the time of admission in the Emergency department (ED) to intervention (clot-busting agent or groin puncture time for clot-retrieval) would be a good surrogate indicator to understand a site’s propensity to treat their patients in optimal timeframes. It was also recognized that there could be various factors at play including hospital practices, study team involvement, investigator availability and patient illness-specific situations that can determine the time to treatment and or intervention. Nevertheless, the macro purpose was to determine if there were consistent delays while treating subjects at a site.
Data-driven model:
Two types of data-driven models were used to estimate subject management risk at a site.
The Acuity Decision Factory module which allows data-driven models to be applied to study data was utilized. The time taken from being admitted to ED to administering the clot buster and ED to groin puncture (if randomized) were taken for all the subjects enrolled at a site and then classified into three grades of risk using 50th percentile and 90th percentile as thresholds. 1 point for < 50th percentile (ED to clot buster time of 0.7 hrs & ED to groin puncture time of 2.9 hours respectively), 2 points for 51 to 89th percentile (ED to clot buster time of 0.71 to 1.39 hrs & ED to groin puncture time of 1.81 to 2.89 hours respectively) and 3 points for >90th percentile (ED to clot buster time of > 1.4 hrs & ED to groin puncture time of > 2.9 hours respectively) at subjects across all sites.
k-means clustering is a method to partition observations into clusters in which each observation belongs to the cluster with the nearest mean. The data pertinent to the aforementioned surrogate indicators was scaled in the statistical software R (R is free software designed for statistical computing) for comparison.
The model broadly categorized subjects into 2 types:
The data for each category was totaled for all subjects at a site level to derive the incidence of subjects where the procedure was delayed. The incidence of delayed treatment and/or intervention across subjects provided the risk profile of the sites.
Results
Table 1
Site
Overall risk
J
4.87
High
E
2.2
High
C
1.87
High
F
1.53
Medium
H
1.45
Medium
I
1.2
Medium
D
0.87
Low
B
0.8
Low
A
0.72
Low
G
0.45
Low
Table 2 Table 3
(b) k-means cluster method: Results are described in Table 4.
Table 4
Observations
Validation of the exploratory analysis
Since the exploratory analysis was carried out independently of the study team members, it was important to ascertain if the output of the models aligned with assessment of site performance and risks per the site monitoring team. Qualitative inputs from the site monitoring operations team were taken based on their monitoring experiences during the course of the study. There was consensus that the observations from the exploratory analysis agreed with their opinion of the high and medium risk sites. It was also interesting to note that the monitoring team highlighted that the exercise had indeed detected the highest and lowest risk site (Sites J and A respectively).
Further, when the monitoring team was provided access to the data visualizations, they reported that a single complete subject centralized manual review required about 25 minutes for review that subsequently reduced to about 15 minutes after getting used to the modified review methodology. This supported the ease of use of a RBM agnostic technology solution in data review. In conventional centralized manual review which relies on spreadsheets, the same review process could have required at least an hour for review.
Conclusion
This exploratory exercise helped in drawing the following conclusions:
Abby Abraham is a co-founder of Algorics and serves as Vice President, Clinical Solutions. The author wishes to thank Tina Soulis and the team at Neuroscience Trials, Australia for participating and providing scientific inputs that helped in building this case study.
How Digital Technology and Remote Assessment Strategies Can Aid Clinical Trial Research
July 24th 2020While there's been hopeful news on treatments and vaccines, sponsors should plan to discuss necessary strategies and contingencies at the outset of new studies or re-opening of halted studies during the COVID-19 pandemic.