The Siren Song of Deferred IRT Functionality

Craig Mooney

The siren was a creature in Greek mythology who lured sailors with her beautiful voice only to crash them against a rock or otherwise shipwreck them. So tempting, yet so potentially dangerous. In the interactive response technology (IRT) space one such siren song is deferred functionality. Now maybe this comparison is a bit dramatic, but I like a good metaphor and am prone to historical ones.

When timelines are short, and teams are focused on the “go live date” they often choose to defer some IRT functionality to a future release in order to reach first patient in (FPI) deadlines. My experience—from seven years working at a sponsor organization and from serving as an IRT provider in four separate engagements (including one as a business owner)—has repeatedly demonstrated that this approach leads to inefficiencies, compromised quality, and regulatory misalignment.

Let’s take a look, beginning with cause, motivation, and the inevitable solutions that bring us here.

The cause

In a dynamic clinical space, study designs can be fluid right up until the final protocol has been published. Protocol designs are influenced by emerging science, ongoing data review, and the marketplace. This often leaves supporting IRT vendors with a smaller than practical window between when the protocol is final and when all systems and processes must be available.

Even as eClinical technologies have improved to allow for shorter development timelines, regulatory/quality expectations and administrative overhead have increased and eaten away at any time savings these technology improvements have yielded. Quite simply, today it’s more complicated to start and run a clinical trial than it was 10 years ago. If we are honest with ourselves, we have not seen a dramatic reduction in timelines despite better technology.

The motivation

Into this environment clinical teams dispatch and focus on an FPI date which is chosen as the anchor for that study. Once published, this date is communicated as immovable and is the measuring stick for efficiency and proper project management. Since the date is often coupled with market or industry expectations, any change is difficult to negotiate or defend. That is of course right up to the minute it can’t be achieved, but more on that later. From my perspective, when you begin is not nearly as important as when you end. Blind allegiance to the study’s FPI date can cause a team to make decisions that may not be in the best, long term interests of the study.

The typical solution

Deferred functionality is the deployment of a system (in my examples here, IRT) that is not 100% complete or that doesn’t provide full coverage of the entire protocol. This incomplete system is meant to serve the minimum protocol requirements that are to be used in the time between this initial deployment and a final deployment that encompasses all requirements. In other words, the IRT vendor will initially only put live the elements of the system deemed critical to meet the FPI date until the whole system can be ready later. And in extreme circumstances, manual procedures may be required to supplant the system.

You might be asking how could you defer a part of the system? Good question. Here are just a few examples of the types of designs or scenarios in which I have seen a deferred functionality approach used:

• A crossover study is deployed without the crossover design/process since it will not be used until after 4 weeks of run-in followed by 6 weeks of treatment for a total of 10 weeks.
• The unblinding module is deferred for a study that has a 4-week open-label portion as no patients would require unblinding.
• The data from a third party that will impact patient dosing is not required until week 3.
• A single patient must be enrolled at a single site to meet a predetermined milestone.

It seems like deferred functionality is a great idea if these elements are not required until later in the study but taking a closer look can reveal some real concerns.

Inefficiencies

Administrative burden leads to a lack of efficiency for human, financial, and technical resources. Let’s start by recognizing that we are in a regulated industry governed by laws and organizational SOPs. Once the first live version of a study is deployed any changes must go through the change control process. That means rewriting and updating all the necessary documents in this process. Instead of once you must do it twice or even three times

Your UAT team would be much more efficient if they could tackle the entire project at one time. When a deferred functionality approach is taken, not only do they have to write scripts all over, but they will also need to do a certain level of regression testing as the IRT system is now considered a new version.

Scripts are just one part of the UAT package. Release documentation, peer/quality review, and TMF processes also need to be repeated. It is easy to dismiss this extra burden until you calculate how much additional time it requires and, in the moment, almost no one does. The UAT team is just one example of those engaged in the process who will be required to do duplicative work.

The same is true for sites. When manual processes are put in place for the same reason, the burden lands heavily on them. Duplication of effort and syncing up systems and data in the future creates chaos and fertile ground for errors and even the potential for impacts to the patient. I have sat with research pharmacists at one of the leading oncology centers in the world who have said they would not participate in such a study again.

Can you imagine almost any other circumstance where you find it acceptable for your team to duplicate work? If the benefit was certain and easily observable, you might make the argument. But keep reading, you’ll see that deferring IRT functionality is not the easy win some think it is.

Quality risks

Beyond the duplication of effort there is risk to the integrity of the study. In our industry errors have occurred, big ones. They only get mentioned in hushed tones at the cocktail hour of industry meetings, but they have happened. In my long career I have seen them up close and at a distance and several of them they have been born out of deferred functionality. When teams are engaged in delivering a single product there is greater focus on the integration and context of how the pieces go together.

My experience is that deferred functionality often creates a siloed mindset which leads to a lack of context. These deferred parts are seen individually, not as an integrated part of the whole system. In my professional experience, I’ve seen organizations reconsider or abandon the practice of deferred functionality altogether because of the risks it poses to study integrity.

Regulatory perspective

I concede that deferring functionality is a common practice in our industry. I personally have taken the approach during my career. But my perspective changed when I was challenged about this practice during a regulatory inspection. We were questioned why the organization would deploy an IRT system that did not meet the full requirements of the protocol. “No, its fine,” I tried to explain. “We won’t need to use that functionality for some time”. This was not a satisfactory response and led to further investigation and evaluation. I discussed this with others in the industry and they also felt that deferring functionality was a common practice. But as time went on, I started to hear more of what I had experienced during the regulatory inspection; this was a practice that regulators were opposed to.

Look to the data

Not too long after this, I incurred a lot of personal and professional pain as the result of having deferred IRT functionality in a study in which I was involved. I was converted. I wasn’t going to let this happen again. Frankly it also felt a bit lazy. Deferred functionality seemed to be the default approach whenever there was a chance a study’s timeline might be tight. It was just such an accepted practice. It seemed that at the slightest road bump, no one even considered what could be done to deliver the whole system. But I also needed data to understand it and to convince others there was little or nothing to be gained in deferring IRT functionality. More broadly, the data challenged the notion that IRT is the rate limiter for most studies to start on time.

What I felt and what has been reinforced over the years is that we hurry up to wait. The data I was able to review demonstrated that the time between deployment and first action is longer than most realize. It is the exception and not the rule that IRT systems are used in close proximity to their deployment. There are often very good reasons for this, such as changes required by the key drivers of protocol designs (science, data, and market). This is not a conviction of any one group, organization or even our industry.

I am suggesting you do what brought me to this understanding and look to the data. Your IRT business partner can help in identifying the time between deployment and first shipment or first participant screened/enrolled. You might be surprised what you learn. It’s important to be reflective and data-driven in understanding if deferred functionality is truly required.

Conclusion

I recognize that from time to time a situation like COVID-19 will come along and that the interests of patients and national health make deferred functionality a reality and even a necessity. This type of emergency notwithstanding I would encourage you to give very close scrutiny to any consideration of deferred functionality, lest you fall victim to the tempting siren song and find yourself—or your study—shipwrecked.

Craig Mooney, vice president, scientific eTech-enabled services, Calyx