When Does More Subgroup Analysis Equal More Subgroup Analysis?

A recent Op-Ed piece in the New York Times by Clifton Leaf asked the question “Do Clinical Trials Work?”

He covers a lot of ground in his piece, but one thing that caught our attention was his discussion of the use of subgroups by Genentech/Roche in the development of their breast cancer drug Herceptin. Genentech designed the Herceptin trial using representative subgroups of only the types of patients most likely to actually use the drug, were it to be approved.

Leaf applauded this approach and asked why it isn’t used more often. In a broader sense, he questioned the fundamental nature of clinical trial design and the methodology behind them.

Subgroup Analysis in HTAs

We have analyzed the use of subgroups by reimbursement agencies in their health technology assessments (HTAs) – we presented some findings at last year’s Canadian Agency for Drugs and Technologies in Health (CADTH) Conference – and that work reveals another facet of this issue.

Health Technology Agencies may be unfamiliar to a U.S. audience. Like private payers in the U.S. (insurance companies), these agencies decide which drugs should be recommended for reimbursement in countries with nationalized healthcare.

HTA agencies use clinical trials as evidence in their HTAs. The use of subgroups in clinical trials should be reflected by the use of subgroups in HTAs. Does this imply that if more clinical trials study subpopulations, more HTAs will cite them and if so, will they reach better conclusions?

As part of our analysis, we examined:

13 disease conditions
7 years (2005-2011)
9 agencies across 7 countries

Only 12% of these HTAs referenced analysis based on subgroups. There was some variation, but the proportion of reviews citing analysis based on subgroups was low among all of the agencies. Interestingly, twice as many (24%) of the reviews mentioned a desire to use subgroup analysis as actually used it. Of course, the true gap between desire and use was likely higher – presumably not all of those reviews that would have used evidence from subpopulations if it had been available revealed that fact.

This gap between the desire to review analysis based on subgroups and actually doing so may be telling. It suggests that there is a shortage of clinical trials using appropriate subgroup analysis.

Is there such a shortage?

We compared this 12% rate of use by HTAs to a sample of comparable clinical trials published in top journals. Almost 60% of these clinical trials used some form of subgroup analysis. While our study didn’t match the HTAs to appropriate candidate clinical trials, the large disparity in overall rates suggests that a simple lack of clinical trials using subgroup analysis isn’t the whole reason HTAs so rarely use subpopulations.

What else explains it?

The answer probably lies in their need for appropriate subpopulation analysis. Other researchers have found that a lot of subpopulation analysis is poorly conceived or executed, and is often misinterpreted. The problem may be less that there is insufficient quantity than that there is insufficient quality of subpopulation analysis.

This blog post initially appeared on Context Matters, Inc. It was reposted with permission.