FDA Issues Cautionary Reminder to Medical Device Manufacturers Regarding Data Submitted from Third-Party Laboratories
Agency reports concerns amid an increase in submitted data that has been fabricated, duplicated, or unreliable from third-party laboratories.
The FDA this week issued a warning to companies that manufacture medical devices regarding the quality of data submitted from third-party laboratories. The agency reported that there has been a considerably high amount of fabricated data coming from these labs, with a significant volume coming from China and India. The FDA expressed concerns that these issues could lead to significantly greater issues, such as market authorization delays, greater danger to patients, and supply chain issues.
The letter states that these manufacturers remain obligated to carefully check third-party labs and verify all test results prior to submission. While acknowledging that the action could create a challenge, the FDA stated its expectations were for device firms to properly identify testing results that appear to be inconsistent with previously verified information regarding the devices.1
“While the FDA encourages device firms to partner with third-party labs that have been voluntarily accredited under the Accreditation Scheme for Conformity Assessment (ASCA) program, doing such does not substitute for conducting an independent assessment of all third-party data,” the agency said in a statement. “To protect patients and healthcare providers from substandard products and ensure that US patients have access to safe, effective, and high-quality medical devices, the FDA is pursuing various actions to identify and confront data integrity violations including through our Bioresearch Monitoring Program. At the same time, the FDA is calling upon the medical device industry to be vigilant and proactive about ensuring the integrity of all data they include in medical device submissions.”
According to a November 2023 paper published in JAMA Ophthalmology, a group of researchers discovered that artificial intelligence (AI) has the ability to fabricate data by producing semi-random datasets as a means of meeting goals. As part of their study, the authors utilized GPT-4 with the aim of getting it provide data accompanying a conclusion that lacked scientific evidence. For this study, they aimed to have AI generate convincing data stating that penetrating keratoplasty had worse patient outcomes than deep anterior lamellar keratoplasty for sufferers of keratoconus, a condition effecting vision.2,3
In an interview with Medical Device Network, study co-author Andrea Taloni, MD, explained that their findings were only a small part of what Chat-GPT is capable of.
“The reality is that if someone was to create a fake data set, it is unlikely that they would use just one prompt,” she told Medical Device Network. “There is this sort of tug of war between those who will inevitably try to generate fake data and all of our defensive mechanisms, including statistical tests and possibly software trained by AI.”
Since 2022, the FDA has offered medical device manufacturers what it calls the 510(k) Third Party Review Program. As part of the program, accredited Third Party Review Organizations are given authorization to review specific low-to-moderate risk medical devices. However, the agency makes the final decision regarding recommendations and documentation from these third-party reviews, informing them of a final decision before allowing them to pass on further information to the device manufacturers.4
“The FDA makes the final determination on the 510(k) submission based on the review and recommendation received from the 3P510k Review Organization,” the agency said in a statement explaining the program. “If the 3P510k Review Organization did not appropriately apply the 510(k) decision criteria or if there are substantive review quality issues with their documentation, the FDA may need to re-review all or part of the 510(k) submission. However, the FDA is updating this program to avoid the routine re-review of 510(k) submissions already reviewed by a 3P510k Review Organization.”
References
1. Fraudulent and Unreliable Laboratory Testing Data in Premarket Submissions: FDA Reminds Medical Device Manufacturers to Scrutinize Third-Party-Generated Data. FDA. February 20, 2024. Accessed February 22, 2024.
2. AI’s data fakery is ‘scary’ say researchers, but the problem is already huge. Medical Device Network. December 5, 2023. Accessed February 22, 2024. https://www.medicaldevice-network.com/features/ai-medical-data-fakery-scary-say-researchers-but-problem-already-huge/
3. Large Language Model Advanced Data Analysis Abuse to Create a Fake Data Set in Medical Research. JAMA Network. November 9, 2023. Accessed February 22, 2024. https://jamanetwork.com/journals/jamaophthalmology/article-abstract/2811505
4. 510(k) Third Party Review Program. FDA. August 19, 2022. Accessed February 22, 2024. https://www.fda.gov/medical-devices/premarket-submissions-selecting-and-preparing-correct-submission/510k-third-party-review-program
