Mapping Baseline ChatGPT Usage to Support Drug Development
Despite limitations to its widespread use within healthcare, there is great potential for ChatGPT’s application in drug development.
Image credit: Tierney | stock.adobe.com
Abstract
Background: ChatGPT usage remains limited to administrative and support work among drug developments professionals; however, a small percentage use it for more advanced tasks, such as predicting pharmacodynamics and tracking adverse events.
Methods: We surveyed 127 drug development professionals between August and October 2023, asking respondents about how they use ChatGPT to support their work. Questions gauged usage levels for support ranging from administrative tasks, such as editing and proofing to more advanced tasks that included generating new chemical structures, identifying new drug targets and novel therapy areas, etc. The survey also asked respondents about their organization’s official stance and outlook on ChatGPT.
Results: The survey revealed a diverse demographic, with 63% of the respondents being White, and holding doctorate degrees (43%). About half (49%) were employed in pharmaceutical, biopharmaceutical, or contract research organization, while 31% worked in healthcare. Approximately 46% shared that they started using ChatGPT in early 2023. However, organizational attitudes varied, 52% had no formal stance, while 30% encouraged ChatGPT’s integration in the workplace. Usage varied from basic tasks like monthly outlining (39%) to advanced tasks like predicting drug efficacy monthly (less than 10%). Overall, while there's moderate AI familiarity and adoption, organizational attitudes and usage levels of ChatGPT vary widely.
Conclusion: Most respondents (52%) shared that their organization did not issue a formal stance regarding whether and how ChatGPT should be used, and 81% said that they plan on using ChatGPT to support their work. Approximately 40% of respondents share that they regularly use ChatGPT to carry out supportive tasks, such as creating outlines, editing and organizing written work, and gathering articles. Less than 10% shared that they regularly used ChatGPT to support more advanced work, including generating new chemical structures, predicting pharmacodynamics, and monitoring adverse reactions.
Introduction
The application of large language modeling (LLM) has exploded over the past two years, most notably with the release of ChatGPT in November 2022. ChatGPT is an LLM developed by OpenAI, an organization whose purpose is to use machine learning algorithms to create human-like responses.
This is accomplished through the processing of large sets of data to respond to user inputs, leading to improvements of its responses through repetitive feedback.1 ChatGPT has been used successfully in a wide range of industries, including education,2 finance,3 and even medicine.4 In the business space, ChatGPT is gaining momentum where it has the potential to streamline customer service,5,6 assist in market research,6 and consolidate business decisions.7
In medicine, ChatGPT has passed the US medical licensing exam,8 demonstrating the biochemical knowledge equivalent to a third year medical student,9 and has been cited as having the potential to improve clinical workflow.10 However, there are still ongoing limitations to its widespread use within healthcare, stemming from medico-legal complications and harmful or irrelevant results or hallucinations.11
Despite these limitations, the potential for ChatGPT’s application in drug development abounds. For example, ChatGPT can not only be used to assist in every day administrative tasks, but can also be used for predicting and explaining drug interactions; converting molecular formulations of compounds to their simplified molecular input line-entry system (SMILES) notation12 and assisting in basic searches for protein data bank files.12
Furthermore, ChatGPT can be fine-tuned for even more specific purposes to support drug discovery, such as generating potential chemical structures after being trained on the dataset of drug-like molecules12 or formulating reaction arrays for chemical reactions in high throughput experimentation.13
ChatGPT can also simplify the writing process, generate queries for systematic reviews, and help non-native English writers better communicate their scientific ideas.14 Among patients, ChatGPT can be used to assist patient understanding of their medication protocol, potentially improving drug adherence.15 In sum, ChatGPT seems to offer endless possibilities within different realms of drug development.
The US pharmaceutical industry is worth $567 billion in 202216 and disruptive technologies such as ChatGPT are bound to have profound effects within the industry. Understanding the current perceptions and usage of ChatGPT in such a large industry remains critical. However, most of the work currently done on the state of ChatGPT use in the pharmaceutical industry are still theoretical and largely limited to opinion pieces. To our knowledge, this is the first study to quantify the use of ChatGPT within the pharmaceutical industry.
Methods
The survey was administered among employees working in drug development in the U.S. (N=127). Survey respondents comprised representatives from at least 45 unique organizations. The survey contained questions designed to assess the integration of ChatGPT (GPT3.5 and/or GPT-4) into respondents' daily job functions and explore associated challenges and barriers.
ChatGPT usage was assessed using a series of questions about tasks, ranging from administrative tasks such as creating outlines, spell checking, and organizing written work to more sophisticated, tailored usage, such as predicting pharmacokinetics and pharmacodynamics as well as predicting the efficacy and safety of potential drugs. The estimated completion time was approximately 15 minutes. Data collection occurred from August to October 2023.
All methods were performed in accordance with the relevant guidelines and regulations by including a statement in the methods section to this effect. The study was reviewed by and received exemption status from the Tufts University School of Medicine Institutional Review Board.
The survey was distributed among participants from an internal, proprietary distribution list containing the emails of drug development professionals. Additionally, the survey was shared on Tufts Center for the Study of Drug Development (CSDD)’s social media platforms, specifically LinkedIn and the Tufts CSDD website.
Monetary compensation was not offered for completing the survey. However, participants were provided with the opportunity to vote for their preferred charity from a list of options, which included the American Civil Liberties Union Foundation, Southern Poverty Law Center, The Center for Constitutional, Child Foundation, and Black Mamas Matter Alliance. The charity receiving the highest number of votes was awarded a $300 donation. Respondents also had the option to receive a high-level summary of the study results by providing their email address.
Table 1. Respondent Demographics
Results
A total of 127 participants completed the survey. The majority identified as White (63%), followed by Asian (21%), multi-racial (7%), Black (5%), and Latino (4%). Half, 50%, identified as men and 46% as women. The mean age was 50 years, and the average work experience was 21 years.
Educational backgrounds varied, with 43% holding a doctorate's degree, 32% a master's degree, 19% a bachelor's degree, 5% a professional degree, 1% an associate degree, and 1% having completed high school. Organizational sizes were distributed as follows: 45% from small-sized organizations, 31% from large-sized organizations, and 24% from mid-sized organizations.
Approximately 50% of respondents were currently employed in pharmaceutical/biopharmaceutical or a contract research organization; 31% of participants worked in an academic medical center, large healthcare system, private clinical trial investigative site, or community-based clinical trial investigative site; and 19% worked in consulting or were solution providers. Additional demographic information is provided in Table 1 for reference.
Most participants held positions at the manager, senior manager, or director level, comprising 46% of the respondents. Entry level employees accounted for 5% of the respondents. The remaining 49% occupied roles such as C-level executives, owners, senior individual contributors, vice presidents, senior vice presidents, and other positions.
Organizations’ Attitudes Toward ChatGPT
Approximately 52% of respondents shared that their organizations did not provide a formal stance regarding the utilization of ChatGPT. In contrast, 30% of respondents shared that their organizations actively promoted the integration of ChatGPT, while 18% discouraged or prohibited its usage.
Relatedly, when asked about the availability of ChatGPT Plus subscriptions in their workplace, most respondents, 69%, shared that their workplace currently did not provide it and had no plans to do so. When asked about risk management related to ChatGPT, 45% expressed that because they had no intention to use ChatGPT, consequently, they had no plans for risk mitigation.
On the other hand, 17% shared that their organization hired a team of AI specialists tasked with establishing parameters and internal guidelines for its use. Additionally, 10% are offering training to leverage ChatGPT capabilities, while 8% have formed partnerships with OpenAI. See Table 2.
Table 2. Respondent Responses
ChatGPT Usage in Professional Work
Next, among those who said that they are using ChatGPT in their everyday work, we explored for what specific types they were ChatGPT. See Table 3 for a detailed description of the different tasks.
The tasks were categorized into three usage levels, ranging from basic to advanced. Basic tasks included administrative activities such as creating outlines, editing, and proofing reports or manuscripts. Approximately 39% of respondents shared that they used ChatGPT to create outlines for work while 35% shared that they used it for gathering articles and organizing references.
Next, intermediary tasks included slightly more advanced activities such as data management and analysis or explaining novel or difficult concepts. About 39% of respondents shared that they use ChatGPT to help explain difficult to understand concepts; however, less than 20% of respondents shared that they use ChatGPT to handle data.
Lastly, the advanced category contained activities, such as generating new chemical structures and predicting toxic effects of new drugs. Only 14% of respondents shared that they use ChatGPT to identify new drug targets. Less than 10% of respondents shared that they use ChatGPT to assist in the more advanced tasks related to drug development, such as predicting pharmacodynamics, writing code for docking, or monitoring adverse events.
Table 3. Respondents’ Professional work usage
Discussion
Our study provided baseline ChatGPT usage among drug development organizations to understand whether and how organizations are adopting ChatGPT into daily use. Overall, 52% of organizations represented in our survey did not provide any guidance on the utilization of ChatGPT. Most organizations in our sample—close to 70%—shared that they do not plan on formally implementing ChatGPT. Both results suggest a general hesitancy17 toward using ChatGPT across organizations.
Next, we examined individual’s usage and attitudes towards ChatGPT, which was moderately more favorable to the adoption and usage of ChatGPT. For example, close to 40% of respondents shared that they used ChatGPT to handle basic tasks, such as creating outlines, editing, and organizing written work on a regular or monthly basis. When asked about intermediary tasks, such as analysis, management, storage, and retrieval of data, less than 20% of participants shared that they used ChatGPT for these tasks on a monthly basis.
Next, when asked about more advanced tasks, such as generating new chemical structures or predicting toxic effects of drugs, less than 10% of participants shared that they use ChatGPT to support them in these tasks monthly. Most participants shared that they do not plan on using ChatGPT for more advanced tasks.
What these findings suggest is that most organizations and professionals working in drug development—whether in pharmaceutical sponsor companies or academic research centers—are hesitant about incorporating ChatGPT into their workflow. However, the percentage of those in our sample who regularly use ChatGPT to support their work was higher than the average usage rate among active users of ChatGPT in the general American population, suggesting that those working in the field of drug development may be more comfortable using ChatGPT to support their work.17
The study also points to promising potential for future usage in various aspects of drug development, even for the more advanced tasks. For example, identifying new drug targets had the highest level of moderately regular usage—at least a few times a year—among the more advanced tasks.
This finding suggests that although usage of ChatGPT to support robust tasks, such as generating new chemical structures or monitoring adverse drug events, is relatively low compared to less complex tasks, such as editing or performing literature reviews, the potential to scale its usage remains. How drug development companies develop their own datasets for training or create partnerships with OpenAI remain to be seen. Once more robust features are built into ChatGPT and more studies demonstrate its efficacy,18 more drug development professionals may utilize ChatGPT to help them to streamline and perform advanced workflows.
This study has several limitations that should be considered. First, the sample size was relatively small and was only from the United States, which limited the generalizability of the findings. However, data were collected from professionals from biopharmaceutical companies, contract research organizations, and academic institutions. This wide range of industries and views provide a preliminary baseline benchmark of current and future usage of ChatGPT among professionals working in drug development.
Additionally, the study did not evaluate the impact of ChatGPT on actual work outcomes. Future studies may wish to tie different levels of ChatGPT usage to performance metrics—such as study level outcomes; individual work metrics, such as productivity, job satisfaction, and workload—to evaluate its impact on real-world outcomes. Nonetheless, before such studies can be conducted, benchmarks need to be clarified on what constitutes ChatGPT usage in drug development and how AIs will be trained on datasets (i.e. protein data bank files).
In conclusion, the study provided an overview of the baseline usage of ChatGPT among drug development professionals in the United States. While the results from our sample suggest that usage of ChatGPT to support everyday work is still in its infancy, the study also highlights the potential for increased usage of ChatGPT to assist professionals with more advanced tasks.
About the Authors
Jennifer Y. Kim, Tufts Center for the Study of Drug Development (CSDD), Tufts School of Medicine, Boston, MA USA.
Ruby Madison Ford, Tufts Center for the Study of Drug Development (CSDD), Tufts School of Medicine, Boston, MA USA.
Zhida Shang, McGill University, School of Medicine, Montreal, Canada.
References
1. OpenAI Platform. Accessed May 15, 2024. https://platform.openai.com
2. Baidoo-Anu D, Owusu Ansah L. Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. Published online January 25, 2023. doi:10.2139/ssrn.4337484
3. George AS, George ASH, Martin ASG. A Review of ChatGPT AI’s Impact on Several Business Sectors. Partners Universal International Innovation Journal (PUIIJ). 2023;01(01):9-23. doi:10.5281/zenodo.7644359
4. Eysenbach G. The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation With ChatGPT and a Call for Papers. JMIR Medical Education. 2023;9(1):e46885. doi:10.2196/46885
5. Sudirjo F, Diantoro K, Al-Gasawneh JA, Azzaakiyyah HK, Ausat AMA. Application of ChatGPT in Improving Customer Sentiment Analysis for Businesses. Jurnal Teknologi Dan Sistem Informasi Bisnis. 2023;5(3):283-288. doi:10.47233/jteksis.v5i3.871
6. Rivas P, Zhao L. Marketing with ChatGPT: Navigating the Ethical Terrain of GPT-Based Chatbot Technology. AI. 2023;4(2):375-384. doi:10.3390/ai4020019
7. Generative AI for Business Decision-Making: A Case of ChatGPT | Management Science and Business Decisions. Published online July 1, 2023. Accessed May 15, 2024. https://publish.thescienceinsight.com/index.php/msbd/article/view/63
8. Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198. doi:10.1371/journal.pdig.0000198
9. Ghosh A, Bir A. Evaluating ChatGPT’s Ability to Solve Higher-Order Questions on the Competency-Based Medical Education Curriculum in Medical Biochemistry. Cureus. 15(4):e37023. doi:10.7759/cureus.37023
10. Rao A, Pang M, Kim J, et al. Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study. Journal of Medical Internet Research. 2023;25(1):e48659. doi:10.2196/48659
11. Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell. 2023;6:1169595. doi:10.3389/frai.2023.1169595
12. Sharma G, Thakur A. ChatGPT in Drug Discovery.; 2023. doi:10.26434/chemrxiv-2023-qgs3k
13. Mahjour B, Hoffstadt J, Cernak T. Designing Chemical Reaction Arrays using phactor and ChatGPT. Published online February 28, 2023. doi:10.26434/chemrxiv-2023-2tfdv
14. Sallam M. ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns. Healthcare (Basel). 2023;11(6):887. doi:10.3390/healthcare11060887
15. Juhi A, Pipil N, Santra S, Mondal S, Behera JK, Mondal H. The Capability of ChatGPT in Predicting and Explaining Common Drug-Drug Interactions. Cureus. 15(3):e36272. doi:10.7759/cureus.36272
16. Insights10. North America Pharmaceutical Market Analysis Report [2022-30] I Insights10. Accessed May 15, 2024. https://www.insights10.com/report/north-america-pharmaceutical-market-analysis/www.insights10.com/report/north-america-pharmaceutical-market-analysis/
17. Gelles-Watnick EP and R. Most Americans haven’t used ChatGPT; few think it will have a major impact on their job. Pew Research Center. Published August 28, 2023. Accessed May 15, 2024. https://www.pewresearch.org/short-reads/2023/08/28/most-americans-havent-used-chatgpt-few-think-it-will-have-a-major-impact-on-their-job/
18. Ueda D, Walston SL, Matsumoto T, Deguchi R, Tatekawa H, Miki Y. Evaluating GPT-4-based ChatGPT’s clinical potential on the NEJM quiz. BMC Digital Health. 2024;2(1):4. doi:10.1186/s44247-023-00058-5
