TARU PUBLICATIONS
Journal of Statistics and Management Systems cover

Journal of Statistics and Management Systems

(Since 1998)
Open Access ·Peer-reviewed·ISSN (Online): 2169-0014·ISSN (Print): 0972-0510

Indexed in ABDC Australia

Powered by:Powered by

Monthly Journal: Publishes peer-reviewed aticles on theoretical and applied statistics and management systems, expoloring industrial statistics, actuarial and decision sciences.

Issues up to 2022 co-published with and available at:Taylor & Francis Online
submissions@tarupublications.com
Taru Online First Current IssueSpecial IssuesBrowse all Issues
Journal of Statistics and Management Systems
Open Access Research Article

Comparative analysis of medical knowledge and inferential capabilities of modern LLMs

* ,

* Corresponding author · click or hover a name for details

pp. 1713–1722Vol. 27Issue 8November 2024DOI: 10.47974/JSMS-1403XML
Received:
21 Jun 2024
Published Online:
18 Dec 2024
Article type:
Research Article
Language:
EN
Article no.:
JSMS-1403
Pages:
1713–1722

Abstract

This paper presents an evaluation of four large language models (LLMs)—GPT-4o, Claude-3.5-Sonnet, Gemini-1.5-Pro, and Llama-3.1-405B-T based on word problems that were formulated to check LLMs level of “understanding” of medical knowledge and their “inferential capabilities”. We constructed a small synthetic dataset by providing pattern to LLM and then asking LLM to generate few word problems related to heartbeat, and evaluated responses of different LLMs for each word problem. Models were evaluated based on their robustness to changing scenarios. Moreover, their responses were compared in order to understand their level of learning about medical knowledge. Consistency of models was checked by providing them answer choices in random order. Then models’ answers were compared with their previous solutions. It was found that all the LLMs involved in the study showed inconsistent behavior. The study also reveals that Gemini and LLama did not show required resilience to changing input. GPT-4o and Claude gave similar answers for most of the queries and showed robustness to changing inputs. The study reveals very interesting insights about strengths and weaknesses of each of LLM and can be very helpful for healthcare professionals in deciding level of adoption of modern artificial intelligence (AI) technologies for identification of heartbeat related disorders like Arrhythmia. 

Keywords

Large language models (LLMs)AI technologiesHealthcareReasoningHeartbeatArrhythmia

Subject Classifications

Primary 00A05Secondary 68T50

References

[1] I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar, “GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models,” Oct. 07 (2024), arXiv: arXiv:2410.05229. doi: 10.48550/arXiv.2410.05229.
[2] K. Cobbe, A. S. Patel, J. S. Brown, and L. D. Miller, “Training verifiers to solve math word problems,” arXiv, Nov. 18 (2021), arXiv:2110.14168. doi: 10.48550/arXiv.2110.14168.
[3] N. Dziri, S. J. Lee, P. M. Sharma, and R. T. Zhang, “Faith and fate: Limits of transformers on compositionality,” arXiv, Oct. 31 (2023), arXiv:2305.18654. doi: 10.48550/arXiv.2305.18654.
[4] OpenAI, “GPT-4 technical report,” arXiv, Mar. 4 (2024), arXiv:2303.08774. doi: 10.48550/arXiv.2303.08774.
[5] “GPT-4o,” Wikipedia. Accessed: Dec. 15 (2024). [Online]. Available: https://en.wikipedia.org/w/index.php?title=GPT-4o&oldid= 1262551536
[6] “Introducing Claude 3.5 Sonnet \ Anthropic.” Accessed: Dec. 15 (2024). [Online]. Available: https://www.anthropic.com/news/claude-3-5-sonnet
[7] “Gemini (language model),” Wikipedia. Accessed: Dec. 15 (2024). [Online]. Available: https://en.wikipedia.org/w/index.php?title=Gemini_(language_model)&oldid=1263935127
[8] “Gemini Pro - Google DeepMind.” Accessed: Dec. 15 (2024). [Online]. Available: https://deepmind.google/technologies/gemini/pro/
[9] “Llama (language model),” Wikipedia. Accessed: Dec. 15 (2024). [Online]. Available: https://en.wikipedia.org/w/index.php?title=Llama_(language_model)&oldid=1264032930
[10] “Open-source artificial intelligence,” Wikipedia. Accessed: Dec. 15 (2024). [Online]. Available: https://en.wikipedia.org/w/index.php?title=Open-source_artificial_intelligence&oldid=1263322594
[11] “Quora,” Wikipedia. Accessed: Dec. 15 (2024). [Online]. Available: https://en.wikipedia.org/w/index.php?title=Quora&oldid=1262150310#Poe
[12] M. Sallam, “ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns,” Healthc. Basel Switz., vol. 11, no. 6, p. 887, Mar. (2023), doi: 10.3390/healthcare11060887.
[13] M. Cascella, J. Montomoli, V. Bellini, and E. Bignami, “Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research Scenarios,” J. Med. Syst., vol. 47, no. 1, p. 33 (2023), doi: 10.1007/s10916-023-01925-4.
[14] Y. Li, Z. Li, K. Zhang, R. Dan, S. Jiang, and Y. Zhang, “ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge,” Jun. 24 (2023), arXiv: arXiv:2303.14070. doi: 10.48550/arXiv.2303.14070.

Views: 201Downloads: 107Citations: 0