RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación
Artículo científico (JournalArticle)
There are significant differences among artificial intelligence large language models when answering scientific questions
[EN] Introduction This study investigates the efficacy of large language models (LLMs) for generating accurate scientific responses through a comparative evaluation of five prominent free models: Claude 3.5 Sonnet, Gemini, ChatGPT 4o, Mistral Large 2, and Llama 3.1 70B.Methods Sixteen expert scientific reviewers assessed these models in terms of depth, accuracy, relevance, and clarity.Results Claude 3.5 Sonnet emerged as the highest scoring model, followed by Gemini, with notable variability among the other models. Additionally, retrieval-augmented generation (RAG) techniques were applied to improve LLM performance, and prompts were refined to improve answers. The results indicate that although LLMs such as Claude 3.5 Sonnet have potential for scientific tasks, other models may require more development or additional prompt engineering to reach comparable accuracy. Reviewers' perceptions of artificial intelligence (AI) utility and trustworthiness showed a positive shift after evaluation. However, ethical concerns, particularly with respect to transparency and disclosure, remained consistent.Discussion The study highlights the need for structured frameworks for evaluating LLMs and ethical considerations essential for responsible AI integration in scientific research. These findings should be interpreted with caution, as the limited sample size and domain-specific focus of the exam questions restrict the generalizability of the results., The author(s) declare that financial support was received for the research and/or publication of this article. This work was supported by RTI2018-096724-B-C22, PID2021-125188OB-C32 and TED2021-129932B-C21 projects funded by Spanish Ministry of Economy and Competitiveness and CB12/03/30038 (CIBER Fisiopatologia de la Obesidad y la Nutricion, CIBERobn, Instituto de Salud Carlos III). This research was also funded by Generalitat Valenciana (grant number PROMETEO/2021/059) and Agencia Valenciana de la Innovacion (grant number INNEST/2022/103). F & Aacute;-M were supported by "Margarita Salas" fellowship from the Spanish Ministry of Universities. EB-C was supported by the "Requalification for university teachers grant" from the Spanish Ministry of Universities and European Union Next Generation program.
Proyecto:
Agencia Estatal de Investigación / Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020 RTI2018-096724-B-C22
,
Agencia Estatal de Investigación / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2021-125188OB-C32
,
Generalitat Valenciana PROMETEO/2021/059
,
Instituto de Salud Carlos III CB12/03/30038
,
Ministerio de Economía y Competitividad TED2021-129932B-C21
,
Agència Valenciana de la Innovació INNEST/2022/103