Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 3
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Assessment of fine-tuned large language models for real-world chemistry and material science applications

  • Van Herck, Joren
  • Gil Matellanes, María Victoria
  • Jablonka, Kevin Maik
  • Abrudan, Alex
  • Anker, Andy S
  • Asgari, Mehrdad
  • Blaiszik, Ben
  • Buffo, Antonio
  • Choudhury, Leander
  • Corminboeuf, Clemence
  • Daglar, Hilal
  • Elahi, Amir Mohammad
  • Foster, Ian T
  • Garcia, Susana
  • Garvin, Matthew
  • Godin, Guillaume
  • Good, Lydia L
  • Gu, Jianan
  • Xiao Hu, Noémie
  • Jin, Xin
  • Junkers, Tanja
  • Keskin, Seda
  • Knowles, Tuomas P J
  • Laplaza, Ruben
  • Lessona, Michele
  • Majumdar, Sauradeep
  • Mashhadimoslem, Hossein
  • McIntosh, Ruaraidh D
  • Moosavi, Seyed Mohamad
  • Mouriño, Beatriz
  • Nerli, Francesca
  • Pevida García, Covadonga
  • Poudineh, Neda
  • Rajabi-Kochi, Mahyar
  • Saar, Kadi L
  • Hooriabad Saboor, Fahimeh
  • Sagharichiha, Morteza
  • Schmidt, K J
  • Shi, Jiale
  • Simone, Elena
  • Svatunek, Dennis
  • Taddei, Marco
  • Tetko, Igor
  • Tolnai, Domonkos
  • Vahdatifar, Sahar
  • Whitmer, Jonathan
  • Wieland, D C Florian
  • Willumeit-Römer, Regine
  • Züttel, Andreas
  • Smit, Berend
The datasets and Jupyter Notebooks used in this work are available at https://github.com/JorenBE/GPT-Challenge., The current generation of large language models (LLMs) has limited chemical knowledge. Recently, it has been shown that these LLMs can learn and predict chemical properties through fine-tuning. Using natural language to train machine learning models opens doors to a wider chemical audience, as field-specific featurization techniques can be omitted. In this work, we explore the potential and limitations of this approach. We studied the performance of fine-tuning three open-source LLMs (GPT-J-6B, Llama-3.1-8B, and Mistral-7B) for a range of different chemical questions. We benchmark their performances against "traditional" machine learning models and find that, in most cases, the fine-tuning approach is superior for a simple classification problem. Depending on the size of the dataset and the type of questions, we also successfully address more sophisticated problems. The most important conclusions of this work are that, for all datasets considered, their conversion into an LLM fine-tuning training set is straightforward and that fine-tuning with even relatively small datasets leads to predictive models. These results suggest that the systematic use of LLMs to guide experiments and simulations will be a powerful technique in any research study, significantly reducing unnecessary experiments or computations., The research of J. V. H., and B. S. is supported by the Swiss Science Foundation through a Project Funding (214872) and Advanced Grant (216165). M. V. G. and C. P. gratefully acknowledge financial support from the Spanish Agencia Estatal de Investigación (AEI) through Grants TED2021-131693B-I00 (M. V. G. and C. P.) and CNS2022-135474 (M. V. G.), funded by MICIU/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR. M. V. G. acknowledges support from the Spanish National Research Council (CSIC) through Programme for internationalization i-LINK 2023 (Project ILINK23047). M. V. G. acknowledges the access granted by the Galician Supercomputing Center (CESGA) to the FinisTerrae III supercomputer, funded by the Spanish Ministry of Science and Innovation, the Galician Government, and the European Regional Development Fund (ERDF), and the access granted by the CSIC to the Drago supercomputer. Parts of the work of K. M. J. were supported by the Carl Zeiss Foundation. S. G., M. G., N. P., B. S., and J. V. H. are partly supported by the USorb-DAC Project through a grant from The Grantham Foundation for the Protection of the Environment to RMI's climate tech accelerator program, Third Derivative. The work of A. S. A. is supported by Novo Nordisk Foundation grant NNF23OC0081359. S. M. M. and M. R. K. work is partly supported by grant number DSI-CGY3R1P16 from the Data Sciences Institute at the University of Toronto. M. A. expresses gratitude to the European Research Council (ERC) for evaluating the project with the reference number 101106377 titled “CLARIFIER” and accepting it for funding under the HORIZON TMA MSCA Postdoctoral Fellowships – European Fellowships. Furthermore, M. A. acknowledges the funding by UK Research and Innovation (UKRI) under the UK government's Horizon Europe funding guarantee (EP/Y023447/1; organization reference:101106377). L. L. G., A. A., and T. P. J. K. gratefully acknowledge funding from the European Research Council under the European Union's Horizon 2020 research and innovation program through the ERC grant DiProPhys (agreement ID 101001615) (L. L. G., A. A., T. P. J. K.). The National Institutes of Health Oxford-Cambridge Scholars Program (L. L. G.), the Cambridge Trust's Cambridge International Scholarship (L. L. G.), the Intramural Research Program of the National Institute of Diabetes and Digestive and Kidney Diseases at the National Institutes of Health (L. L. G.). The European Research Council under the European Union's Seventh Framework Programme (FP7/2007–2013; T. P. J. K.) and the Frances and Augustus Newman Foundation (T. P. J. K.). K. L. S. acknowledges funding from the Schmidt Science Fellowship in partnership with the Rhodes Trust and from St. John's College Research Fellowship programme. F. N. and M. T. thank the Italian MUR for provision of funding through the PRIN 2020 Project doMino (ref 2020P9KBKZ). The research of N. X. H. was supported by the NCCR MARVEL, a National Centre of Competence in Research, funded by the Swiss National Science Foundation (grant number 205602). B. M. acknowledges the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement no. 945363. E. S. acknowledges the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement no. 949229, CryForm)., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

From text to insight: large language models for chemical data extraction

  • Schilling-Wilhelmi, Mara
  • Ríos García, Martiño
  • Shabih, Sherjeel
  • Gil Matellanes, María Victoria
  • Miret, Santiago
  • Koch, Christoph T
  • Márquez, José A
  • Jablonka, Kevin Maik
The online book is available at https://matextract.pub for which the source code is available at https://github.com/lamalab-org/matextract-book and archived at https://doi.org/10.5281/zenodo.14249541., The vast majority of chemical knowledge exists in unstructured natural language, yet structured data is crucial for innovative and systematic materials design. Traditionally, the field has relied on manual curation and partial automation for data extraction for specific use cases. The advent of large language models (LLMs) represents a significant shift, potentially enabling non-experts to extract structured, actionable data from unstructured text efficiently. While applying LLMs to chemical and materials science data extraction presents unique challenges, domain knowledge offers opportunities to guide and validate LLM outputs. This tutorial review provides a comprehensive overview of LLM-based structured data extraction in chemistry, synthesizing current knowledge and outlining future directions. We address the lack of standardized guidelines and present frameworks for leveraging the synergy between LLMs and chemical expertise. This work serves as a foundational resource for researchers aiming to harness LLMs for data-driven chemical research. The insights presented here could significantly enhance how researchers across chemical disciplines access and utilize scientific information, potentially accelerating the development of novel compounds and materials for critical societal needs., The research of M. S.-W. and K. M. J. was supported by the Carl-Zeiss Foundation as well as Intel and Merck via the AWASES programme. S. S., C. T. K., J. A. M., and K. M. J. are part of the NFDI consortium FAIRmat funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project 460197019. M. R. G. and M. V. G. acknowledge financial support from the Spanish Agencia Estatal de Investigación (AEI) through grants TED2021-131693B-I00 and CNS2022-135474, funded by MICIU/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR, and support from the Spanish National Research Council (CSIC) through Programme for internationalization i-LINK 2023 (Project ILINK23047), Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists

  • Mirza, Adrian
  • Alampara, Nawaf
  • Kunchapu, Sreekanth
  • Ríos-García, Martiño
  • Emoekabu, Benedict
  • Krishnan, Aswanth
  • Gupta, Tanya
  • Schilling-Wilhelmi, Mara
  • Okereke, Macjonathan
  • Aneesh, Anagha
  • Asgari, Mehrdad
  • Eberhardt, Juliane
  • Elahi, Amir Mohammad
  • Elbeheiry, Hani M.
  • Gil Matellanes, María Victoria
  • Glaubitz, Christina
  • Greiner, Maximilian
  • Holick, Caroline T.
  • Hoffmann, Tim
  • Ibrahim, Abdelrahman
  • Klepsch, Lea C.
  • Köster, Yannik
  • Kreth, Fabian Alexander
  • Meyer, Jakob
  • Miret, Santiago
  • Peschel, Jan Matthias
  • Ringleb, Michael
  • Roesner, Nicole C.
  • Schreiber, Johanna
  • Schubert, Ulrich S.
  • Stafast, Leanne M.
  • Wonanke, A. D.Dinga
  • Pieler, Michael
  • Schwaller, Philippe
  • Jablonka, Kevin Maik
Large language models (LLMs) have gained widespread interest owing to their ability to process human language and perform tasks on which they have not been explicitly trained. However, we possess only a limited systematic understanding of the chemical capabilities of LLMs, which would be required to improve models and mitigate potential harm. Here we introduce ChemBench, an automated framework for evaluating the chemical knowledge and reasoning abilities of state-of-the-art LLMs against the expertise of chemists. We curated more than 2,700 question–answer pairs, evaluated leading open- and closed-source LLMs and found that the best models, on average, outperformed the best human chemists in our study. However, the models struggle with some basic tasks and provide overconfident predictions. These findings reveal LLMs’ impressive chemical capabilities while emphasizing the need for further research to improve their safety and usefulness. They also suggest adapting chemistry education and show the value of benchmarking frameworks for evaluating LLMs in specific domains. (Figure presented.), This work was supported by the Carl Zeiss Foundation, and a ‘Talent Fund’ of the ‘Life’ profile line of the Friedrich Schiller University Jena. In addition, M.S.-W.’s work was supported by Intel and Merck via the AWASES programme. Parts of A.M.’s work were supported as part of the ‘SOL-AI’ project funded by the Helmholtz Foundation model initiative. K.M.J. is part of the NFDI consortium FAIRmat funded by the Deutsche Forschungsgemeinschaft (the German Research Foundation) project no. 460197019. K.M.J. thanks FutureHouse (a non-profit research organization supported by the generosity of Eric and Wendy Schmidt) for supporting PaperQA2 runs via access to the API. We also thank Stability.AI for the access to its HPC cluster. M.R.-G. and M.V.G. acknowledge financial support from the Spanish Agencia Estatal de Investigación (AEI) through grants TED2021-131693B-I00 and CNS2022-135474, funded by Ministerio de Ciencia, Innovación y Universidades (MICIU)/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR. M.V.G. acknowledges support from the Spanish National Research Council through the Programme for internationalization i-LINK 2023 (project no. ILINK23047). A.A. gratefully acknowledges financial support for this research by the Fulbright US Student Programme, which is sponsored by the US Department of State and German-American Fulbright Commission. Its contents are solely the responsibility of the author and do not necessarily represent the official views of the Fulbright Programme, the Government of the USA or the German-American Fulbright Commission. M.A. expresses gratitude to the European Research Council for evaluating the project with the reference no. 101106377 titled ‘CLARIFIER’, and accepting it for funding under the HORIZON TMA MSCA Postdoctoral Fellowships—European Fellowships. Furthermore, M.A. acknowledges the funding provided by UK Research and Innovation under the UK government’s Horizon Europe funding guarantee (grant reference EP/Y023447/1; organization reference 101106377). M.R. and U.S.S. thank the ‘Deutsche Forschungsgemeinschaft’ for funding under the regime of the priority programme SPP 2363 ‘Utilization and Development of Machine Learning for Molecular Applications—Molecular Machine Learning’ (SCHU 1229/63-1; project no. 497115849). A.D.D.W. acknowledges funding from the European Union Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement no. 101107360. P.S. acknowledges support from the National Centre of Competence in Research Catalysis (grant no. 225147), a National Centre of Competence in Research grant funded by the Swiss National Science Foundation. In addition, we thank the OpenBioML.org community and their ChemNLP project team for valuable discussions. Moreover, we thank P. Márquez for discussions and support and J. Kimmig for feedback on the web app. In addition, we acknowledge support from S. Kumar with an initial prototype of the web app. We thank B. Smit for feedback on an early version of the manuscript., Peer reviewed