Repositorio Digital de la UPF
Publication
Artículo científico (JournalArticle)
ChatGPT Search as a tool for scholarly tasks: evolution or devolution?
Includes supplementary materials for the online appendix., ChatGPT Search was launched on October 31 by OpenAI as a new AI-powered search engine. Among its features, it stands out for its ability to retrieve information from various online sources, including scholarly databases, which potentially allows the use of this tool for academic tasks, both quantitative and qualitative. To test its features, five academic tasks are designed: two quantitative (collecting hit count estimates from Google Search and scraping bibliometric indicators from ResearchGate); two qualitative tasks (performing a narrative synthesis of an academic topic and generating a brief academic author profile), and a mixed task (identifying, collecting and describing a list of publications from Google Scholar Profiles). The results show the inability of ChatGPT Search to conduct quantitative tasks correctly, fabricating the results (hallucination). Qualitative tasks are performed with better results; however, errors are detected, which prevent recommending the tool without manual analysis and refinement. Finally, the ability to generate links to scientific publications can open up competition among academic sites to be mentioned in the ChatGPT Search responses, giving rise to Academic Generative Engine Optimization (A-GEO)., Study supported by project “Publicized research: new indicators to measure the effects of timely marketing on scholarly evaluation (PRIME TIME)” (PID2022-142569NA-I00), funded by MCIN/AEI/10.13039/501100011033,and by “ERDF A way of making Europe”, and by Project “Parameters and strategies to increase the relevance of media and digital communication in society: curation, visualiza-tion and visibility (CUVICOM)”, funded by MICIN/AEI/PID2021-123579OB-I00 and by FEDER/EU.