Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 8
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Towards safer chatbots: Automated policy compliance evaluation of custom GPTs

  • Rodriguez Torrado, David
  • Seymour, William
  • Álamo, Jose M. Del
  • Such, Jose
User-configured chatbots built on top of large language models are increasingly available through centralized marketplaces such as OpenAI’s GPT Store. While these platforms enforce usage policies intended to prevent harmful or inappropriate behavior, the scale and opacity of customized chatbots make systematic policy enforcement challenging. As a result, policy-violating chatbots continue to remain publicly accessible despite existing review processes. This paper presents a fully automated method for evaluating the compliance of Custom GPTs with its marketplace usage policy using black-box interaction. The method combines large-scale GPT discovery, policy-driven red-teaming prompts, and automated compliance assessment using an LLM-as-a-judge. We focus on three policy-relevant domains explicitly addressed in OpenAI’s usage policies: Romantic, Cybersecurity, and Academic GPTs. We validate our compliance assessment component against a human-annotated ground-truth dataset, achieving an F1 score of 0.975 for binary policy violation detection. We then apply the method in a large-scale empirical study of 782 Custom GPTs retrieved from the GPT Store. The results show that 58.7% of the evaluated GPTs exhibit at least one policy-violating response, with substantial variation across policy domains. A comparison with the base models (GPT-4 and GPT-4o) indicates that most violations originate from model-level behavior, with customization tending to amplify these tendencies rather than create new failure modes. Our findings reveal limitations in current review mechanisms for user-configured chatbots and demonstrate the feasibility of scalable, behavior-based policy compliance evaluation., The work of David Rodriguez was partially supported by the CEDAR project, funded by the Horizon Europe research program (2021–2027) under grant agreement no. 101135577. The work of Jose M. Del Alamo was partially supported by the Re-InitS project, funded by the Plan Estatal de Investigación Científica y Técnica y de Innovación 2024–2027 (Ministerio de Ciencia e Investigación (Spain) - MCIN/AEI/10.13039/501100011033) under grant agreement PID2024-155230OB-C43. The work of Jose Such was partially supported by Grant PID2023-151536OB-100 funded by MICIU/AEI/10.13039/501100011033 and by ERDF/EU, and by a PROMETEO 2024 (CIPROM/2023/23) grant of Conselleria d’Educació, Universitats i Ocupació (Generalitat Valenciana) ., Peer reviewed
Archivo Digital UPM
Publicación Artículo científico (JournalArticle)

Insights and caveats from mining local and global temporal motifs in cryptocurrency transaction networks

  • Cuadrado Latasa, Félix|||0000-0002-5745-1609
  • Arnold, Naomi A
  • Zhong, Peijie
  • Ba, Cheick Tidiane
  • Steer, Ben
  • Mondragon, Raul
  • Cuadrado Latasa, Félix|||0000-0002-5745-1609
  • Lambiotte, Renaud
  • Clegg, Richard G
Archivo Digital UPM
Publicación Tesis doctoral (DoctoralThesis)

Contributions to the Automated Assessment of Mobile Applications’ Compliance with Privacy and Data Protection Requirements

Archivo Digital UPM
Publicación Artículo científico (JournalArticle)

A formal model for reliable data acquisition and control in legacy critical infrastructures

  • Blanco Sánchez, José Miguel|||0000-0001-9460-8540
  • Álamo Ramiro, José María del|||0000-0002-6513-0303
  • Dueñas López, Juan Carlos|||0000-0001-9689-4798
  • Cuadrado Latasa, Félix|||0000-0002-5745-1609
Archivo Digital UPM
Publicación Artículo científico (JournalArticle)

GPT vs human legal texts annotations: a comparative study with privacy policies

  • Cevallos Salas, David|||0000-0002-3098-3090
  • Estrada Jiménez, José|||0009-0001-3931-9532
  • Guamán Loachamín, Danny Santiago|||0000-0003-2794-3079
  • Rodriguez Torrado, David|||0000-0002-0911-4608
  • Álamo Ramiro, José María del|||0000-0002-6513-0303
Archivo Digital UPM
Publicación Artículo científico (JournalArticle)

Detecting suspicious activity in the NFT ecosystem using temporal graph analysis

  • Sliti, Wassim|||0000-0003-3867-8792
  • Cuadrado Latasa, Félix|||0000-0002-5745-1609
  • Campos Hernáez, Leandro
  • Dueñas López, Juan Carlos|||0000-0001-9689-4798
Archivo Digital UPM
Publicación Artículo científico (JournalArticle)

Knowledge Graphs and Critical Infrastructures: Friends or Foes?

  • Blanco Sánchez, José Miguel|||0000-0001-9460-8540
  • Cuadrado Latasa, Félix|||0000-0002-5745-1609
  • Álamo Ramiro, José María del|||0000-0002-6513-0303
  • Dueñas López, Juan Carlos|||0000-0001-9689-4798
Archivo Digital UPM
Publicación Artículo científico (JournalArticle)

An adaptive algorithm for the detection of massive events in urban metro networks

  • Fernández Getino, Ángel|||0009-0002-1419-3096
  • Andión Jiménez, Javier|||0000-0001-5683-6403
  • Blanco Sánchez, José Miguel|||0000-0001-9460-8540
  • Dueñas Lopez, Juan Carlo|||0000-0001-9689-4798