RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación
Artículo científico (JournalArticle)
Beyond Surface Metrics: Modeling Abstractivity in Summarization via Action-Based Spectrums
[EN] Traditional views of abstractive summarization treat it as a free-form rephrasing of the source text, yet this notion is vague and difficult to measure; existing evaluations rely on surface-level metrics -such as novel $n$ -gram counts, extractive fragment coverage, or extractive fragment density- that obscure the stylistic nuances that distinguish summaries along an abstractivity spectrum. We therefore propose a fine-grained, continuous characterization that does not claim to unify extractive and abstractive summarization into a single continuous dimension, but rather presents them as two endpoints of a spectrum defined by the frequency of text-transformation actions (sentence reduction, combination, paraphrasing, reordering, etc.) drawn from the foundational summarization literature. To operationalize this view, we introduce CLAsum, a human-annotated dataset comprising 600 unique samples; these arise from 200 document-summary pairs sourced from CNN/DailyMail and XSum datasets, each annotated by three independent annotators who assign one of five perceived abstractivity levels and record the prevalence of seven abstractivity-inducing actions per pair. Our analysis shows a strong Pearson's correlation of 0.6 between the overall frequency of these actions and human perception of abstractivity, which validates the spectrum. Moreover, automatic prediction of the resulting Abstractivity-Inducing Features (AIFs) captures signals that existing metrics miss and yields significant performance gains of up to 9.1% in abstractivity-level measurement on the evaluated regression methods. This work delivers both conceptual clarity and practical tools for diagnosing and informing the design of controllable, style-aware summarization systems capable of producing summaries with precisely specified levels of abstractivity, thereby advancing the field toward truly user-tailored, task-adaptive summarization., This work was supported in part by MICIU/AEI/10.13039/501100011033 and FEDER, EU, under Project PID2024-155948OB-C55; and in part by Spanish Ministerio de Universidades for University Teacher Training under Grant FPU21/05288.