- Igualde-Sáez, Arnau|||0009-0009-0824-6957
- Rhomrassi-Amhar, Lamyae
- Hoyas, Sergio|||0000-0002-8458-7288
- García Sabater, José Pedro|||0000-0002-9097-2852
- Fullana Alfonso, Màrius Josep|||0000-0002-0196-7876
- Conejero, J. Alberto|||0000-0003-3681-7533
- Ahsini Ouariaghli, Yusef
- Vinuesa, Ricardo
[EN] Multimodal Large Language Models (MLLMs) promise advanced vision-language
capabilities, yet their effectiveness in visually presented mathematics remains
underexplored. This paper analyses the development and evaluation of MLLMs for
mathematical problem-solving, focusing on diagrams, multilingual text, and symbolic notation. The computational demands of evaluating these large-scale models
across multilingual datasets necessitate high-performance computing infrastructure,
as systematic benchmarking of state-of-the-art MLLMs requires distributed processing of thousands of inference requests and parallel evaluation across multiple model
architectures. We then assess several models-including GPT-4o, Pixtral, Qwen-VL,
Llama 3.2 Vision variants, and Gemini 2.0 Flash-in a multilingual Kangaroo-style
benchmark spanning English, French, Spanish, and Catalan. Our experiments reveal
four key findings. First, overall accuracy remains moderate across geometry, visual
algebra, logic, patterns, and combinatorics: no single model excels in every topic.
Second, whilst most models see improved accuracy with questions that do not have
images, the gain is often limited; performance for some remains nearly unchanged
without visual input, indicating underutilisation of diagrammatic information. Third,
substantial variation exists across languages and difficulty levels: models frequently
handle easier items but struggle with advanced geometry and combinatorial reasoning. Notably, Gemini 2.0 Flash achieves the highest accuracy on image-based tasks,
followed by Qwen-VL 2.5 72B and GPT-4o, though none approach human-level
performance. Fourth, a complementary analysis aimed at distinguishing whether
models reason or simply recite reveals that Gemini and GPT-4o stand out for their
structured reasoning and consistent accuracy. In contrast, Pixtral and Llama exhibit
less consistent reasoning, often defaulting to heuristics or randomness when unable
to align their outputs with the given answer options., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. AIS
acknowledges the financial support from the European Union, Next GenerationEU, under the public subsidies of the Programa Investigo , within the framework of the Recovery, Transformation, and Resilience Plan (Reference INV/2023/25). SH acknowledges the funding of project PID2021-128676OB-I00
and JAC acknowledge funding of project PID2022-138860NB-I00 by FEDER/MCIN, funded by MCIN/
AEI/10.13039/501100011033 and by ERDF A Way of Making Europe by the European Union. RV
acknowledges the funding provided by Digital Futures, in their Demonstrator-project programme. MJFA
acknowledges the financial support of the Generalitat Valenciana Project grant CIAICO/2022/252. We
also thank funding for open access charge: CRUE-Universitat Politècnica de València.