A significant milestone has been reached in the TrustLLM project with the successful training of two baseline language models, marking a major step forward in the development of efficient, trustworthy multilingual AI systems. The foundational work, carried out under Work Package 6 (WP6) – Efficient LLM Training and Usage, was led by Forschungszentrum Jülich (FZJ). Part of this work package is also the coordination of measures between the work packages.
The baseline models include a 2.6 billion parameter model, trained on MareNostrum 5 at the Barcelona Supercomputing Centre (BSC), and a larger 7.8 billion parameter model, which leveraged computing resources from both MareNostrum 5 and the JUWELS Booster at Jülich Supercomputing Centre (JSC). These models represent the technical groundwork upon which further innovations in the TrustLLM ecosystem will be built.
Central to the training process were two advanced pretraining strategies: maximal update parameterisation and time transfer. These techniques enabled the team to optimise hyperparameters and ensure efficient scaling during the training phase, ensuring optimal performance and resource use in large-scale language model development.
While the technical implementation proceeded overall smoothly, the team of WP6 highlighted one persistent challenge: the acquisition of high-quality training data. This is particularly challenging in Germanic and other multilingual languages and remains a critical bottleneck in developing models that accurately reflect Europe’s linguistic diversity.
Despite this, the baseline models already demonstrate significant promise.
“Our goal is for the model to undergo post-training enhancements that will enable it to surpass other EU multilingual language models and be ready for real-world deployment. Our immediate next step is to scale up and train an even larger model.”
Looking ahead, WP6 will focus on scaling and training a larger model based on the current technical foundation, while fine-tuning and trustworthiness alignment are led by other work packages.
The long-term vision remains clear: to deliver an open, efficient, and trustworthy multilingual large language model that can compete on a global stage, supporting Europe’s digital sovereignty and AI leadership.