The WINWIN AI Competence Center under Ukraine's Ministry of Digital Transformation, together with a group of researchers, has launched the Ukrainian LLM Leaderboard — the first national ranking of large language models that measures their performance specifically in the Ukrainian language. The initiative fills a systemic gap in the industry: until now, global leaderboards tested neural networks primarily in English, while Ukrainian competence was assessed indirectly, through machine translation, which masked specific errors, calques, and failures in understanding local context. Now the state has an objective basis for selecting technological solutions for government applications, and businesses have a tool for evaluating the profitability of integrating artificial intelligence into their product lines.
Why Previous Approaches Did Not Work
The key problem with older methodologies was that model quality in Ukrainian was not tested directly, but through a chain of machine translation. This approach concealed typical defects of the local language: calques from English, errors in grammatical structures, and the model's inability to correctly process specific vocabulary. The new evaluation rules were developed by experts from the Kyiv School of Management University (KSMU) and the lang-uk community — Yuriy Panov and Dmytro Chaplinsky — who have been training neural networks to understand Ukrainian for over a decade. It is separately noted that the same methodology has already been successfully applied when training the domestic Lapa LLM model, confirming its reproducibility and practical value.
Methodology: Four Directions and Local Datasets
The platform measures model effectiveness across several key directions, combining Ukrainian and international datasets adapted to the local context. In the machine translation block, evaluation is conducted at the level of individual sentences and entire paragraphs using the FLORES-200, LongFLORES, and WMT-22 datasets. The contextual question-and-answer section tests reading comprehension on the Belebele and SQuAD datasets. The "logic, knowledge, and reasoning" direction includes tests used by educational institutions and in the ZNO format (ZNO-Eval), as well as complex logical tasks from Winogrande, ARC Easy/Challenge, TriviaQA, and MMLU. Finally, instruction-following is assessed by the accuracy of executing complex requests within the IFEval benchmark.
Who Is on Top: The Leaders of the Ranking
According to the updated leaderboard data, the top positions are held by specialized and optimized models. Among the leaders by average rank, three solutions stand out. The google/gemma-4-26B-A4B-it model, using reasoning algorithms, demonstrates one of the highest scores in instruction-following and logical problem-solving. The MamayLM series (12B and 27B models based on Gemma-3 from the INSAIT Institute) shows strong results in machine translation and logical tests. The Ukrainian-language lapa-v0.1.2-instruct model from the developer community, in turn, occupies one of the best positions in the translation block by FLORES and WMT metrics. All results, codes, and datasets are published openly on the Hugging Face platform, allowing any researcher to reproduce the evaluation.
Open Code and Future Plans
The developers have announced an expansion of the Ukrainian LLM Leaderboard's functionality. The platform will gain modules for testing models' work with images, evaluating the ethics of responses, and analyzing the financial cost of using them in Ukrainian. Particular attention will be paid to the needs of the public sector: the ranking plans to include the assessment of processing regulatory texts, administrative procedures, and the security of handling personal data. Thus, the leaderboard is evolving from an academic quality measure into a full-fledged tool for procurement and regulatory policy in the field of artificial intelligence.
Significance for the State and Business
The practical value of the initiative goes beyond academic interest. For the state, the ranking becomes a transparent basis for selecting technological solutions when developing government digital services, where quality performance in Ukrainian is not an option but a mandatory requirement. For business, the tool allows for an objective assessment of the profitability of AI integration by comparing the accuracy and cost of models within a single methodology. The openness of the code and datasets further lowers the barrier to entry for developers and stimulates the formation of a local ecosystem oriented toward Ukrainian-language artificial intelligence.
