What you get
Each example is a function signature in one programming language (Python, C, C++, Java, and others) with a problem description in one natural language. Every problem comes with hidden unit tests that decide whether a solution is correct.
| Split | Phase | Released | Use it to |
|---|---|---|---|
| Train | Development | Oct 30, 2026 | Train your system |
| Dev | Development | Oct 30, 2026 | Tune your system and practice submitting on CodaBench |
| Test | Evaluation | Jan 15, 2027 | Make your final solutions. New problems, hidden tests. |
One problem, many languages
The task builds on mHumanEval ↗ (NAACL 2025), a multilingual extension of HumanEval. The same problem is described in more than 100 natural languages, grouped into resource tiers by how much text each language has on the web:
- High-resource: languages such as English and Spanish.
- Lower-resource: languages with far less text, where code models struggle most.
The problem also comes in several programming languages, and the solution must be written in the one requested. The correct program does not change with the natural language. The test set uses newly written problems so systems are scored on problems they cannot have memorized.
def sum_even(nums: list[int]) -> int: """ Return the sum of all even numbers in the list. >>> sum_even([1, 2, 3, 4]) 6 """ return sum(n for n in nums if n % 2 == 0)
/* Return the sum of all even numbers in the list. sum_even({1, 2, 3, 4}, 4) == 6 */ int sum_even(const int *nums, int len) { int s = 0; for (int i = 0; i < len; i++) if (nums[i] % 2 == 0) s += nums[i]; return s; }
// Return the sum of all even numbers in the list. // sum_even({1, 2, 3, 4}) == 6 int sum_even(const std::vector<int>& nums) { int s = 0; for (int n : nums) if (n % 2 == 0) s += n; return s; }
/** * Return the sum of all even numbers in the list. * sumEven(new int[]{1, 2, 3, 4}) == 6 */ static int sumEven(int[] nums) { int s = 0; for (int n : nums) if (n % 2 == 0) s += n; return s; }
Illustrative problem, not from the test set. Click a tab to pin it.
Starter kit Released Oct 30, 2026
Download it with the data on CodaBench. It gets you from zero to a valid submission.
Train and dev problems
Problems with descriptions across resource tiers, for building and tuning your system.
Public baselines
Baseline results to compare against, so you know where you stand.
Submission format
How to package your solutions in the file format CodaBench accepts.
Download links will appear here on October 30, 2026.
Cite the source benchmark
@inproceedings{raihan2025mhumaneval,
title = {m{H}uman{E}val - A Multilingual Benchmark to Evaluate Large Language Models for Code Generation},
author = {Raihan, Nishat and Anastasopoulos, Antonios and Zampieri, Marcos},
booktitle = {Proceedings of NAACL 2025},
year = {2025},
url = {https://aclanthology.org/2025.naacl-long.570/}
}