One attempt
The share of problems whose single solution passes all of its test cases.
Execution-based, no partial credit
Shared Task 1 · LangCode Workshop at NAACL 2027
One problem, 50 natural languages, one correct program
Given a programming problem described in one of 50 natural languages, among them Amharic, Georgian, Hawaiian, Nigerian Pidgin and Telugu, write a solution in the requested programming language, such as Python, C or Lua, that passes the hidden test cases. Your system should solve the problem no matter which natural language it is posed in.
Submit your system paper on OpenReview by Feb 5, 2027 (11:59 PM AoE). Choose Shared Task 1 system paper as the paper type. Taking part in the Test phase is required to submit a system paper.
The task
A function signature in one programming language (Python, C, Lua and others) and a problem description in one of 50 natural languages.
One solution in the same programming language. It counts only if it passes every test case.
Code models are prompted mostly in English and benchmarked mostly on English. Many programmers describe what they want in another language, and a model that solves a problem in English can fail on the same problem written in a lower-resource language. This task measures that gap and rewards systems that close it.
The share of problems whose single solution passes all of its test cases.
Execution-based, no partial credit
The leaderboard also gives Pass@1 for each language class, from class 0 (the least resourced) to class 3, and for each programming language.
The ranking uses overall Pass@1
Each problem has about 50 test cases. The data shows 2 of them; the rest stay hidden, and all are scored.
The test problems are held out until Jan 8
Full metric and leaderboard columns: see Leaderboard.
Two phases, both on CodaBench
Phase 1
Oct 27, 2026 – Jan 7, 2027
Phase 2 · Final
Jan 8 – Jan 24, 2027
Next steps
LangCode 2027 · NAACL 2027
Teams present their systems at LangCode 2027, the First Language and Code Workshop, co-located with NAACL 2027, June 1–5.