Shared Task 1 · LangCode Workshop at NAACL 2027

shared_task[1]Multilingual Code Generation

One problem, 100+ natural languages, one correct program

Given a programming problem described in one of more than 100 natural languages, write a solution in the requested programming language, such as Python, C, C++, or Java, that passes hidden unit tests. Your system should solve the problem no matter which natural language it is posed in.

June 1–5, 2027
San Francisco, California
Submissions on CodaBench

Read the task How to participate

Updates

Oct 20, 2026Training and development data release; CodaBench registration opens Jan 15, 2027Test data release / evaluation begins Jan 24, 2027Registration deadline Jan 31, 2027System submission deadline Feb 3, 2027Official results released Feb 15, 2027Shared-task paper submission deadline Mar 26, 2027Notification of acceptance Apr 16, 2027Camera-ready paper due Jun 1–5, 2027Workshop at NAACL 2027, San Francisco LangCode 2027Co-located with NAACL 2027 · June 1–5, 2027 · San Francisco, California AlsoShared Task 2: Adversarial Prompt Detection Feb 5, 2027Regular workshop papers due. See the Call for Papers

The task

Write the same program, whatever language the problem is in

Input

A function signature in one programming language (Python, C, C++, Java, and others) and a problem description in one of 100+ natural languages.

Output

One solution in the same programming language. It counts only if it passes every hidden unit test.

Background

Code models are prompted mostly in English and benchmarked mostly on English. Many programmers describe what they want in another language, and a model that solves a problem in English can fail on the same problem written in a lower-resource language. This task measures that gap and rewards systems that close it.

How systems are scored

Pass@1

One attempt

The share of problems whose single generated solution passes all hidden tests.

Execution-based, no partial credit

avg

Every tier counts

Pass@1 is macro-averaged across language resource tiers, so high-resource languages alone cannot win.

Each resource tier carries the same weight

tests

Hidden tests

Test cases are never released. The test problems are newly written and held out.

Limits contamination from pretraining data

Full metric and the two tracks (constrained and open): see Leaderboard.

Two phases, both on CodaBench

Development, then evaluation

Phase 1

Development phase

Oct 20, 2026 – Jan 15, 2027

Data: Training and dev sets, plus the starter kit
You: Build and tune your system
CodaBench: Submit solutions for the dev problems to see your Pass@1 and check your file format. Practice only: it does not count toward the final ranking.

Phase 2 · Final

Evaluation phase

Jan 15 – Jan 31, 2027

Data: New held-out problems, released Jan 15, with hidden unit tests
You: Run your final system on it
CodaBench: Submit your test predictions by Jan 31. These decide the official ranking, published Feb 3.

Next steps

LangCode 2027 · NAACL 2027

See you in San Francisco

Teams present their systems at LangCode 2027, the First Language and Code Workshop, co-located with NAACL 2027, June 1–5.