Shared Task 1 · LangCode Workshop at NAACL 2027

shared_task[1]Multilingual Code Generation

One problem, 50 natural languages, one correct program

Given a programming problem described in one of 50 natural languages, among them Amharic, Georgian, Hawaiian, Nigerian Pidgin and Telugu, write a solution in the requested programming language, such as Python, C or Lua, that passes the hidden test cases. Your system should solve the problem no matter which natural language it is posed in.

Updates
Oct 27, 2026Trial and development data release; CodaBench registration opens Jan 8, 2027Test data release / test phase begins Jan 17, 2027CodaBench registration deadline Jan 24, 2027System submission deadline Jan 25, 2027Official results released Feb 5, 2027Shared-task paper submission deadline Mar 26, 2027Notification of acceptance Apr 16, 2027Camera-ready paper due Jun 1–5, 2027Workshop at NAACL 2027, San Francisco LangCode 2027Co-located with NAACL 2027 · June 1–5, 2027 · San Francisco, California AlsoShared Task 2: Adversarial Prompt Detection Feb 5, 2027Regular workshop papers due. See the Call for Papers
Submissions open

Submit your system paper on OpenReview by Feb 5, 2027 (11:59 PM AoE). Choose Shared Task 1 system paper as the paper type. Taking part in the Test phase is required to submit a system paper.

Submit on OpenReview

The task

Write the same program, whatever language the problem is in

Input

A function signature in one programming language (Python, C, Lua and others) and a problem description in one of 50 natural languages.

Output

One solution in the same programming language. It counts only if it passes every test case.

Background

Code models are prompted mostly in English and benchmarked mostly on English. Many programmers describe what they want in another language, and a model that solves a problem in English can fail on the same problem written in a lower-resource language. This task measures that gap and rewards systems that close it.

How systems are scored

Pass@1

One attempt

The share of problems whose single solution passes all of its test cases.

Execution-based, no partial credit

0–3

Every class shown

The leaderboard also gives Pass@1 for each language class, from class 0 (the least resourced) to class 3, and for each programming language.

The ranking uses overall Pass@1

tests

Hidden tests

Each problem has about 50 test cases. The data shows 2 of them; the rest stay hidden, and all are scored.

The test problems are held out until Jan 8

Full metric and leaderboard columns: see Leaderboard.

Two phases, both on CodaBench

Development, then test

Phase 1

Development phase

Oct 27, 2026 – Jan 7, 2027

Data
Trial and dev sets, the scorer and the submission format
You
Build and tune your system
CodaBench
Submit solutions for the dev problems to see your Pass@1 and check your file format.

Phase 2 · Final

Test phase

Jan 8 – Jan 24, 2027

Data
1,000 test problems, released Jan 8, each with 2 of its test cases shown
You
Run your final system on it
CodaBench
Submit your test solutions by Jan 24. These decide the official ranking, published Jan 25.

LangCode 2027 · NAACL 2027

See you in San Francisco

Teams present their systems at LangCode 2027, the First Language and Code Workshop, co-located with NAACL 2027, June 1–5.