Shared Task 2 · LangCode Workshop at NAACL 2027

shared_task[2]Adversarial Prompt Detection

Guardrails for code-mixed and transliterated prompts

Given a prompt sent to a code model, decide whether it is safe or unsafe. Every prompt is code-mixed (one of 10 languages mixed with English) or transliterated (English written in another script), with programming and security terms often kept in English.

Updates
Oct 29, 2026Train, dev and dev-test data release; CodaBench registration opens Jan 8, 2027Test data release / test phase begins Jan 17, 2027CodaBench registration deadline Jan 24, 2027System submission deadline Jan 25, 2027Official results released Feb 5, 2027Shared-task paper submission deadline Mar 26, 2027Notification of acceptance Apr 16, 2027Camera-ready paper due Jun 1–5, 2027Workshop at NAACL 2027, San Francisco LangCode 2027Co-located with NAACL 2027 · June 1–5, 2027 · San Francisco, California AlsoShared Task 1: Multilingual Code Generation Feb 5, 2027Regular workshop papers due. See the Call for Papers
Submissions open

Submit your system paper on OpenReview by Feb 5, 2027 (11:59 PM AoE). Choose Shared Task 2 system paper as the paper type. Taking part in the Test phase is required to submit a system paper.

Submit on OpenReview

The task

Catch a malicious coding prompt in any language or script

Input

One prompt to a code model, code-mixed with English or transliterated into the script of Arabic, Bengali, Chinese, Hebrew, Hindi, Japanese, Korean, Russian, Tamil or Thai.

Output

One label: S (safe) or U (unsafe).

Background

Code-model safety filters are calibrated almost entirely on English. Rewrite a malicious request in another language, or spell it out in another script, keep the code keywords intact, and many filters stop working. In MALICE, the best classifier's accuracy fell from 0.93 on English to 0.21 on code-mixed malicious prompts, and even the most safety-aligned model tested produced about 3× more malicious code.

Safe prompts in this task are on the same security topics as the unsafe ones: how to detect an attack, defend a system or test one's own setup. A guardrail that flags every security word fails them, so a system has to read what each prompt asks for.

Findings from MALICE ↗ (Raihan, Meher, Dhingra, Zampieri; Findings of EMNLP 2026).

How systems are scored

Safety

Safety Score

RU + RS − 100 (Youden's J): a missed attack and a flagged safe prompt cost the same.

The ranking

RU

Unsafe recall

The share of unsafe prompts your system labels U. Every missed attack lowers it.

Shown on the leaderboard

RS

Safe recall

The share of safe prompts your system labels S. Every false alarm lowers it.

Shown on the leaderboard

Labeling every prompt U, every prompt S, or at random scores 0; a perfect system scores 100. Full definition: see Leaderboard.

Two phases, both on CodaBench

Development, then test

Phase 1

Development phase

Oct 29, 2026 – Jan 7, 2027

Data
Train, dev and dev-test sets, the scorer and a starter notebook
You
Build and tune your system
CodaBench
Submit labels for the dev-test set to see your Safety Score and check your file format.

Phase 2 · Final

Test phase

Jan 8 – Jan 24, 2027

Data
30,000 test prompts, released Jan 8, without labels
You
Run your final system on it
CodaBench
Submit your test labels by Jan 24. These decide the official ranking, published Jan 25.

LangCode 2027 · NAACL 2027

See you in San Francisco

Teams present their systems at LangCode 2027, the First Language and Code Workshop, co-located with NAACL 2027, June 1–5.