Shared Task 2 · LangCode Workshop at NAACL 2027

shared_task[2]Adversarial Prompt Detection

Guardrails for code-mixed and transliterated prompts

Given a prompt sent to a code model, decide whether it is benign or adversarial, and if adversarial, what kind of attack it is. Prompts are code-mixed across low-resource languages or transliterated into non-Latin scripts, with programming keywords kept as they are.

June 1–5, 2027
San Francisco, California
Submissions on CodaBench

Read the task How to participate

Updates

Oct 20, 2026Training and development data release; CodaBench registration opens Jan 15, 2027Test data release / evaluation begins; surprise languages revealed Jan 24, 2027Registration deadline Jan 31, 2027System submission deadline Feb 3, 2027Official results released Feb 15, 2027Shared-task paper submission deadline Mar 26, 2027Notification of acceptance Apr 16, 2027Camera-ready paper due Jun 1–5, 2027Workshop at NAACL 2027, San Francisco LangCode 2027Co-located with NAACL 2027 · June 1–5, 2027 · San Francisco, California AlsoShared Task 1: Multilingual Code Generation Feb 5, 2027Regular workshop papers due. See the Call for Papers

The task

Catch a malicious coding prompt in any language or script

Input

One prompt to a code model, code-mixed across low-resource languages or transliterated into a non-Latin script.

Output

Benign or adversarial (Subtask A), and the attack type for adversarial prompts (Subtask B).

Background

Code-model safety filters are calibrated almost entirely on English. Rewrite a malicious request in a low-resource language, or spell it out in another script, keep the code keywords intact, and many filters stop working. In MALICE, the best classifier's accuracy fell from 0.93 on English to 0.21 on code-mixed malicious prompts, and even the most safety-aligned model tested produced about 3× more malicious code.

Findings from MALICE (Raihan, Meher, Dhingra, Zampieri; Findings of EMNLP 2026).

Two subtasks

Benign or adversarial?

Binary classification of each prompt sent to a code model. This is the main leaderboard.

Label: 0 benign, 1 adversarial

Which attack?

For adversarial prompts, identify the attack type. Finer-grained, and closer to what a real guardrail needs to log.

Attack categories are listed with the training data

Every prompt is either code-mixed or transliterated. See Data for the 18 languages.

Two phases, both on CodaBench

Development, then evaluation

Phase 1

Development phase

Oct 20, 2026 – Jan 15, 2027

Data: Training and dev sets, plus the starter kit
You: Build and tune your system
CodaBench: Submit predictions on the dev set to see your macro-F1 and check your file format. Practice only: it does not count toward the final ranking.

Phase 2 · Final

Evaluation phase

Jan 15 – Jan 31, 2027

Data: A new test set, released Jan 15, without labels, including surprise languages
You: Run your final system on it
CodaBench: Submit your test predictions by Jan 31. These decide the official ranking, published Feb 3.

Next steps

LangCode 2027 · NAACL 2027

See you in San Francisco

Teams present their systems at LangCode 2027, the First Language and Code Workshop, co-located with NAACL 2027, June 1–5.