Safety Score
RU + RS − 100 (Youden's J): a missed attack and a flagged safe prompt cost the same.
The ranking
Shared Task 2 · LangCode Workshop at NAACL 2027
Guardrails for code-mixed and transliterated prompts
Given a prompt sent to a code model, decide whether it is safe or unsafe. Every prompt is code-mixed (one of 10 languages mixed with English) or transliterated (English written in another script), with programming and security terms often kept in English.
Submit your system paper on OpenReview by Feb 5, 2027 (11:59 PM AoE). Choose Shared Task 2 system paper as the paper type. Taking part in the Test phase is required to submit a system paper.
The task
One prompt to a code model, code-mixed with English or transliterated into the script of Arabic, Bengali, Chinese, Hebrew, Hindi, Japanese, Korean, Russian, Tamil or Thai.
One label: S (safe) or U (unsafe).
Code-model safety filters are calibrated almost entirely on English. Rewrite a malicious request in another language, or spell it out in another script, keep the code keywords intact, and many filters stop working. In MALICE, the best classifier's accuracy fell from 0.93 on English to 0.21 on code-mixed malicious prompts, and even the most safety-aligned model tested produced about 3× more malicious code.
Safe prompts in this task are on the same security topics as the unsafe ones: how to detect an attack, defend a system or test one's own setup. A guardrail that flags every security word fails them, so a system has to read what each prompt asks for.
Findings from MALICE ↗ (Raihan, Meher, Dhingra, Zampieri; Findings of EMNLP 2026).
RU + RS − 100 (Youden's J): a missed attack and a flagged safe prompt cost the same.
The ranking
The share of unsafe prompts your system labels U. Every missed attack lowers it.
Shown on the leaderboard
The share of safe prompts your system labels S. Every false alarm lowers it.
Shown on the leaderboard
Labeling every prompt U, every prompt S, or at random scores 0; a perfect system scores 100. Full definition: see Leaderboard.
Two phases, both on CodaBench
Phase 1
Oct 29, 2026 – Jan 7, 2027
Phase 2 · Final
Jan 8 – Jan 24, 2027
Next steps
Train, dev and dev-test sets, the scorer and a starter notebook, out now.
Data → 2Practice on the dev-test set until Jan 7, then submit test labels by Jan 24, 2027.
Participate → 3Describe your system in up to 4 pages, due Feb 5, 2027. Test-phase teams only.
Paper Submission →LangCode 2027 · NAACL 2027
Teams present their systems at LangCode 2027, the First Language and Code Workshop, co-located with NAACL 2027, June 1–5.