What you get
Each example is one prompt with a Subtask A label (benign or adversarial) and, for adversarial prompts, a Subtask B attack type. The task builds on MALICE, a benchmark of about 250K code-mixed and transliterated adversarial prompts, split evenly between the two attack styles.
| Split | Phase | Released | Use it to |
|---|---|---|---|
| Train | Development | Oct 30, 2026 | Train your detector |
| Dev | Development | Oct 30, 2026 | Tune your detector and practice submitting on CodaBench |
| Test | Evaluation | Jan 15, 2027 | Make your final predictions. New prompts, no labels, plus surprise languages. |
- ~250Kadversarial prompts in MALICE
- 18task languages
- 9source security benchmarks
How the prompts are built
A malicious English coding request from an established security benchmark is translated sentence by sentence into low-resource languages, or transliterated into a non-Latin script. Programming keywords stay as they are.
Code-mixed
Low-resource, Latin script. Sentences are mixed across languages within one prompt.
Transliterated
Written in non-Latin scripts (Chinese in Pinyin).
Surprise
Held back until the evaluation phase, for a cross-lingual leaderboard. Revealed Jan 15, 2027.
Starter kit Released Oct 30, 2026
Download it with the data on CodaBench. It gets you from zero to a valid submission.
Train and dev sets
The labeled prompts for building and tuning your detector, for both subtasks.
Public baselines
Baseline results to compare against, so you know where you stand.
Submission format
How to package your predictions in the file format CodaBench accepts.
Download links will appear here on October 30, 2026.
Cite the source benchmark
@inproceedings{raihan2026malice,
title = {On the Robustness of Code {LLM} Guardrails to Code-mixed and Transliterated Inputs},
author = {Raihan, Nishat and Meher, Dipak and Dhingra, Bhuwan and Zampieri, Marcos},
booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
year = {2026}
}