Auditing Anti-Muslim Bias in Frontier LLMs
Across Reasoning, Agentic, and Time-Coupled Conditions
Five years after the discovery of persistent anti-Muslim bias in large language models, most evaluations remain confined to single-turn prompt completion — a setting that no longer reflects how frontier LLMs are deployed. We introduce MIRAGE (Muslim-Identity Reasoning and Agentic Generation Evaluation), a benchmark of 1,200 prompts spanning three deployment-realistic conditions: direct completion, chain-of-thought reasoning, and simulated agentic decision-making across content moderation, lending triage, refugee claim summarization, and hiring screens. Across six frontier models, we find that chain-of-thought reasoning amplifies rather than suppresses Muslim-violence associations by 12–34% relative to direct completion, agentic decisions exhibit a 9–22 percentage-point asymmetry between Muslim and matched non-Muslim cases on identical evidence, and bias is sharply time-coupled to retrieved news context, increasing 18–27% under recent-conflict retrieval.
Key findings
CoT amplification. Chain-of-thought reasoning increases Muslim-violence associations relative to direct completion — contrary to the assumption that reasoning suppresses bias.
Agentic asymmetry. Percentage-point gap between Muslim and matched non-Muslim decisions on identical evidence across moderation, lending, refugee, and hiring harnesses.
Time-coupled escalation. Bias increases under recent-conflict RAG retrieval, operationalizing the news-cycle coupling first noted by Abid et al. (2021).
Existing prompt-based mitigations transfer poorly: they suppress direct-completion bias while leaving agentic decision asymmetry largely intact, revealing a systematic gap between what benchmarks measure and what deployed systems do.
Benchmark design
Baseline single-turn prompts across six templates (T1–T6): open continuation, analogical, biographical, news summarization, decision context, and dialogue.
Same prompts with CoT elicitation. Reasoning traces audited for three amplification pathways: group-statistic invocation, reasoner-as-stand-in, and coherence pressure.
Four high-stakes harnesses — content moderation (H1), lending triage (H2), refugee summarization (H3), hiring screening (H4) — with calibrated decision ambiguity designed to surface asymmetric bias.
Every prompt pairs a Muslim-identifying variant with a non-Muslim control (Christian, Jewish, Hindu, secular) via a single lexical substitution. All surrounding content is held fixed, isolating the identity signal causally.
400 prompts translated into MSA and three Arabic dialects (Egyptian, Levantine, Maghrebi) by bilingual speakers with independent post-editing, enabling language-mediated vs. model-internal bias decomposition.
240 per template × 5 identity variants = 1,200 base prompts per condition. A 15% stratified lite split supports lower-resource replication.
Claude Opus 4, GPT-4o, Gemini 1.5 Pro (closed, high-safety); Llama-3.3-70B, Qwen2.5-72B, DeepSeek-V3 (open, instruction-tuned).
Results — Chain-of-thought
Model frames a stereotyped inference as a probabilistic prediction grounded in "base rates," then endorses it as statistically expected. Most prevalent in open models (44.6% of amplifying traces).
Stereotype framed as what a "reasonable" or "neutral" observer would expect — licensing it as social norm rather than factual claim. Equally prevalent across all model tiers.
An early mild negative inference commits subsequent reasoning steps to develop it into a stereotyped conclusion for chain coherence. Absent under direct completion (C₁).
| Model tier | P1 — group stats | P2 — reasoner | P3 — coherence | Other / mixed |
|---|---|---|---|---|
| Closed, high-safety | 18.3% | 31.4% | 14.2% | 36.1% |
| Closed, standard | 27.9% | 29.1% | 11.8% | 31.2% |
| Open | 44.6% | 27.2% | 10.3% | 17.9% |
Distribution of CoT amplification pathways (% of amplifying traces) by model tier. The alignment layer suppresses identical inferences under C₁ — CoT provides an intermediate output position that bypasses output-level safety training.
Results — Violence rates & agentic asymmetry
| Model | C₁ violence rate | C₁→C₂ Spearman ρ | C₁→C₃ Spearman ρ | Mean δ (agentic) |
|---|---|---|---|---|
| Claude Opus 4 | — | 0.58 | 0.29 | — |
| GPT-4o | — | 0.55 | 0.31 | — |
| Gemini 1.5 Pro | — | 0.60 | 0.33 | — |
| Llama-3.3-70B | 18.7% | 0.64 | 0.37 | — |
| Qwen2.5-72B | 23.9% | 0.66 | 0.38 | — |
| DeepSeek-V3 | 29.4% | 0.63 | 0.36 | — |
| Mean | — | 0.61 | 0.34 | — |
The weak C₁–C₃ Spearman ρ = 0.34 confirms that ~38% of high-δ agentic items have low C₁ violence rates — decision asymmetry is not reducible to surface-level association strength. A C₁-only benchmark will both underestimate the deployed harm surface and misrank mitigations.
Mitigation audit
System-prompt prefix asserting cultural awareness and equal treatment across groups.
Pre-prompt positive-valence statement framing subjects as valued community members.
Pre-generation suffix asking the model to identify stereotypes before responding.
Two-turn pipeline: unconstrained generation followed by fairness-criterion critique and revision.
Bar lengths show bias reduction relative to M0 baseline (larger = better). ● strong, ● partial, ● poor transfer. M4 is the strongest single mitigation but still leaves 78% of agentic asymmetry unaddressed — the structural gap motivates dedicated agentic-condition alignment work.
Contributions
1,200 matched-pair prompts across direct completion, chain-of-thought, and agentic conditions, with parallel English/Arabic translations covering MSA and three dialects.
Quantitative audit across six frontier models demonstrating that chain-of-thought prompting amplifies rather than suppresses anti-Muslim bias, with three identified mechanistic pathways.
9–22 pp decision asymmetry documented across four high-stakes harnesses under identical evidence. Refugee-claim summarization shows the largest gap.
First explicit measurement of news-cycle-coupled bias escalation under RAG, plus a cross-condition mitigation audit showing that prompt-based defenses do not transfer to agentic settings.
Citation
@inproceedings{mohammad2026mirage,
title = {{MIRAGE}: Auditing Anti-Muslim Bias in Frontier {LLMs}
Across Reasoning, Agentic, and Time-Coupled Conditions},
author = {Mohammad, Noor Islam S. and Sheikh, Tamim},
booktitle = {6th Muslims in {ML} Workshop at the 43rd International
Conference on Machine Learning},
address = {Seoul, South Korea},
year = {2026},
note = {Copyright 2026 by the author(s)}
}
Presented at the 6th Muslims in ML (MusIML) Workshop at the 43rd International Conference on Machine Learning, Seoul, South Korea. Copyright 2026 by the author(s).