Coherence Under Commitment

Abstract

The Coherence Evaluation Trap

Large language models deployed for logical reasoning in knowledge-intensive domains exhibit a subtle but critical failure: coherence can be vacuously achieved through systematic abstention. A model that withholds commitment to either entailment or refutation satisfies negation consistency while providing no utility.

We introduce Coherence Under Commitment (CUC), a dual-query evaluation paradigm that jointly measures consistency and decisiveness. CUC contributes three innovations: a commitment score c(φ) = p(φ) + p(¬φ) quantifying probability mass allocated to decisive outcomes; a deterministic elicitation protocol via normalized YES/NO log probabilities; and a 3-way decision framework (TRUE/FALSE/UNCERTAIN) operationalizing the coherence-commitment tradeoff into metrics.

Experiments on four open-weight LLMs (1B–3B) across 204 FOLIO examples expose a sharp frontier. Qwen2.5-3B achieves near-zero contradiction (E[v_neg] = 0.025) but only 7.4% coverage, while TinyLlama-1.1B reaches 79.4% coverage with violations on every single example. Coherence-only evaluation would rank the abstaining model first — CUC exposes this as vacuous. The frontier generalizes to LOGIQA v2 (ρ = 0.97) and is provably scale-invariant.

The Problem

When Coherence Is a Lie

Negation consistency — not simultaneously affirming φ and ¬φ — is a minimal desideratum. But it can be trivially satisfied: a model that assigns near-equal, low probability to all outcomes achieves zero violations without doing any reasoning. Standard evaluation is blind to this.

⚠ Ranked #1 by standard coherence eval

Qwen2.5-3B · The Abstainer

Near-zero contradiction because it assigns negligible probability to all outcomes. It isn't reasoning — it's refusing to answer.

E[v_neg] (violations)0.025 ↓

Coverage (predictions made)7.4% ↓

E[c] mean commitment0.115 ↓

Examples abstained on>9 in every 10

⚠ Ranked #4 by standard coherence eval

TinyLlama-1.1B · The Overcommitter

Violates probability axioms on every single example. It doesn't reason — it affirms indiscriminately, assigning 169.8% total probability mass.

Coverage (predictions made)79.4% ↑

E[v_neg] (violations)0.698 ↑

% examples with any violation100%

E[c] mean commitment1.698 (169.8% mass)

Key algebraic identity: v_neg(φ) = max(0, c(φ) − 1). Any model with c(φ) ≤ 1 achieves zero negation violations — regardless of reasoning quality. Low commitment is sufficient for zero violation. This is the trap that standard evaluation cannot see.

Method

Three interconnected innovations that jointly determine whether a model is usefully coherent — not merely apparently so.

Commitment Score

c(φ) = p(φ) + p(¬φ)

Measures total probability mass allocated to decisive outcomes. When c(φ) ≪ 1, the model treats the query as unknown — regardless of apparent coherence. The first metric designed specifically to detect vacuous coherence.

Deterministic Elicitation

p(x) = exp(log P(YES|x)) / Σ

Normalized log-probabilities over YES/NO tokens. Fully reproducible: identical inputs produce identical outputs across all runs. Requires exactly 2 forward passes per example, no sampling, no decoding variance.

3-way Decision Framework

Structured rule with confidence threshold τ and margin δ (default: τ = 0.60, δ = 0.10):

TRUEif p(φ) ≥ τ and p(φ) ≥ p(¬φ) + δ

FALSEif p(¬φ) ≥ τ and p(¬φ) ≥ p(φ) + δ

UNCERTAINotherwise

Dual-Query Evaluation Protocol

Query Q_φ

Is φ logically entailed by P?
Answer YES or NO.

→ Extract p(φ) via log P(YES | Q_φ)

Query Q_¬φ

Is ¬φ logically entailed by P?
Answer YES or NO.

→ Extract p(¬φ) via log P(YES | Q_¬φ)

Results

The Frontier Exposed

FOLIO v0.0 validation set, 204 examples, four open-weight LLMs spanning 1B–3B parameters. Accuracy differs by at most 0.098 across all models — yet commitment and violation span an order of magnitude.

Model	Acc ↑	Coverage ↑	Acc_cov ↑	E[c] ↑	E[v_neg] ↓	%v_neg>0 ↓
Phi-2 (2.7B)	0.441	0.417	0.565	1.164	0.195	0.789
Qwen2.5-1.5B	0.402	0.309	0.508	0.674	0.166	0.461
Qwen2.5-3B ⚠ Vacuous	0.382	0.074	0.800	0.115	0.025	0.064
TinyLlama-1.1B	0.343	0.794	0.346	1.698	0.698	1.000

Table 1. CUC on FOLIO v0.0 (n = 204). ↑ higher is better, ↓ lower is better. Bold values mark the column extremum — these should be read in conjunction, not in isolation. Default thresholds: τ = 0.60, δ = 0.10. All values carry 95% bootstrap CIs (B = 1,000, seed = 42).

The ranking inversion: Under accuracy alone, Qwen2.5-3B (0.382) and TinyLlama (0.343) differ by just 0.039 points — despite maximally opposite behaviors. Under coherence-only eval, Qwen2.5-3B leads by 6.6× on E[v_neg] — an advantage that collapses to 7.4% vs. 79.4% coverage once commitment is accounted for. CUC resolves both distortions simultaneously.

CUC vs. Prior Evaluation Paradigms

Property	Accuracy	ECE	Sel. Pred.	SelfCheck	CUC (ours)
Detects vacuous coherence via abstention	✗	✗	✓	✗	✓
Detects overcommitment (contradiction)	✗	✗	✗	✓	✓
Sensitive to ECE inflation via filtering	✗	✗	✗	✗	✓
No architectural modification required	✓	✓	✓	✓	✓
Measures cross-query logical consistency	✗	✗	✗	✗	✓
Fully deterministic across runs	✓	✓	✗	✗	✓

Ablations

Four Targeted Ablations

All four ablations confirm the frontier is robust — not a calibration artifact, not a surface effect of keyword choice, not an artifact of model scale alone.

τ δ

Threshold Sensitivity

Tested 8 (τ, δ) configurations from (0.50, 0.05) to (0.80, 0.20). The coverage–accuracy tradeoff is monotone across all settings. Frontier model rankings never change.

ρ = 1.0 across all threshold configs

A?

Elicitation Format

YES/NO vs. True/False vs. Entailed/Refuted. Absolute commitment shifts by up to 0.15 across formats. Rank ordering is perfectly preserved (ρ = 1.0). Format affects magnitude only, never relative position.

ρ = 1.0 across YES/NO, T/F, Ent/Ref

∩

Component Analysis

Coherence-only rewards abstention (ranks Qwen-3B first). Commitment-only rewards overcommitment (ranks TinyLlama first). Coverage-only ignores prediction quality. Only CUC detects both failure modes and ranks correctly.

CUC: sole framework with correct ranking

↑s

Scale Effects

Scaling Qwen2.5 from 1.5B to 3B reduces commitment by 83% and coverage by 23.5 points with negligible accuracy change (−0.020). The larger model learns to hedge, not to reason.

−83% commitment, −23.5 pp coverage

Scale Effects Within the Qwen2.5 Family (Table 6)

Model	E[c] ↑	E[v_neg] ↓	Coverage ↑	Acc	Acc_cov	ECE_cov
Qwen2.5-1.5B	0.674	0.166	0.309	0.402	0.508	0.187
Qwen2.5-3B	0.115	0.025	0.074	0.382	0.800	0.089
Δ (3B − 1.5B)	−0.559	−0.141	−0.235	−0.020	+0.292	−0.098

Standard evaluation would report the 3B model as unambiguously superior (4.9× lower violation, 2.1× lower ECE). CUC reveals these gains are purchased at an 83% reduction in coverage — the model learns aggressive abstention, not better reasoning.

Generalization

The Frontier Holds Across Benchmarks

ρ=0.97

Spearman rank correlation
FOLIO vs. LOGIQA v2
(p < 0.05, n = 304)

Cross-Dataset Validation · LOGIQA v2

All four original models were evaluated on the LOGIQA v2 test split (304 examples) using rule-based linguistic negation — since LOGIQA lacks FOLIO's formally verified negation fields. The frontier structure is qualitatively preserved across datasets: Qwen2.5-3B remains the lowest-commitment model (Coverage = 6.9%) and TinyLlama-1.1B remains universally violated.

Near-perfect Spearman rank correlation (ρ = 0.97) with FOLIO-derived frontier orderings confirms that frontier positions are not artifacts of benchmark label distribution or premise style.

Scale Extension · Llama-3 Family

Scaling Llama-3.2-1B → Llama-3.1-8B within a second model family replicates the hedging-under-scale pattern: commitment drops 39% (0.841 → 0.529) and coverage drops 17.3 points (56.9% → 39.6%) with negligible accuracy change (−0.014). The core finding is not specific to the Qwen2.5 training pipeline.

Theoretical Scale-Invariance

Theorem B.1 establishes that the frontier v_neg = max(0, c − 1) is a direct consequence of the algebraic identity between c(φ) and v_neg(φ) — no model, at any scale or architecture, can escape below it. Theorem B.3 proves that perfect coherence is achievable without any reasoning ability simply by assigning p(φ) = p(¬φ) ≤ 0.5. Both results hold at 1B and 100B parameters identically.

Citation

Cite This Work

If you use CUC or the evaluation toolkit in your research, please cite:

@inproceedings{mohammad2026cuc,
  title        = {Coherence Under Commitment: Probing Generalization and
                   Vacuous Memorization in LLM Logical Reasoning},
  author       = {Mohammad, Noor Islam S. and Hasan, Mahmudul},
  booktitle    = {Proceedings of the Efficient Multimodal Question Answering
                   Workshop at the 43rd International Conference on
                   Machine Learning (ICML 2026)},
  series       = {Proceedings of Machine Learning Research},
  volume       = {306},
  year         = {2026},
  address      = {Seoul, South Korea},
  url          = {https://pmlrbd.github.io/auc.ml/},
  organization = {PMLR}
}

CoherenceUnder Commitment

The Coherence Evaluation Trap

When Coherence Is a Lie

Qwen2.5-3B · The Abstainer

TinyLlama-1.1B · The Overcommitter