An end-to-end framework that jointly predicts dense metric depth, per-pixel reliability, and aleatoric uncertainty — with 10–67× fewer parameters than competing baselines.
Despite impressive progress in depth estimation, these fundamental problems remain unaddressed by prior work.
Downstream modules like motion planning and obstacle avoidance must know where to trust depth estimates. Textureless walls, reflective surfaces, occlusion boundaries, and distant objects receive identical treatment — yet have vastly different geometric difficulty.
Ground-level LiDAR models (KITTI: d ≤ 80 m, near-horizontal) generalize poorly to aerial cameras (VisDrone: bird's-eye), drone video (d ∈ [0,50] m), or indoor RGB-D (NYU V2: d ≤ 10 m). No prior work validated a single architecture across all four modalities.
Standard metrics like RMSE and δ₁ quantify average accuracy but not trustworthiness. They can heavily favor models that perform well on easy pixels while remaining dangerously overconfident in hard, ambiguous regions.
R³DC decomposes depth completion into three ordered stages, each with a specialized mechanism. The key insight: a single decoder cannot simultaneously infer global structure and respect sharp boundaries.
Each block is carefully designed to address a specific challenge in cross-domain depth completion with sparse inputs.
Unlike prior work that uses DCN only in propagation, R³DC embeds it directly in the depth encoder. Learned offsets Δpₖ and modulation mₖ align the receptive field with valid sparse points, mitigating artifacts at depth discontinuities.
Depth features query RGB key/value via multi-head attention at stages ℓ ∈ {1,2,3}. GroupNorm stabilizes the query branch; logits clamped to [−8,8]. Memory bounded to O(Nmax²) via pooling to 512 tokens.
At 1/16 resolution: spatial transformer (8 heads, GELU MLP), followed by CBAM channel-and-spatial attention. Captures long-range dependencies while keeping attention tractable.
AffNet takes [u₁; R̂] as input to predict 8 affinity weights per pixel. High-confidence pixels resist perturbation; low-confidence regions absorb spatial context. T=6 iterations with hard Dirichlet boundary conditions at sparse anchors.
4 blocks: transposed Conv4×4 upsample → DCN lateral skip → pre-activation ResBlock → CBAM → CMA with RGB skip. Preserves high-frequency depth boundaries lost in attention pooling.
For NYU Depth V2: frozen DA-V2 ViT-S backbone + 3-layer MLP ICH (16,642 params). Predicts dataset-specific shift & scale to convert unscaled relative priors to metric range [0,10] m.
Multi-task depth learning faces inherent gradient conflicts. A balanced composite loss allows each objective to contribute its signal without domination.
Removes global depth bias via variance − scale terms. Critical for cross-domain transfer: KITTI (80 m) ↔ NYU (10 m).
Focal weight (1−e−|e|)² continuously up-weights occlusion boundaries. γ=2 provides quadratic emphasis while preserving differentiability.
Window size wₛ=7. Preserves edge consistency and local contrast structure beyond simple pixel-wise losses.
Forces reproduction of known sensor measurements. Prevents metric-scale drift, critical at extreme sparsity (0.5–2%).
Penalizes back-facing surface normals via random triplet sampling. Domain-agnostic: no camera intrinsics required.
Self-supervised uncertainty: |d̂−d|/σ̂ + ln σ̂. Penalizes confident-wrong predictions; prevents σ̂→0 collapse. No GT uncertainty labels needed.
Standard metrics answer "How accurate is the model on average?" RADI answers three complementary questions about trustworthiness. These questions are orthogonal — a model can pass one and fail the others.
Question: Does the model know where it is wrong?
Spearman rank correlation between predicted reliability R̂ and the negative absolute error. A positive ρ means high confidence predicts low error — the desired semantic behavior.
Evaluated across 4 spatial regions: All · Edge (Sobel>0.05) · Textureless (luma σ<8) · Far-depth (d>0.75 dmax). Why Spearman? Invariant to absolute scale; robust to heavy-tailed depth error distributions.
Question: Does the architecture actually use its confidence during refinement?
High REC alone proves the model knows where it's wrong. RBS independently validates that reliability-gated CSPN++ actually improves depth by measuring normalized RMSE reduction from D₀ to D₁.
R³DC achieves RBS > 41% in all spatial regions on NYU Depth V2 (corrected metric-space values), confirming that confidence-gated propagation adds genuine value.
Question: Are the confidence values numerically calibrated?
Reliability values are grouped into B=15 equal-width bins. For bin b, the empirical accuracy āb is the fraction of pixels where relative error < τ=0.10. Perfect calibration → ECE=0.
A random uniform predictor yields ECE≈0.25. R³DC achieves ECE = 0.041, 6× better than the MC-Dropout baseline (0.118) and 83% better than random.
Standard RMSE selects Model B. RADI correctly surfaces that Model A is preferable for motion planning and obstacle avoidance: its reliability map reveals precisely where it cannot be trusted.
R³DC is evaluated across four heterogeneous benchmarks with identical loss weights and optimizer settings — rigorously testing generalization, not benchmark overfitting.
| Method | Params | KITTI | NYU Depth V2 | VisDrone †† | |||
|---|---|---|---|---|---|---|---|
| RMSE↓ | δ₁↑ | δ₁↑ | RMSE↓ | RMSE↓ | δ₁↑ | ||
| Depth Completion Methods | |||||||
| NLSPN | 26.8M | 0.762 | N/E | N/E | N/E | N/E | N/E |
| CSPN++ | 17.4M | 0.744 | N/E | N/E | N/E | N/E | N/E |
| PENet | 131M | 0.730 | N/E | N/E | N/E | N/E | N/E |
| GuideFormer | 27.3M | 0.625 | N/E | N/E | N/E | N/E | N/E |
| CompletionFormer | 12.7M | 0.217 | N/E | N/E | N/E | N/E | N/E |
| BP-Net | 30.4M | 0.213 | N/E | N/E | N/E | N/E | N/E |
| Monocular Depth Estimation | |||||||
| AdaBins | 78.2M | — | — | 0.902 | 0.288 | — | — |
| DepthFormer | 34.3M | — | — | 0.921 | 0.270 | — | — |
| NeWCRFs | 270M | — | — | 0.922 | 0.264 | — | — |
| ZoeDepth | 345M | — | — | 0.951 | 0.270 | — | — |
| R³DC / R³DC+ — One Model Family, Four Domains | |||||||
| R³DC KITTI (ours) | 1.95M | 0.240 | 0.947 | — | — | — | — |
| R³DC+ICH NYU (ours) | 94.6M | — | — | 0.927 | 0.353 | — | — |
| R³DC+ v3 VisDrone (ours) | 11.22M | — | — | — | — | 2.33 | 0.928 |
†† Novel benchmarks: no prior depth completion methods report results on VisDrone or Drone-Videos. N/E = not evaluated in prior work. Bold = best; underline = second best.
| Depth Model | Reliability Source | REC ρ (All)↑ | REC ρ (Edge)↑ | REC ρ (Textureless)↑ | RBS (%)↑ | CAL ECE↓ |
|---|---|---|---|---|---|---|
| Baselines (no revision stage — RBS=0 by construction) | ||||||
| CompletionFormer | Uniform (≡0.5) | 0.000 | 0.000 | 0.000 | 0.0% | 0.248 |
| CompletionFormer | Inv-gradient | 0.147 | 0.131 | 0.169 | 0.0% | 0.183 |
| CompletionFormer | Depth-error proxy | 0.214 | 0.198 | 0.231 | 0.0% | 0.142 |
| CompletionFormer | MC-Dropout | 0.271 | 0.249 | 0.292 | 0.0% | 0.118 |
| CompletionFormer | Ensemble (3 seeds) | 0.303 | 0.281 | 0.319 | 0.0% | 0.097 |
| R³DC+ICH (ours) | Learned R̂ | 0.371 | 0.358 | 0.389 | 41.3% | 0.041 |
R³DC's jointly trained reliability head achieves REC 43% higher than the depth-error proxy heuristic, confirming non-trivial calibration value beyond simple confidence estimates.
All ablations run on KITTI (epoch 8, Val RMSE), modifying one factor at a time from the full R³DC configuration (B=64, 1.95M params).
From 50% to 0.5% input density, RMSE increases by only 4.1× while density decreases by 100×, demonstrating sub-linear degradation.
The reliability maps consistently assign high confidence to textured surfaces and low confidence to reflective objects, occlusion boundaries, and far-range regions — validated by per-region RADI scores.
VisDrone and Drone-Videos evaluations rely on physics-motivated synthetic depth prior. Proxy validation on DublinCity LiDAR yields Pearson r=0.81, but real aerial LiDAR is needed for full validation. RADI conclusions restricted to KITTI and NYU.
Current approach treats each frame independently. Online reliability adaptation or temporal smoothing could improve predictions for dynamic video sequences. Semi/self-supervised training on unlabeled frames is a natural extension.
Under the official benchmark (structured Velodyne input, mm units), R³DC (1.95M) scores 786.4 mm RMSE vs. CompletionFormer's 708.2 mm — not state-of-the-art. The principal contribution is 6–67× parameter reduction at comparable accuracy, plus per-pixel reliability absent in all baselines.
The reliability head R̂ receives no direct GT supervision — learned implicitly through CSPN++ gating loss and shared features with σ̂. Adding a direct supervision signal (reliability cross-entropy) could accelerate calibration.
If you find R³DC or the RADI metric useful, please consider citing: