Four displays, eight physical windows
Four displays on an RTX 5090 worked in Windows. They worked on Linux with overlay layers disabled. With overlays enabled, the Linux driver rejected the same modes. That pointed at how the driver handled overlay resources. Reproduction and target modes.
The setup:
| Display | Connection | Target mode |
|---|---|---|
| MSI MAG321UX OLED | HDMI | 3840 × 2160, 240 Hz, HDR |
| Samsung Odyssey G93SC | DisplayPort | 5120 × 1440, 240 Hz, HDR |
| LG HDR 4K | DisplayPort | 3840 × 2160, 60 Hz, HDR |
| HYTE Y70ti | DisplayPort | 2560 × 682, 73.778 Hz, SDR |
These modes need six hardware tiles. The driver reserved two layers per tile, which is twelve physical windows. The GPU has eight. It reserved them even for layers that usage validation had already marked unusable.
The fix had three parts
The patch allocates physical windows from validated layer usage. It keeps hardware ownership in step with the new assignments. It leaves the advertised overlay planes and software mappings alone.
Allocation alone did not work. Earlier attempts gave black screens and Xid 56 errors, and I rolled them back. The working version handles assignment, ownership, and modeset sequencing together. Patch and investigation account.
The production diff makes three changes:
- Allocate only usable layers. An excluded layer does not keep or take a physical window. Allocation still runs again when layer usage changes and the tile count does not.
- Move ownership on the shutdown path. Assignment and ownership changes go through head shutdown before reprogramming. The code writes the final owner even when the cached value looks correct, because initialization may have queued a different binding.
- Decide sequencing after preparation. The choice to split window flips from the core modeset update happens after
PreUpdate, so assignment changes found there count.
A second, smaller change retries a failed head after it reclaims resources. Before, an incomplete allocation could report success. The same change removes an undefined shift when building a 32-window mask. That was a separate boundary bug. Production diff.
What was validated
Hardware checks ran on Ubuntu, NVIDIA 610.43.02, kernel 7.0.0-30-generic, with overlays enabled. The submitted patch is based on 610.57.04.
| Evidence | Result |
|---|---|
| Full shutdown and power-on | All four target modes came up. HDR readback matched on the three HDR displays. |
| Two overlay runs, twenty commits each | Overlays worked at baseline, released for the target modes, and returned on the way back. |
| 2,359,296 allocator-model combinations under ASan/UBSan | Coverage of the allocation model in software. |
Outside this validation: simultaneous active overlays at the maximum modes, where validation marks those layers unavailable, plus other hardware, long sessions, and suspend and resume. A recurring platform startup problem blocked warm-reboot testing. Validation details and regression harness.
The display that stayed blank
Later, a Samsung display stayed blank after wake. The connector reported connected and enabled. The driver still held an attached group that expected display-stream compression (DSC), and a successful DPCD read showed the sink’s DSC enable bit off.
The patch restores DSC when a capable DisplayPort sink returns to an attached single-stream group. It uses the existing setter, so sink decompression and driver tracking are both restored. NVIDIA PR #1349.
Read the display-wake investigation →
Source and current state
Both pull requests were open on September 20, 2026:
- #1338 — Physical-window allocation, ownership, and sequencing
- #1349 — DSC restoration on SST reinsertion
- DSC hardware validation and regression tests
The allocation PR records a joint investigation with GPT-6 Astra. Source, reproduction steps, and validation limits are all in the pull requests.