This honestly sounds like you’re seeing a bug.
Well, it probably is. I’ll soon be getting a GPU with a bigger VRAM and putting that GPU into my home server for Jellyfin as a replacement for QuickSync (NVENC is better, imo).
cargo clippy
There is one game called Cities: Skylines 2 that always fills up my VRAM, so yeah, I’m getting an OOM, but on the VRAM (I have GTX 1660super with 6 gigs of VRAM and I have 32 gigs of system RAM). I encourage you to try playing this game with a moderately sized city and with this GPU.
If you’re running this GPU under Windows, it’s fine. But good luck doing that under Linux.
https://forums.developer.nvidia.com/t/non-existent-shared-vram-on-nvidia-linux-drivers/260304?page=2
Try VMware Player. Its VM performance is pretty good, and it supports Wayland with Its drivers.
AFAIK they also helped when the Mt.Gox hack happened.