Docker NVIDIA HW doesn't work after a few hours

[BlackDub 27]

I have NVIDIA Container Toolkit installed/running and docker-emby with hw transcode but after a few hours the hw transcode does not work and makes software transcode, I have to restart the container for it to work again.

Some logs when it fails shows this:

07:54:09.253 ffmpeg version 5.1-emby_2023_06_25 Copyright (c) 2000-2022 the FFmpeg developers and softworkz for Emby LLC
07:54:09.253 built with gcc 10.3.0 (crosstool-NG 1.25.0)
07:54:09.254 Execution Date: 2024-04-23 07:54:09
07:54:09.264 [AVHWDeviceContext @ 0x26d0880] cu->cuInit(0) failed -> CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
07:54:09.264 Device creation failed: -542398533.
07:54:09.264 Failed to set value 'cuda=cuda:0' for option 'init_hw_device': Generic error in an external library
07:54:09.264 Error parsing global options: Generic error in an external library
07:54:09.264 EXIT

after_restart_ffmpeg-transcode-642aff78-dae8-40fd-815d-84c1cf2e1284_1.txt after_restart_embyserver (3).txt before_restart_embyserver (2).txt before_restart_ffmpeg-transcode-cf67089f-cf9e-41de-90df-7bf24088ff59_1.txt

[Luke 37111]

Hi, has the gpu session limit been reached perhaps?

[BlackDub 27]

8 hours ago, Luke said:

Hi, has the gpu session limit been reached perhaps?

No, it happens to me when it is not being used. I found other threads talking about it but I don't know the solution.
I try with docker emby and linux emby docker, same problem. i need a fix 

[BlackDub 27]

https://github.com/HaveAGitGat/Tdarr/issues/666#issuecomment-1368171781

https://github.com/jellyfin/jellyfin/issues/9287