CentOS 7 - 1050 Ti - Transcoding Detection Issue

[nterupt 0]

Hello,

 

I am currently running the latest version of Emby inside a VM on ESXi 5.5.

 

I have passed through an NVIDIA 1050 Ti to this VM and installed the drivers.

 

Try as I might, it does not appear that Emby is detecting that it can use this video card for transcoding.

 

I have verified that my driver is installed correctly and rebooted numerous times.  I have also verified that the emby user is part of the video group. Here is the output of "nvidia-smi"

 

Tue Mar 31 04:33:38 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64       Driver Version: 440.64       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 105...  Off  | 00000000:13:00.0 Off |                  N/A |
| 35%   44C    P0    N/A /  75W |      0MiB /  4040MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Any help you can provide would be greatly appreciated.

hardware_detection-63721224942.txt

Edited March 31, 2020 by nterupt

[Luke 37065]

HI there @@nterupt, hardware transcoding in VM containers is only experimental right now, so we may not yet have all of the answers.

 

At the very least you said you've enabled the GPU in the VM, but are you sure you've done that correctly? Are you able to run vainfo to see if it detects it?

[nterupt 0]

Hi,

 

I installed vainfo by running "yum install va-utils", but wasn't already installed.

 

I get the following output:

 

"error: can't connect to X server!
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns -1
libva error: va_getDriverName() failed with unknown libva error,driver_name=(null)
vaInitialize failed with error code -1 (unknown libva error),exit"

 

Do I need to have an X11 windowing system installed to make this work (Gnome, KDE)?  Currently, since this is a headless installation, I've been using a minimal server installation without X11 to conserve resources.

[Luke 37065]

Did you check out our hardware acceleration setup guide?

https://support.emby.media/support/solutions/articles/44001160148-hardware-acceleration-overview

[nterupt 0]

Yes, that just said to install the official NVIDIA drivers which I did.  It also says that Emby can run headless (no monitor connected), but it doesn't mention if X windows is required.

 

Are there any other debugging steps or logs I can handoff that would tell you all more of what is not working correctly? I'm not entirely sure what mechanism Emby is using to determine if it has detected my card.  But if you give me some details I can run with that to investigate further how my system is deviating from what Emby is usually expecting.

 

For example emby runs API call X, and if that returns an error, we ignore the device.

[Luke 37065]

It uses the same things to detect it as ffmpeg, so essentially, if emby can't detect it then ffmpeg won't be able to utilize it. So that's why I asked to try with vainfo.

 

I can't say for sure if X11 is required. @@softworkz will know more.

[softworkz 3335]

@@nterupt - vainfo does not apply to Nvidia GPUs, so you can forget about all of its output.

 

From the detection log I can see that the Nvidia board is detected but there's an error as soon as it tries to use it.

 

I'm afraid, I can't help with Docker environments. My suggestion would be to install Emby to the host OS directly.

[Luke 37065]

Have you taken a look at this?

https://emby.media/community/index.php?/topic/76937-docker-hwa-nvidia-instructions/

[nterupt 0]

Hey All,

I am back with this issue.  I had done some research that told me that NVIDIA drivers don't work properly when Geforce cards are passed through to a virtual machine.  So, I ended up getting a P2000 and am still running into this issue. This VM is CentOS 7 with 8 vCPUs and 32GB of RAM allocated

nvidia-smi output:

[root@emby ~]# nvidia-smi
Sat Mar  6 20:35:46 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.56       Driver Version: 460.56       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P2000        Off  | 00000000:13:00.0 Off |                  N/A |
| 79%   54C    P0    19W /  75W |      0MiB /  5059MiB |      3%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

 

This part of the HW detection log sticks out to me:

"Devices": [
        {
            "DeviceIndex": 0,
            "DeviceInfo": {
                "Name": "Quadro P2000",
                "Description": "Quadro P2000",
                "ComputeCapMajor": 6,
                "ComputeCapMinor": 1,
                "Properties": [
                    "ClockRate": 1480500,
                    "MultiprocessorCount": 8,
                    "Integrated": 0,
                    "CanMapHostMemory": 1,
                    "ComputeMode": 0,
                    "ConcurrentKernels": 1,
                    "PciBusId": 19,
                    "PciDeviceId": 0,
                    "TccDriver": 0,
                    "MemoryClockRate": 3504000,
                    "GlobalMemoryBusWidth": 160,
                    "AsyncEngineCount": 2,
                    "UnifiedAddressing": 1,
                    "PciDomainId": 0,
                    "ComputeCapabilityMajor": 6,
                    "ComputeCapabilityMinor": 1,
                    "ManagedMemory": 1,
                    "MultiGpuBoard": 0,
                    "MultiGpuBoardGroupId": 0
                ]
            },
            "Error": {
                "Number": 801,
                "Message": "Failed creating CUDA context for device 0"
            }
        }
    ],

Any ideas how I could go about debugging this error message?  Any new debugging mechanisms crop up in the past year or so?

Only idea I have is to upgrade to a more recent version of ESXi, I'm currently on 5.5, maybe 6.7 or 7.0 will have some heretofore unknown fix.

Are their commands I can run from my own command line to get this same error?

hardware_detection-63750654687.txt

[softworkz 3335]

On 4/1/2020 at 8:55 AM, softworkz said:

I'm afraid, I can't help with Docker environments. My suggestion would be to install Emby to the host OS directly.

...as the very first step.

[nterupt 0]

Alright, I'll look into that.  I've had Emby virtualized for quite a long time, and have the server running a lot of other services as well.  Was looking to avoid building an entirely dedicated Emby PC just to get HW transcoding, but maybe I can scrap something together from spare parts just for testing.

[softworkz 3335]

That won't move you any step further.

What does it hurt to install Emby on the host OS, check the detection and see whether it's working there... You don't need to setup libraries nor any users connecting to it...

[nterupt 0]

Sorry, I think there is a misunderstanding.  The server is running the ESXi 5.5 Hypervisor (not an OS that can run Emby).  On top of that I am running a CentOS guest operating system.  I am using ESXi HW passthrough to give this virtual machine direct access the Quadro P2000.  Emby is running in this CentOS guest operating system.

There is no Docker whatsoever in this setup.  Therefore, if I don't have any options to debug this guest operating system, my next step is to go with a bare-metal install (since I can't use this GPU or run Emby directly in the ESXi operating system), which would have to use different HW.

Does that make sense?

Edited March 7, 2021 by nterupt
Adding Wikipedia links to ESXi for reference

[softworkz 3335]

Sorry got ya! Makes perfect sense. I had written Docker above for whatever reason and from there on, I had been relying on my own writing.

In that case, I have only miscellaneous ideas left to offer:

• Try putting the Quadro into different PCIx slots (when possible)
• Check out BIOS settings regarding PCI and graphics
• Does the Quadro work as graphics card for the guest OS, when you connect a monitor?
• Could it be a permissions issue? Emby is running under its own user account "Emby" maybe it's not allowed to access the GPU..
• The Nvidia SMI tool is not a good test for verifying Nvidia driver installation and functionality
  I've seen many cases where SMI was working but everything else wasn't 
  For testing, I would rather look for some Nvidia 3D demo that also mentions use of CUDA 

[nterupt 0]

My current theory is that ESXi is not properly passing through the GPU to my guest operating system (CentOS).  But, I have two angles of investigation.
 

1) Upgrade ESXi to later, more supported versions that might have a fix (if ESXi is the culprit).

2) Set up a new temporary Emby server running CentOS on bare metal hardware with the P2000 just to make sure the GPU and OS configuration I am using on the ESXi server is working.  If Emby still doesn't detect the card then it's either the P2000, Emby, or CentOS, but I've eliminated ESXi as a problem.  Then I can iterate on OS, different GPUs from there to debug further.  Everything I've read thus far says that this should work on CentOS with this card, and my installation is good, so I don't think this is likely to find a problem.

 

[nterupt 0]

Sorry, just saw your reply.  Those are good ideas.  I'll try some of those, maybe assigning the GPU to a Windows Guest OS and doing some testing there to eliminate CentOS and Emby as sources of the problem.

[softworkz 3335]

2) is basically the original idea that I was suggesting (even though I had thought it was Docker)

[nterupt 0]

Well, I was able to resolve the issue.  Just for posterity, in case anyone else encounters this, I upgraded ESXi.  I went from ESXi 5.5 to a much more recent ESXi 6.7.  Upon booting the Emby VM with the P2000 GPU passed through, it all worked correctly as expected.  Not a very satisfying answer, but I'm guessing they fixed a critical bug somewhere in there.

Thanks @softworkz for your input on this issue, I appreciate it.

[softworkz 3335]

I'm glad to hear that it's fixed.

Thanks for the feedback!