Transcoding in Linux - only 1 GPU transcode running?

[BirdBare 9]

Hello,

I am running emby in a Kubernetes container on Truenas scale. I seem to be only getting 1 GPU transcode where I know my device offers 2 max.

GPU - Nvidia P620.

I've attached an image and log files

I might have missed a setting so I apologize if so :)

 

ffmpeg-transcode-f41c8f3f-9574-44c1-a0f2-24dd8ace4560_1.txt ffmpeg-transcode-8f53af99-5fc5-4562-9308-edb4167a9065_1.txt embyserver(1).txt

[BirdBare 9]

I should have also said that I ran the patch for unlimited transcodes as well.

[Happy2Play 9780]

To me it looks like a limitation in the card.  But the devs would have to comment more.

decoder->cvdl->cuvidCreateDecoder(&decoder->decoder, params) failed -> CUDA_ERROR_OUT_OF_MEMORY: out of memory

But per the Nvidia matix you only have 1 decoder.

Video Encode and Decode GPU Support Matrix [NEW] | NVIDIA Developer

But have no idea how the patch works.

Edited November 10, 2021 by Happy2Play

[Luke 42077]

These are also both tone mapping, which is a very intensive process. Are you sure it can handle two of that at once?

[BirdBare 9]

Oh so it is a memory error? It should be able to handle 2 transcodes at once. I will try without tonemapping. If memory is an issue then I will get a p600 with 4gb ram.

Is the out of memory error from ffmpeg?

[BirdBare 9]

Hmm. So I tried without tone mapping but still only get a single transcode. How much memory does a 4k transcode typically use?

[Luke 42077]

@softworkz may have some insight. Thanks.

[softworkz 5066]

4 hours ago, Luke said:

These are also both tone mapping, which is a very intensive process

That's true, but only when done in sw. The CUDA tone mapping is just about nothing for an Nvidia card.

[BirdBare 9]

2 minutes ago, softworkz said:

That's true, but only when done in sw. The CUDA tone mapping is just about nothing for an Nvidia card.

Good to know! Do you think the issue is available ram then? I can get 5 1080p he transcodes but only 1 4k.

Edited November 10, 2021 by BirdBare

[softworkz 5066]

35 minutes ago, Luke said:

@softworkz may have some insight. Thanks.

@BirdBare - Let's find out whether it's a memory issue or an NVDEC limitation.

For testing that, choose a video with a small frame size, e.g. 720p, choose an even lower quality in the clients (to make sure it transcodes) and try to play simultaneously.

[BirdBare 9]

Just now, softworkz said:

@BirdBare - Let's find out whether it's a memory issue or an NVDEC limitation.

For testing that, choose a video with a small frame size, e.g. 720p, choose an even lower quality in the clients (to make sure it transcodes) and try to play simultaneously.

See above message for 1080p testing.

[softworkz 5066]

1 minute ago, BirdBare said:

Good to know! Do you think the issue is available ram then? I can get 5 1080p he transcodes but only 1 4k.

OK, that already answers the question. A 4k frame requires 4x the memory of a 1080 frame.

[BirdBare 9]

So I should at least get two 4k transcodes according to my recent test. I'll push it and see the max 1080p transcodes for this card.

 

EDIT: I'm wrong above. I need to test the max. Hopefully I can hit 8 simultaneous.

Edited November 10, 2021 by BirdBare

[softworkz 5066]

Just now, BirdBare said:

So I should at least get two 4k transcodes according to my recent test. I'll push it and see the max 1080p transcodes for this card.

Only when you could do 8 parallel 1080 transcodes.

But you can't exactly calculate like that. It also very much depends on the amount of intermediate processing steps:

>>>>>>  Processing Plan
Info    Name                                        CanDoInHardware      WillDoInHardware     Reason                                  
Info    NVDEC Quadro P620 - H.265 (HEVC)         >> True                 True                 Hardware Codec                           
Info    VideoInput                               >> True                 True                 Matching hardware context                
Info    Scaling                                  >> True                 True                                                          
Info    ToneMapping (when possible)              >> True                 True                                                          
Info    VideoOutput                              >> True                 True                 Hardware encoder                         
Info    NVENC Quadro P620 - H.264 (AVC)          >> True                 True                 Hardware Codec                           

From one-step to another, it takes memory for the intermediate frames while they travel through the pipeline.

[softworkz 5066]

I almost forgot: Your 4k videos are 10bit and your 1080 videos are probably 8bit..

[BirdBare 9]

Understood. It's sad to hear. I'll do the test but it may be best to look for a card with more ram.

[softworkz 5066]

One thing we could try is to reduce parallelization in the pipeline processing, maybe(!) that helps reducing the used memory.

To test this, could you please install the diagnostics plugin from the plugin catalog?

[softworkz 5066]

TEST 2

Do as follows:

Under "Diagnostic Options", you'll see two find-replace text boxes for modifying ffmpeg commands:

Search: -filter_complex 

Replace: -threads:v:0 1 -filter_complex_threads 1 -filter_complex 

Edited November 10, 2021 by softworkz
Simplified instructions

[softworkz 5066]

Note that I edited the previous post.

[softworkz 5066]

BTW, you can also use the nvidia-smi command line tool to check GPU memory usage.

[BirdBare 9]

Hello!

So I tried the above command and did not see any reduction in ram usage. I attached the logs just in case it didn't apply properly.

With the nvidia-smi command I see about 1300mB ram usage for 1 4k transcode. So indeed it is a ram issue. That is super frustrating to say the least. Do you have any more suggestion to reduce ram usage with software?

I share my 4k library with family so 2 transcodes would be perfect if we can manage.

embyserver(2).txt ffmpeg-transcode-09b25518-b998-4551-bb60-08efe758e14a_1.txt

[BirdBare 9]

I forgot to mention. I really appreciate the help thus far! I absolutely love this community.

[softworkz 5066]

43 minutes ago, BirdBare said:

Hello!

So I tried the above command and did not see any reduction in ram usage. I attached the logs just in case it didn't apply properly.

With the nvidia-smi command I see about 1300mB ram usage for 1 4k transcode. So indeed it is a ram issue. That is super frustrating to say the least. Do you have any more suggestion to reduce ram usage with software?

I share my 4k library with family so 2 transcodes would be perfect if we can manage.

embyserver(2).txt 121.58 kB · 0 downloads ffmpeg-transcode-09b25518-b998-4551-bb60-08efe758e14a_1.txt 24.21 kB · 0 downloads

The diagnostic options change did not apply in fact.

Those settings don't survive reboots. Please don't forget to click Save and don't reboot after setting the diag options.

[BirdBare 9]

6 hours ago, softworkz said:

The diagnostic options change did not apply in fact.

Those settings don't survive reboots. Please don't forget to click Save and don't reboot after setting the diag options.

How about now? I didn't see a difference in ram usage still so I am hesitant to say I did it correctly.

 

 

embyserver(3).txt ffmpeg-transcode-5b0edcdb-2943-43ea-8387-686cb3d78b0b_1.txt ffmpeg-transcode-7b8c5cd6-f93c-4d64-a399-dadddefdead6_1.txt

[BirdBare 9]

Just now, BirdBare said:

How about now? I didn't see a difference in ram usage still so I am hesitant to say I did it correctly.

 

 

embyserver(3).txt 487.67 kB · 0 downloads ffmpeg-transcode-5b0edcdb-2943-43ea-8387-686cb3d78b0b_1.txt 64 kB · 0 downloads ffmpeg-transcode-7b8c5cd6-f93c-4d64-a399-dadddefdead6_1.txt 42.33 kB · 0 downloads

Oh I searched the text. It applied. Still only 1 transcode so far