Docker/Unraid - NVidia transcoding fail for some file

[nodiaque 62]

Hello,

I'm having an issue with transcoding that seems to affect only user on browser and when I'm using hardware encoding for transcoding. I have a NVidia Quadro RTX 4000 that I use for hardware transcoding, both decoding and encoding.

I tried Firefix, Edge, Chrome, all result in the samething. It's for a 4k HVEC video. I start to play and it's working fine, and suddenly it lag. When I check at the transcoding log, I see it failed at transcoding. Sometime, I end with a playback error and sometime it finally get through but will continue lagging. I tried replicating the problem on my lg tv by forcing a transcoding (lower bitrate) but I didn't had the problem so it seems to be a browser problem. Might have the same problem on google tv but I haven't tried.

I tried turning off hardware encoding only and it was playing fine.

I'm on unraid 6.11.5

Emby server: 4.7.13.0

I attached the transcoding log and the emby server log.

Thank you

emby_System_Logs_ffmpeg-transcode.txt System_Logs_embyserver.txt

[Luke 42077]

OK @softworkzwill take a look at this. Thanks.

[nodiaque 62]

I just read what I wrote and there's a typo at the end. But what I did is switch to software encoding (cpu) and no more problem. So something with the hardware encoding .

[softworkz 5066]

19 hours ago, nodiaque said:

I just read what I wrote and there's a typo at the end. But what I did is switch to software encoding (cpu) and no more problem. So something with the hardware encoding .

Did you set up Nvidia to be accessible from within the Docker container?

 

 

Does anybody know where we have instructions for this?

[nodiaque 62]

Yes it is, I used nvidia transcoding for the past 2 years on unraid with emby. Transcoding usely work fine but for some movie in 4K HDR, it crashes like that.

Also, I removed it only for H264 encoding, I still use it for H265 encoding and all decoding

Edited June 27, 2023 by nodiaque

[softworkz 5066]

Can you try to move the CUVID HEVC decoder to the top of the list (don't forget Save) and retry?

Is this a Dolby Vision format? ffmpeg detects it as SDR yuv420p (not HDR).

Can you show an ffmpeg log with software transcoding?

[nodiaque 62]

OK, I'll get back shortly with the info

[nodiaque 62]

OK, If I put CUVID on top in decoder, I'm getting lot's of lag on playback.

This time, the video is 4k Dolby vision + HDR10.

Last time, it was just h265 video file

So using CUVID Encoder and/or Decoder, I get lot's of lag.

If using NVDEC Decoder H265 with H264 software encoding, no problem.

If I do NVDEC Decoder H265 and NVDEC H264 encoder, I lag.

I cannot say for NVDEC 265 encoder, I think I never saw it being used, it always goes from 265 to 264 even when it's just a bitrate problem (that's also another question, why when 265 is supported but just want a lower bitrate, it doesn't use 265 encoding to transcoding 265). 

Attach is a log file of the 4k DV + HDR10 video using NVDEC decoder x265 and Software x264 encoder and another one with the CUVID x265 decoder that lag (seems to crash cause I have like 1 log file each 5 sec)

softwaretranscode_emby_System_Logs_ffmpeg-transcode-215dd810-a0b2-47fb-a711-5b7488bc6f02_1.txt_.txt cuvid decoding_emby_System_Logs_ffmpeg-transcode-8a72c9e3-ab77-46a5-aead-e475c6acc3f1_1.txt

[softworkz 5066]

The problem here is tone mapping. Software tone mapping is expensive, especially when on 4k frames without downscaling.

What's additionally expensive is transferring 4k frames from GPU memory to system memory, because those frames are uncompressed (about 120MB per 4k frame).
This is what happens in case when the CUVID decoder is chosen, and that why this is much slower.

In the case where you chose the NVDEC Emby had an optimization to do software decoding because it's better in this case, so there was no HW involved at all in this case.

---

You did not include a log with NVENC enabled. Ideally, it should perform everything in hardware: decoding, tone mapping and encoding, so we'd need to find out why this doesn't work.

Please make sure you have hardware tone mapping enabled as well as the NVENC encoder.

Thanks

PS: We do not use x265 for streaming, only for the conversion feature. You will find the reasoning in all detail elsewhere here in the forums.

[nodiaque 62]

The first post have a log with NVENC enabled, this is what I used before.

Thanks for the information on x265 encoding, I'll be sure to check back on this.

I'll rerun tests scenarios cause I did mix a lot of stuff here and I want to be sure to understand everything.

I'll start with the same video as original post, which is a x265 video file no hdr, no dv. This mean no tone mapping. I had issue with this one when I was using NVENC as encoder and decoder (this is what the log are in the first post).

So here's the test I'll do and post result.

1) 4K H265, no HDR, no DV

1-A) - NVENC X265 decoder, NVENC x264 Encoder

1-B) NVENC x265 decoder, CUVID x264 Encoder

1-C) NVENC x265 decoder, Software x264 encoder

1-D) CUVID X265 decoder, NVENC x264 Encoder

1-E) CUVID x265 decoder, CUVID x264 encoder

1-F) CUVID x265 decoder, software x264 encoder

2) 4k H265 DV + HDR10 profile

2-A) - NVENC X265 decoder, NVENC x264 Encoder

2-B) NVENC x265 decoder, CUVID x264 Encoder

2-C) NVENC x265 decoder, Software x264 encoder

2-D) CUVID X265 decoder, NVENC x264 Encoder

2-E) CUVID x265 decoder, CUVID x264 encoder

2-F) CUVID x265 decoder, software x264 encoder

 

Reason for me to do this is because I'm feeling something is not according to what you just explain. Like I have problem with NVENC decode and encode together, while I had problem with CUVID Decode with anything encoder. There was so much thing going on (I had other user on the server too so now I'll test with no one to be sure I get the right logs), this way I'll have a clear picture.

 

edit: there's no such thing as CUVID encoder so these test are removed

Edited June 28, 2023 by nodiaque

[softworkz 5066]

I think it might be a good idea to install the Transcoding Test Plugin. It allows to run a wide range of tests on a fixed set of test files and conditions. This can save you a lot of work and makes it much easier for me to diagnose when I see the full set of tests:

 

 

[softworkz 5066]

The UI is currently broken in several ways. Especially the display of lists is confusing. 

I'm still waiting for @Luketo apply the fixes for this.

[nodiaque 62]

OK, I've redone my test.

One thing to note, it doesn't happen with all x265 file, which is another weird thing.

Video 4k no hdr no dv, there's subtitle in MOV format (might be the issue, it's Embedded subtitles?).

Video 4k dv + hdr profile, subtitles SRT embedded

 

4k dv hdr10 - NVDEC NVENC - crash and stop
4k dv hdr10 - CUVID NVENC - crash and stop (video stop playing but logfile show it's still transcoding but video doesn't play need to refresh browser)
4k dv hdr10 - NVDEC software - all good (log files says no tone mapping but it's enable with the option either hardware or software, so why no tone mapping?)
4k dv hdr10 - CUVID software - lag playback
4k dv hdr10 - Software Software - no problem

4k no dv, no hdr - NVDEC NVENC - crash after a while or lag and will crash on seek
4k no dv, no hdr - CUVID NVENC - no problem
4k no dv, no hdr - NVDEC software - no problem
4k no dv, no hdr - CUVID software - no problem
4K no dv, no hdr - Software software - no problem

 

4k no dv no hdr - CUVID-Software.txt 4k no dv no hdr - movie info.txt 4k no dv no hdr - NVDEC-NVENC.txt 4k no dv no hdr - NVDEC-Software.txt 4k no dv no hdr - Software-Software.txt 4d dv hdr - NVDEC-NVENC.txt 4k dv hdr - CUVID-NVENC.txt 4k dv hdr - movie info.txt 4k dv hdr - NVDEC-Software.txt 4k dv hdr - Software-Software.txt 4k no db no hdr - CUVID-NVENC.txt

Edited June 28, 2023 by nodiaque
anonymise files

[nodiaque 62]

oh, just saw when I posted you replied something. Tell me if you want me to still do the other test plugin you ask.

For information, these are my current transcoding setting in both Transcoding and Advanced Transcoding

 

[softworkz 5066]

I got your files. I deleted them from the forums because they are showing your server url.

Edited June 28, 2023 by softworkz

[nodiaque 62]

yeah I saw that, I edited them and re-added them with the url removed.

[softworkz 5066]

 Ok, let's drill this down - looking at the HDR10 movie first:

16 minutes ago, nodiaque said:

4k dv hdr10 - NVDEC NVENC - crash and stop
4k dv hdr10 - CUVID NVENC - crash and stop (video stop playing but logfile show it's still transcoding but video doesn't play need to refresh browser)

From the transcoding side, it's all perfect. No crash, no stop. Decent performance (> 5x).

16 minutes ago, nodiaque said:

4k dv hdr10 - NVDEC software - all good (log files says no tone mapping but it's enable with the option either hardware or software, so why no tone mapping?)
4k dv hdr10 - Software Software - no problem

Both cases are identical. (Like I explained above, it doesn't use hw decoding because it would be worse - see the case with cuvid decoding)

But sw tone mapping is being used:

 

 

16 minutes ago, nodiaque said:

4k dv hdr10 - CUVID software - lag playback

Yes, like explained above.

[softworkz 5066]

As for why it doesn't play in the browser: This is most likely due to the bitrate. We had some calculation in place that sets the double input bitrate for the output when transcoding from h265 to h264 (changed in the beta channel).

In your example, this is 25 Mbps to 50 Mbps and that's usually too much for browser playback. SW transcoding works because it doesn't "obey" those parameters and usually produces lower bitrates.

Please try changing the quality setting from auto to some lower bitrate, I'm sure it will play then.

[nodiaque 62]

yeah didn'T see anything in the log for the movie that crashed, but playback just stop, then I can see another transcoding log that says the exact same thing but movie still doesn't play. After a while, the playback get paused and even if I press play, nothing happen. I have to refresh browser. I did see a "playback error" in the dashboard.

I just saw what you meant for if I disable HW Encoding, it doesn't use it in decoding. nvidia-smi and log clearly show that nothing is happening in the gpu, didn't catch that before.

I'll try changing quality like you said with NVDEC + NVENC

 

[nodiaque 62]

ok just tried with the 4k dv hdr10 movie file.

Lowest setting possible on 4k is 40mbps, that freeze completely after some time, no error and can see the transcoding is still ongoing in the log file (like last time)

At 1080p 25mbps and 30mbps, no problem after 5 minutes

At 1080p 40mbps, it froze

At 1080p 50mbps, it didn't want to put this one and it auto-switch to 4k 40mbps.

That's why NVDEC-NVENC

 

Edited June 28, 2023 by nodiaque

[softworkz 5066]

For the other case (non-HDR), I think I remember now that there's a bug when transcoding NVDEC/CUVID to NVENC without any filter in-between.

This was is fixed in the beta channel long ago. The 4.7.x versions were never expected to live for that long, so that's a bit unfortunate now, as we're not making any more changes to 4.7.x (unless it really has to be), so all I can suggest is 

• Try the beta channel
  or
• Wait for the 4.8 release
  or
• Reduce quality so it scales down to 1080p. Then it will work.

Thanks

[softworkz 5066]

Just now, nodiaque said:

ok just tried with the 4k dv hdr10 movie file.

Lowest setting possible on 4k is 40mbps, that freeze completely after some time, no error and can see the transcoding is still ongoing in the log file (like last time)

At 1080p 25mbps and 30mbps, no problem after 5 minutes

At 1080p 40mbps, it froze

At 1080p 50mbps, it didn't want to put this one and it auto-switch to 4k 40mbps.

Yes, the browsers are limiting this. 

A few years ago I was able to achieve higher bitrates by increasing the browsers' cache size, but I'm not sure whether this would still work.

[nodiaque 62]

ok, I see. Is there a time table for 4.8 out of beta?

I tried to find the plugin you linked but can't find it in the catalog. I found emby diagnostic though.

[Luke 42077]

1 hour ago, nodiaque said:

ok, I see. Is there a time table for 4.8 out of beta?

I tried to find the plugin you linked but can't find it in the catalog. I found emby diagnostic though.

Hi, no ETA but we're working hard on it.

[softworkz 5066]

1 hour ago, nodiaque said:

I tried to find the plugin you linked but can't find it in the catalog. I found emby diagnostic though.

My mistake. It's only available for the beta, sorry for the confusion.