VAAPI Transcoding quality issue

[EHRETic 2]

Hi there 

I'm currently in the process of moving away from Plex to Emby, I've my reasons but please don't forget about this below, I don't want to hurt any sensibility.
So I recently noticed on a movie that I had huge artefacts/jumps when looking a film on my Chromecast.

Some informations about my setup:

• Hardware is a Minisforum MS-01 Core i9-13900H, 96GB RAM and 2 TB disk 
• OS is Almalinux 9 server
• I run Emby & Plex instances on Docker on this computer, GPU is "passthroughed" (with: - /dev/dri/renderD128:/dev/dri/renderD128)
• Emby version is 4.8.8.0
• I didn't install any extra drivers 

Test protocol:

• The hardware is the same, so are Docker passthrough settings for Emby & Plex 
• Movie is always the same, so the sequence (in my case, it's Hunger Games last episode, when Katniss is going alone to Panem in the plane, hiding behind a crate (sequence @ 29:00)
• Movie file is in 4K HDR 10 HEVC 

I started to look a little bit deeper because I suspect transcoding quality issues and this is what I found so far:

• Emby in Quicksync transcoding : very bad quality, the worst probably, a lot of artifacts (forced by deselecting VAAPI from advanced settings)
• Emby in VAAPI transcoding - VAAPI HW tone mapping : really dark, almost impossible to see artifacts, so I'll will not retain this
• Emby in VAAPI transcoding - Extra-T OpenCL tone mapping with Hable algorithm: quite a lot of artifacts, flat spots in dark areas, jumps
• Emby in VAAPI transcoding - Extra-T OpenCL tone mapping with Reinhard algorithm: seems better than Hable, but not really good
• Now : Plex : Auto quality with slow transcoding preset : no issue
• My Emby test instance : VM with Nvidia GPU passthroughed from ESX host to VM to Docker instance, NVDEC & Hable : no issue either

I've to precise that I could always watch my movie, there is no play issue, only quality but I'd like to sort that out.

It is also hard to describe it, you really have to see it.

So, where do we start?

Edited September 19, 2024 by EHRETic
Forgot Emby version

[adminExitium 260]

 

[guunter 37]

3 hours ago, EHRETic said:

Hi there 

I'm currently in the process of moving away from Plex to Emby, I've my reasons but please don't forget about this below, I don't want to hurt any sensibility.
So I recently noticed on a movie that I had huge artefacts/jumps when looking a film on my Chromecast.

Some informations about my setup:

• Hardware is a Minisforum MS-01 Core i9-13900H, 96GB RAM and 2 TB disk 
• OS is Almalinux 9 server
• I run Emby & Plex instances on Docker on this computer, GPU is "passthroughed" (with: - /dev/dri/renderD128:/dev/dri/renderD128)
• Emby version is 4.8.8.0
• I didn't install any extra drivers 

Test protocol:

• The hardware is the same, so are Docker passthrough settings for Emby & Plex 
• Movie is always the same, so the sequence (in my case, it's Hunger Games last episode, when Katniss is going alone to Panem in the plane, hiding behind a crate (sequence @ 29:00)
• Movie file is in 4K HDR 10 HEVC 

I started to look a little bit deeper because I suspect transcoding quality issues and this is what I found so far:

• Emby in Quicksync transcoding : very bad quality, the worst probably, a lot of artifacts (forced by deselecting VAAPI from advanced settings)
• Emby in VAAPI transcoding - VAAPI HW tone mapping : really dark, almost impossible to see artifacts, so I'll will not retain this
• Emby in VAAPI transcoding - Extra-T OpenCL tone mapping with Hable algorithm: quite a lot of artifacts, flat spots in dark areas, jumps
• Emby in VAAPI transcoding - Extra-T OpenCL tone mapping with Reinhard algorithm: seems better than Hable, but not really good
• Now : Plex : Auto quality with slow transcoding preset : no issue
• My Emby test instance : VM with Nvidia GPU passthroughed from ESX host to VM to Docker instance, NVDEC & Hable : no issue either

I've to precise that I could always watch my movie, there is no play issue, only quality but I'd like to sort that out.

It is also hard to describe it, you really have to see it.

So, where do we start?

I was curious and tested with my test server intel nucs that also use VAAPI and found the same results. They look fine on my main server with an nvidia gpu

 

[Neminem 703]

Devs will need logs

[EHRETic 2]

Hi,

Here are some logs of the last play + server + hardware detection (please don't mind IP obfuscation) 

 

embyserver.txt ffmpeg-transcode-460fc5f9-f11c-4d44-951a-efa0bebe0790_1.txt hardware_detection-63862347396.txt

[EHRETic 2]

Hi there,

Took me a while to be able to "screenshot" the issue", but this is it: probably one or a few frames will completely be pixelized, making the visual experience pretty bad.

(Yes, this is a full HD movie )

 

[adminExitium 260]

You may also want to try the latest beta version once, just to compare.

[EHRETic 2]

1 hour ago, adminExitium said:

You may also want to try the latest beta version once, just to compare.

Nope, it didn't change a bit by trying 4.9.0.30-beta (but I tried )

[Q-Droid 831]

Have you considered upgrading the kernel to the mainline version? The one you're running did not include support for Raptor Lake CPUs and maybe a newer one could nudge things in the right direction.

For your containers it's simpler to map the /dev/dri path and not the individual render devices even if you list all of them. 

https://wiki.crowncloud.net/?Installing_the_Linux_Kernel_6x_on_AlmaLinux_9

 

[guunter 37]

3 hours ago, Q-Droid said:

Have you considered upgrading the kernel to the mainline version? The one you're running did not include support for Raptor Lake CPUs and maybe a newer one could nudge things in the right direction.

For your containers it's simpler to map the /dev/dri path and not the individual render devices even if you list all of them. 

https://wiki.crowncloud.net/?Installing_the_Linux_Kernel_6x_on_AlmaLinux_9

 

Yeah it worked for me. Upgraded my Proxmox kernel to 6.8.12-2-pve (running ember lxc) and now the colors are the same. Not sure about OP. 

[EHRETic 2]

5 hours ago, Q-Droid said:

Have you considered upgrading the kernel to the mainline version? The one you're running did not include support for Raptor Lake CPUs and maybe a newer one could nudge things in the right direction.

For your containers it's simpler to map the /dev/dri path and not the individual render devices even if you list all of them. 

https://wiki.crowncloud.net/?Installing_the_Linux_Kernel_6x_on_AlmaLinux_9

 

Hi did the kernel update an hour ago, but it didn't change a thing.
(F*** up all my network settings in the process )

I had an old P1000 in the computer, that is why I assigned directly the device renderD128, so I don't know if it can bring anything with /dev/dri but I'll try too.

[EHRETic 2]

OK, I think I found something: so none of changes above did anything (kernel, device presented), except one : "Chromecast max quality"...

Apparently, if I go above 1080 - 30mbps, I start to have way more issues, also "Auto" is really bad.
Reinhard still gives me better results compared to Hable.
With 1080/30mbps+Reinhard in VAAPI, it's close to Plex quality... 

And for what I see, Plex send the max bitrate of 27 "only", as it the max of the movie and it works smootly.

I'm still banging my head to understand the problem because my test instance (NVIDIA) send way above the bitrate of 1080 without a problem on the same Chromecast.

[guunter 37]

5 minutes ago, EHRETic said:

OK, I think I found something: so none of changes above did anything (kernel, device presented), except one : "Chromecast max quality"...

Apparently, if I go above 1080 - 30mbps, I start to have way more issues, also "Auto" is really bad.
Reinhard still gives me better results compared to Hable.
With 1080/30mbps+Reinhard in VAAPI, it's close to Plex quality... 

And for what I see, Plex send the max bitrate of 27 "only", as it the max of the movie and it works smootly.

I'm still banging my head to understand the problem because my test instance (NVIDIA) send way above the bitrate of 1080 without a problem on the same Chromecast.

Auto should be direct play at home if you have the bandwidth. Did you make sure to add your LAN subnet in the local network section?

[EHRETic 2]

6 minutes ago, guunter said:

Auto should be direct play at home if you have the bandwidth. Did you make sure to add your LAN subnet in the local network section?

Networks are in the right sections, but Auto makes things way worse.
I'm starting to wonder if I don't need a new Chromecast, but always, compared to Plex, it should not be necessary.

My CC is an Ultra (4K) connected to LAN (100 mbps - max speed possible on this device), Emby is on the same subnet.

[Q-Droid 831]

54 minutes ago, EHRETic said:

Hi did the kernel update an hour ago, but it didn't change a thing.
(F*** up all my network settings in the process )

I had an old P1000 in the computer, that is why I assigned directly the device renderD128, so I don't know if it can bring anything with /dev/dri but I'll try too.

Did QSV show up for you after the kernel upgrade? 

It's weird for a capable CPU to have such poor quality. 

 

[guunter 37]

4 minutes ago, EHRETic said:

Networks are in the right sections, but Auto makes things way worse.
I'm starting to wonder if I don't need a new Chromecast, but always, compared to Plex, it should not be necessary.

My CC is an Ultra (4K) connected to LAN (100 mbps - max speed possible on this device), Emby is on the same subnet.

Does it have the same issues on your other clients or just the CC Ultra? I think there's 2 emby apps on android? I have a Apple TV so cant confirm but i think theres Emby and Emby for TV.

[guunter 37]

@LukeHas it been asked before to get VPP tone mapping options like in Jellyfin? You can adjust VPP brightness and contrast.

[yocker 91]

Disable tone mapping and see what the result is.
At least that should narrow down the search for the problem.

[EHRETic 2]

Hi there,

Took me a while but I wanted to give some follow-up on this story.

So 2 things "solved" my issue there: 

• I replaced the movie with an higher quality version, the problem didn't came back
• I purchased a NVIDIA Shield (oh my god, the quality is so much better compared to Chromecast!)

So, there is no miracle there and I could not really pinpoint the issue... The quality of the movie was not optimal, the quality of CC 4K Ultra is not optimal either and there is for sure different methods between Emby and Plex.

At the end, my multimedia server is the only server in the house I don't want to troubleshoot, I just want to enjoy it

Thanks to anybody that helped me in this topic!