HEVC Profile Main 10 / QSV Hardware Transcoding / staircase effect

[SoGX 4]

Hello everyone,

I currently have a problem with hardware transcoding when I use the QSV hardware transcoding.
This can be seen when transcoding HEVC videos with the profile "Main 10", i.e. 10 bits. Here, staircase effects occur during transcoding. If I use the software encoder, everything works perfectly.
Likewise, everything works well with HEVC videos with the profile "Main", i.e. 8 bit.

What could be the reason for this, or how can I solve the problem?

 

[SoGX 4]

Hallo zusammen,

beim Transcodieren von H265 Videos mit demrofil "Main 10" erscheinen unschöne Treppeneffekte.

Beim Software Transcoding treten diese nicht auf, ebenfalls nicht bei H265 Videos mit dem Profil "Main"

Edited June 30, 2022 by SoGX

[Luke 32482]

@SoGX

 

Hi there, let's look at an example. Please attach the information requested in how to report a media playback issue. Thanks!

 

[SoGX 4]

@Luke Send you a pm with a link to a sample file.

 Hi, attached the Server Log and two Transcode Logs, one with QSV and one with Software Transcoding.

embyserver.txt ffmpeg-transcode-ef0e82a9-6981-4466-b93e-dfe244029360_1.txt ffmpeg-transcode-100c1632-d484-4bd0-b9ca-375f5ace6753_1.txt

[visproduction 70]

The Original media is 1920 x 804 which is almost exactly 2.39 aspect ratio. (2.3880597)

The output on this line of the first ffmpeg-transcode text shows 720 x 300 is a slightly different aspect ratio: 2.4.  This is often a compromise ratio used for encoding.

  21:54:34.437   Stream #0:0: Video: h264, qsv(progressive), 720x300 [SAR 1:1 DAR 12:5], q=2-31, 4616 kb/s, Level 30, 23.98 fps, 90k tbn

For this stream output playback in 720 x 300 everything will be compressed vertically by 0.5%.  This could be the problem that shows up with the moving logo at the beginning of the video.In addition reducing the pixel size to 37.5% of the original introduces additional pixelization.  It would be preferred to make the reduction size to exactly either 50% or 25% which converts 2 pixels to 1 or 4 pixels to 1.  Your size choice converts 200 pixels to 75 or 10 pixels to 3.75.  When you get into non-whole numbers, the process is to blur across pixels to solve the size change.  Your resize choice is causing the pixelization and it's more noticable with zooming logo text.

Try either forcing the output to not change the original aspect ratio at all and see if that helps.  I don't know where that option is.  I only use dedicated encoders to transcode.

Hope that helps.

[Luke 32482]

OK @softworkz will take a look at this. Thanks.

[SoGX 4]

@softworkz

Hello,

Thank you very much for the explanation. Of course, that explains the poor quality. Unfortunately, I have little influence on it. The difference between the first and the second stream is simply that I have activated and deactivated the hardware transcoding.
The output format is set to 1080p 5MBit in both cases.

As far as I can tell, the problem only occurs with films with the profile "Main 10".

[SoGX 4]

Same Problem with 4k untouched BluRay material.

ffmpeg-transcode-819f11da-07f0-4475-ab8e-3d24670723a9_1.txt

[SoGX 4]

With jellyfin installed on the same server it works like expected.

I attached the jellyfin transcoding log, maybe this can help.

Log.txt

Edited July 9, 2022 by SoGX

[softworkz 2003]

@SoGX - this is a self-made problem.

You have configured CRF 18 for the software encoder:

and you made no settings for the QSV encoder.

With default setting, the libx264 encoding quality is already a bit better than than the hw encoders, now when you tweak libx264 for better quality, the gap will become even more visible.

It's not your fault, though. In the current version, the UI doesn't give you the slightest hint that the setting you make will affect sw encoding only. It's a long-term flaw, unfortunately.

You can change QSV encoding parameters by installing the Diagnostics plugin (another point that we'll hopefully be able to improve soon).
Then, go to "Advanced Transcoding" and click on the buttons next to the QSV H.264 encoding to change QSV encoding quality.

[SoGX 4]

@softworkzThanks for your answer and for the tips

I have installed the Diagnostics Plugin and made a few adjustments in the h264 QSV settings.
Unfortunately, the quality of the video remains poor.
What surprises me is that the problem only affects 10bit h265 videos, the problem does not occur with 8bit h265 videos.
I have sent you a pm with a link to a sample video, maybe that will help a little.

 

 

[SoGX 4]

Hello,

is there any news regarding this problem. Could it be reproduced?
As I see it, the problem seems to be that 10 bit videos seem to be played back in 720 x 300 resolution as soon as you transcode them with QSV (in my case 1080p with 5MBit).

Edited July 18, 2022 by SoGX

[Luke 32482]

Hi, yes, @softworkz is looking into it. Thanks.

[softworkz 2003]

On 7/5/2022 at 7:03 PM, visproduction said:

The Original media is 1920 x 804 which is almost exactly 2.39 aspect ratio. (2.3880597)

The output on this line of the first ffmpeg-transcode text shows 720 x 300 is a slightly different aspect ratio: 2.4.  This is often a compromise ratio used for encoding.

Well spotted. 

It's not a "compromise ratio", though, it's about math. Width and height values need to be multiples of 2.

720 / 1920 * 804 = 301.5.

Now, 302 would be closer to 301.5 than 300, but experience has shown that it is safer to round to the lower value because rounding to the higher can sometimes introduce an ugly line of undefined color at the bottom.

ffmpeg.zip

[softworkz 2003]

@SoGX - I don't see that effect with the exact same command line.

Step 1

Please install the latest drivers from the Intel website.
Retry.

Step 2

Please replace the ffmpeg version with the one attached.
Retry.
(yours is located in C:\Users\Administrator.INTERN\AppData\Roaming\Emby-Server\system\)

ffmpeg.zip

[SoGX 4]

@softworks Can it be a problem that has to do with the hardware?

1. graphics driver has been updated.

2. ffmpeg version has been replaced by the attached one.

Problem still exists

embyserver.txt hardware_detection-63793933796.txt ffmpeg-transcode-9a55a631-7546-4e30-936f-dc97c3ea5e8c_1.txt

[softworkz 2003]

I'm not sure what it could be at this point. Where/how do you view the output? (client)

Let's try this:

• Got to your transcoding temp (C:\Emby-Cache\transcoding-temp)
• Delete everything
• Start playback with hwa
• Zip the first 10 generated segments
• Post the zip here (or PM)

 

[SoGX 4]

On the other hand, if it's a hardware problem, why does it run flawlessly with Jellyfin on the same server? Also with hardware transcoding.
I use Firefox, Chrome and a FireTV as clients.

Transcode.zip

[softworkz 2003]

Thanks. Could you please install the diagnostics plugin and make the following settings for Parameter Adjustment:

[softworkz 2003]

PS: Yes, there is different behavior of the hardware.

[SoGX 4]

I have made the settings in the plugin, unfortunately the same result.

 

ffmpeg-transcode-1cec6371-7162-4dcc-8817-677537f9afc2_1.txt Transcode_2.zip

[softworkz 2003]

Thanks, let's try some  more replacements:

Test 1

Find: -level:v:0 30
Replace: -level:v:0 51

Test 2

Find: -load_plugin:v:0 hevc_hw
Replace:            (leave empty)

Test 3

Find: vpp_qsv
Replace:  scale_qsv

Test 4

Find: vpp_qsv@f1=
Replace:  vpp_qsv@f1=scale_mode=2:

 

You don't need to show results for any of the tests. Just let me know whether any of those will make a change.

Edited July 20, 2022 by softworkz

[SoGX 4]

Test 1,2 and 4 do not seem to make much difference. In test 3, the staircase effect no longer seems to be there, but the image looks very coarse.

[softworkz 2003]

Test 5

Find: vpp_qsv@f1=
Replace:  scale_qsv@f1=mode=2:

(you can also try 1 for the mode with both filters)

 

 

[SoGX 4]

The results in both cases (mode 1 and 2) are comparable to those in test 3.