Failed Hardware Transcode Attempts Break Subsequent Conversions

[Netfool 20]

Running The Nvidia Shield Emby Server 4.5.1.0 on a 2019 Nvidia Shield Pro.

Attempting to use the Nvidia GPU to do hardware trans-codes has never worked for me.  Ever hopeful, I wanted to test under 4.5.1.0 and discovered a completely different problem.

I attempted to transode a file from H264 to HVEC.   That first conversion failed as before.  I also tried to convert the file to a 4MB/sec version.  Hardware trans-coding of that failed as well.  I did the same for 3 additional files to test whether or not it was a problem with that specific file.  All failed to used the hardware encoder.

However, when I went back and examined the logs, in all cases Emby attempted to convert the first file, NOT the file I asked it to convert.  Conversions were started from the Movie page in some cases and from the library thumbnail for the movie in others.

 

A screenshot of the Logs page and relevant logs are attached. (The screenhot to establish the order of the transcode logs).  The Remux log came from playing the last movie to confirm its was not originally shot in 16 by 9 (it was however released letter-boxed) and mostly in monochrome.    It's possible that the initial problem was caused by cancelling the first conversion from the dashboard.  Allowing it to complete software a software trans-code in all cases would have tens of hours.

embyserver.txt ffmpeg-remux-3b027453-a1da-4a8a-811f-c14d291c3ebe_1.txt ffmpeg-transcode-3cb5e847-7345-4bb5-b335-9a1e7e88e3c6_1.txt ffmpeg-transcode-8db60443-f8f7-4cc2-851e-aa53df25e619_1.txt ffmpeg-transcode-bd201aa7-eece-4a32-ae0c-3fe8ae75076a_1.txt ffmpeg-transcode-cc0c53cb-384f-47a0-92d2-8172e76f6c30_1.txt ffmpeg-transcode-d13fff65-36f7-48c7-b57a-9e29c7b38769_1.txt ffmpeg-transcode-d715900d-b7ae-4f42-8d84-a4a8d04c4ca2_1.txt ffmpeg-transcode-da1997bb-81b1-412e-ab21-10b73c21c3a5_1.txt

[Happy2Play 9351]

Looks like some adjustments adjustments will need to made.

12:55:35.293 Stream mapping:
12:55:35.293   Stream #0:0 -> #0:0 (h264 (h264_mediacodecndk) -> hevc (hevc_mediacodecndk))
12:55:35.293   Stream #0:1 -> #0:1 (dts (dca) -> aac (native))
12:55:35.293 Press [q] to stop, [?] for help
12:55:36.299 frame=    0 fps=0.0 q=0.0 size=       0kB time=-577014:32:22.77 bitrate=N/A throttle=off speed=   0x    
12:55:36.401 [h264_mediacodecndk @ 0x29d7947800] NdkDec: MediaCodec output format changed: mime: string(video/raw), stride: int32(1280), slice-height: int32(688), color-format: int32(21), image-data: data, crop: Rect(0, 0, 1279, 687), crop-width: int32(1280), crop-height: int32(688), display-width: int32(1280), display-height: int32(688), color-range: int32(2), color-standard: int32(4), color-transfer: int32(3), hdr-static-info: data, width: int32(1280), height: int32(688)}
12:55:36.403 [hevc_mediacodecndk @ 0x29d7949600] NdkEnc: is_rtk: 0 pix_fmt: 23 pixelFormat: 21
12:55:36.403 [hevc_mediacodecndk @ 0x29d7949600] NdkEnc: Mime: video/hevc Size: 1280x688 rc_mode: 1 rc_max_rate: 1680000 rc_buffer_size: 3360000 bit_rate: 200000
12:55:36.403 [hevc_mediacodecndk @ 0x29d7949600] NdkEnc: Framerate: 24000 / 1001
12:55:36.403 [hevc_mediacodecndk @ 0x29d7949600] NdkEnc: Profile: 8 Level: 512
12:55:36.403 [hevc_mediacodecndk @ 0x29d7949600] NdkEnc: aspect: 1 / 1
12:55:36.403 [hevc_mediacodecndk @ 0x29d7949600] NdkEnc: mediacodec_name: OMX.Nvidia.h265.encoder
12:55:36.415 [hevc_mediacodecndk @ 0x29d7949600] NdkEnc: Failed to configure encoder; check parameters
12:55:36.417 Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
12:55:36.421 [aac @ 0x29d7987500] Qavg: 13013.449
12:55:36.421 [aac @ 0x29d7987500] 2 frames left in the queue on closing
12:55:36.438 Conversion failed!

 

[Netfool 20]

20 hours ago, Happy2Play said:

Looks like some adjustments adjustments will need to made.

Nothing I can adjust in the dashboard I presume?

[Luke 39615]

@softworkz will take a look at this. Thanks.

[Netfool 20]

"(Dear me!  I'm not certain quite
That even now I've got it right)".

-- Eletelephony
   Laura Elizabeth Richards
   https://poets.org/poem/eletelephony

 

Additional Information:  It may not be a bug related to cancelled conversions.  Apparently cancelling a conversion only stops the current conversion process and puts the requested conversion back on top of the conversions queue.  To actually cancel the conversion it appears to be necessary to both cancel the conversion on the dashboard and then go to the "Downloads" page and delete the "download". 

I noticed this the morning after I filed the report since the overnight here were a number of "Downloads" that failed.  Often of the the same file, over and over again.  Deleting all of the "downloads" in the queue seemed to stop the process.

The fact that the terms "Transcodes", "Conversions", and "Downloads" are all used interchangeably in different parts of the UI is both very confusing and not at all well documented. 

Cleaning up that language, and having an option to simply delete "downloads" that failed from the queue, as well as having a separate "Downloads, Conversions, and Transcodes" log that indicates a date-stamp, file name and success or failure would greatly simplify understanding what's happening.

Edited October 5, 2020 by Netfool

[softworkz 4488]

To be honest, HEVC hardware encoders on Android haven't got much attention yet.
I believe it fails because the defaults for profile and level are for AVC rather than HEVC.

You could make the following test:

• Install the Emby Diagnostics Plugin
  https://mediabrowser.github.io/Emby.DiagnosticsPlugin/
• In the diagnostic options section you will find two textboxes for doing find/replace within ffmpeg commands
  Enter the following:
  • Find: -c:v:0 hevc_mediacodecndk 
  • Replace: -c:v:0 hevc_mediacodecndk -profile:v:0 1 -level:v:0 4096 
    (corresponds to HEVC Profile 'Main' and Level '4.1')
• Then retry the conversion

Note: Those settings do not get persisted across restarts

 

[Netfool 20]

22 hours ago, softworkz said:

You could make the following test:

• Install the Emby Diagnostics Plugin

Thanks, I'll give that a try and report back here.

 

But... to the U/I issues, why is it necessary to manually remove failed conversions from "downloads"?

Edited October 6, 2020 by Netfool

[Luke 39615]

20 hours ago, Netfool said:

Thanks, I'll give that a try and report back here.

 

But... to the U/I issues, why is it necessary to manually remove failed conversions from "downloads"?

We are looking into improving it. Thanks.

[Luke 39615]

Hi, were you able to try this?

[Netfool 20]

2 hours ago, Luke said:

Hi, were you able to try this?

I got it set up to try but then I got distracted by a server crash I reported in a different thread.

Thanks for the reminder!  I'll give it a try.

I think I'll change the name of this thread too.  It's a bit misleading.  When a transcode fails, it get rescheduled (apparently ad-infinitum) so it's not that subsequent conversions fail, it that they never even get a chance to run without manual intervention. 

Edited October 28, 2020 by Netfool

[Netfool 20]

@Luke  OH... it looks like it won't let me do that.  How about "HVEC hardware transcodes fail on Shield Pro" ?

[Luke 39615]

Did you try what softworkz suggested?

[Netfool 20]

On 10/28/2020 at 9:45 PM, Luke said:

Did you try what softworkz suggested?

Yes.   No Joy. 

I made the profile substitution, and rotated the logs and then tried a conversion of a large (26GB) file to HEVC. 

The process ran for about 30 seconds showing reasonable progress in the dashboard, then disappeared.  The system then apparently did a log rotation itself, leaving the file showing in the Conversions page as "Transferring 0.0%":

I don't see anything significant in the log files.   Would it help if I re-ran the test with debug logging turned on?

If so... does that auto log rotation indicate that I should re-apply the profile change?

embyserver-63739667462.txt ffmpeg-transcode-c6a7d014-82e1-4240-bc92-7e6e75f03f7c_1.txt

Edited October 30, 2020 by Netfool

[softworkz 4488]

22 minutes ago, Netfool said:

Yes.   No Joy. 

One moment please - first of all, we need to understand that the original problem is fixed by the profile substitution.

HW accelerated encoding to HEVC is working!

Now we have a new and totally different problem.
(I don't have any ideas for that yet)

Edited October 30, 2020 by softworkz

[Netfool 20]

50 minutes ago, softworkz said:

One moment please - first of all, we need to understand that the original problem is fixed by the profile substitution.

HW accelerated encoding to HEVC is working!

Now we have a new and totally different problem.
(I don't have any ideas for that yet)

It does appear to be using hardware HEVC encoder.       ....until it dies 30 seconds later.

[softworkz 4488]

I understand that the outcome is not much different for you - from 'not working' to 'not working'..

But as we got a totally new situation now, maybe you could do some experiments - e.g. with some more "lightweight" files and see how that goes..?

Actually, you're probably the very first person now, that got HEVC hw encoding working on the Shield (with Emby).

Edited October 30, 2020 by softworkz

[Netfool 20]

@softworkz I'll give it a try!    Does that auto log rotation indicate that I should re-apply the profile change?

[softworkz 4488]

1 minute ago, Netfool said:

@softworkz I'll give it a try!    Does that auto log rotation indicate that I should re-apply the profile change?

What do you mean - the regular server log? IIRC it rotates when too big and at midnight, but I'm not sure. On startup as well.

The profile change needs to be re-applied only on restart.

[softworkz 4488]

Ah...probably the whole server is crashing...

[Netfool 20]

Well, my first try with a smaller file failed.   But maybe I needed to do a system restart before I re-patched the profile.

I just called to dinner, so I'll be afk for a while, but I'll do some more testing late tonight or tomorrow and report back.

ffmpeg-transcode-9b331a8b-c384-4852-b74b-7ffa5055ad3f_1.txt

[Netfool 20]

@softworkz

Restarted the Shield Pro and tried again.  Zero successes for two tries.

After re-entering the profile change I tried converting "The Ref".

That was a hard fail, showing up in the Conversions page a "failed".  It generated two ffmpeg-transcode logs.

I then tried converting "The Big Sleep".  That ran for about 6 minutes before stopping with nothing notable in the transcode log.  Shows in the Conversions page as "Transferring 0.0%".

When I tried converting "The Ref" the conversions parameters asked for HEVC and 4 MB/sec.   For "The Big Sleep" it was HEVC and original bitrate.

Screenshot attached to help with deciphering which log is which.  (Why the cryptic ids instead of just timestamps?)

...anything else I should try?

embyserver-63739698721.txt embyserver.txt ffmpeg-transcode-5e3b8d1e-ad86-4bf8-b721-82f88e5a277c_1.txt ffmpeg-transcode-ca344041-e550-4209-a549-fe30f664f143_1.txt ffmpeg-transcode-cac00a8d-25cd-475a-a041-902d3f248449_1.txt

[Netfool 20]

@softworkz ping

[softworkz 4488]

On 10/31/2020 at 8:12 AM, Netfool said:

...anything else I should try?

I guess I need to go and investigate this more. It will take a while until I get to it.

The only thing you could try meanwhile would be to do the same conversion but with H.264/AVC as target coded. 
This would allow us to rule out other (non-HEVC specific) causes.

Thanks

[Netfool 20]

22 hours ago, softworkz said:

I guess I need to go and investigate this more. It will take a while until I get to it.

The only thing you could try meanwhile would be to do the same conversion but with H.264/AVC as target coded. 
This would allow us to rule out other (non-HEVC specific) causes.

Thanks

Well...  in attempting to do that, I think I may have found a semi-related issue:

If you attempt a conversion that fails, the system retries that conversion ad-infinitum, until it is manually deleted from the Conversions page.

If you manually delete the failed conversion and then attempt to convert a different video, the second conversion immediately fails without ever being attempted.  ( i.e. No ffmpeg-transcode log is created).

It's apparently going to take a server restart before another conversion attempt will even be tried.  I'll restart the next time the server is idle and give it another try.

Edited November 7, 2020 by Netfool

[softworkz 4488]

On 11/7/2020 at 10:42 PM, Netfool said:

It's apparently going to take a server restart before another conversion attempt will even be tried. 

Let's see what @Luke thinks.