HW transcode on Rpi4 OpenMax and V4L2

[aptalca 70]

Hi there,

 

Just wanted to report some findings. I have openmax hw encode working on a raspberry pi 4 with emby stable (in a docker container, and on libreelec). That part works great.

 

I also tested v4l2. Although emby has access to the necessary devices, because of a missing patch in ffmpeg, v4l2 encode results in an all green image (can still see video, but in shades of green).

 

The helpful folks over at libreelec (especially 6by9 from raspberry pi) pointed me towards an ffmpeg commit (merged to master a long time ago but not pulled into the 4.2 release) that would fix that.

 

Here's the commit if you're interested in applying this patch to your ffmpeg build on arm32/64: https://github.com/FFmpeg/FFmpeg/commit/d61cf1b1ebc2477749d7d7825a072400ed24af9f#diff-bec5e20dbf17e1c989823fa841c2f8b3

 

Hw decode does not work for either, as expected (missing licence I believe).

[Luke 37064]

Hi, given the age of that commit I would expect that we would already have it, but @@softworkz will know for sure.

[aptalca 70]

If you're building the latest stable 4.2, it's not in there: https://github.com/FFmpeg/FFmpeg/blob/release/4.2/libavcodec/v4l2_buffers.c#L321-L331

[softworkz 3335]

@@aptalca - The commit is already included in our current beta builds and will also be in the next release of Emby

[softworkz 3335]

PS: Nonetheless, thanks for letting us know!

[aptalca 70]

@@softworkz

 

I tried v4l2 encode on emby beta and it falls back to software. Here's the error I get: 

17:48:35.953 [h264_v4l2m2m @ 0x1d6eb30] [Eval @ 0xbefd5b38] Undefined constant or missing '(' in 'high'
17:48:35.953 [h264_v4l2m2m @ 0x1d6eb30] Unable to parse option value "high"
17:48:35.953 [h264_v4l2m2m @ 0x1d6eb30] Error setting option profile to value high.

Full log: https://pastebin.com/H5essUfH

 

EDIT:

 

Unfortunately OpenMax also does not work, Here's the log: https://pastebin.com/rFngLuyW

 

In the same environment, the stable works with both omx and v4l2 (except for the green screen issue on v4l2)

Edited February 28, 2020 by aptalca

[softworkz 3335]

@@aptalca - Would you mind showing the logs where it is working with the stable build?

 

Or alternatively install the Diagnostics plugin (https://mediabrowser.github.io/Emby.DiagnosticsPlugin/ ) and activate "legacy command generation".

[aptalca 70]

Here's omx on stable: https://pastebin.com/jSfZ6AGv

Here's v4l2 on stable: https://pastebin.com/pa8G72tB (works but everything's shades of green)

 

I'll look into the plugin over the weekend

[aptalca 70]

@@softworkz I installed the plugin and turned on the option "legacy command generation"

 

Openmax works: https://pastebin.com/W0RWXRKz

V4l2 doesn't: https://pastebin.com/f4RU61RY

[aptalca 70]

@@softworkz just a heads up, the logs above will expire tomorrow

[softworkz 3335]

omx on stable.txt

v4l2 on stable.txt

legacy Openmax.txt

legacy v4l2.txt

[softworkz 3335]

@@aptalca - The next beta will include fixes for both cases.

 

Thanks a lot for reporting!

[aptalca 70]

No problem, thanks for looking into it.

[aptalca 70]

@@softworkz I just tried beta 4.4.0.25 with both v4l2 and omx

 

V4L2 plays, but the picture has weird lines all over and the color is off: https://pastebin.com/ExntiWre

 

emby_beta_v4l2.txt

 

Here's the v4l2 encoded image: 

 

And here's the direct play image:

 

OMX on the other hand no longer works: https://pastebin.com/01YkMJjQ

 

emby_beta_omx.txt

Edited March 16, 2020 by softworkz
Added attachments

[softworkz 3335]

Thanks a lot for reporting. 

 

For OMX we'll have an additional adjustment in the next beta.

 

For the V4L2 encoder, I'm not sure what the issue could be, but I read somewhere that it would be required to update the RPI to the latest kernel version.

[Luke 37064]

@@aptalca have you had a chance to try again?

[aptalca 70]

@@Luke @@softworkz

 

I just tested 4.4.0.30 and OpenMax works great 

 

For vl42, I'll try with raspbian first and then 3rd party ubuntu (much newer kernel)

[Luke 37064]

Thanks for the feedback !

[neik 837]

I just tested 4.4.0.30 and OpenMax works great 

 

I am wondering what you are transcoding (h264 / h265) and how many concurrent streams it is capable to serve before it starts struggeling?

[aptalca 70]

Decode is sw, this is only hw encoding whatever emby serves (h264, various bitrates). I was testing an encode of 1.5Mbps (reduced bitrate to force transcode)

[softworkz 3335]

SW decoding is expected, we don't support HW decoding on the RPI.

[neik 837]

SW decoding is expected, we don't support HW decoding on the RPI.

 

May I ask why?

According to what I found it should be able to decode h264 up to 1080p60 and h265 up to 2160p60, so should be sufficient for the vast majority I guess.

[Luke 37064]

The decoding api is meant to work with a video surface, which the server doesn't have because the rendering is happening on another device.

[aptalca 70]

Thanks a lot for reporting. 

 

For OMX we'll have an additional adjustment in the next beta.

 

For the V4L2 encoder, I'm not sure what the issue could be, but I read somewhere that it would be required to update the RPI to the latest kernel version.

 

@@softworkz

 

Finally got a chance to try v4l2 with a more recent kernel. Had some challenges as the ubuntu images for the rpi, which come with 5.3 kernels, did not have v4l2 enabled (no video devices) and couldn't figure out how to enable them. But I was able to get libreelec working with the 5.4.29 kernel.

 

I get the same result with v4l2 as I did before, the diagonal jagged lines overlayed.

 

Here's the log: https://pastebin.com/CGBSRuSR

 

On another note, I just wanted to give you a heads up that I have mmal decoding working with omx encode on the rpi4 in a docker container. I just had to expose `/dev/vc-mem` into the container and ffmpeg with mmal can hw decode.

[softworkz 3335]

@@aptalca - Thanks for reporting!

 

So it's good to know that the "diagonal jagged lines" is not a problem specific to Emby :-)

 

 

On another note, I just wanted to give you a heads up that I have mmal decoding working with omx encode on the rpi4 in a docker container. I just had to expose `/dev/vc-mem` into the container and ffmpeg with mmal can hw decode.

 

According to the docs, ffmpeg cannot use MMAL when run from the command line. Could you please post an ffmpeg log for this?