Jump to content


Photo

Hardware acceleration - Emby inside LXC(Debian/Ubuntu) container using Intel iGPU

hardware acceleration linux hvec transcoding renderD128 VAAPI Proxmox

  • Please log in to reply
15 replies to this topic

#1 appoli OFFLINE  

appoli

    Advanced Member

  • Members
  • 33 posts
  • Local time: 10:57 PM

Posted 16 April 2019 - 01:28 PM

Hi All,

 

I have mostly made posts moaning about how stuff doesn't work right/the way I want to, but I'm going to give back to the community today!

 

For the TL;DR skip down a few paragraphs, just me venting my woes

I go into how I got hardware acceleration to work when it wasn't and everything said it should have been on an Ubuntu LXC container

 

I originally built a FreeNAS machine, on which I planned to store loads of stuff including media, and saw that it had media player plugins. I didn't think too much of it at the time (except I didn't want to use Plex because my experience with it showed that it wasted resources/was the 'dummy' version) so I specced out my server and I built it.

I chose a Sky Lake/Kaby Lake cpu for a number of reasons, but one of them was the iGPU. I tried using an Emby plugin but that didn't work (now know it's a whole .NET issue with FreeBSD) and then tried a Docker container of Emby - worked fine, but no hardware acceleration. I wasn't sure why, but I knew I needed more power in the server for the transcoding and other stuff I was using it for so I got a Kaby Lake Xeon cpu, making sure to get one with an iGPU, and kept on plugging away. To cut to the chase:

 

- found out I needed to pass /dev/dri to docker

- found out I needed VAAPI to get use of the gpu, but FreeNAS did not support it so it would need to be in a VM and FreeNAS was going through lots of changes

- looked around for a few other operating systems that could be used for my purposes, tried OMV - it didn't like ZFS

- landed on Proxmox, perfect for my needs - can spin up debian/ubuntu LXC containers easy peasy while passing through whatever I want from the root OS & can make VMs for other OS'/things I want more secure

- found out that my motherboard had the C232 chipset & I needed the C236 chipset to use the iGPU

- Finally bought the right motherboard

 

Honestly, you would think I had done absolutely no research! But a lot of this was new to me and I didn't realize what I would be using the machine for (didn't know how much use I could get out of Emby per se - I already had an HDHomeRun and Apple TVs...).

 

So I swapped in the right motherboard (plus I got some more SATA ports - gonna be cloning my zpool later to a much larger one w/ more redundancy since I'm using the machine for work too now) and went about making sure that dev/dri and fb0 were passed through to the Emby LXC container.

 

*****Skip to Here*****

 

At this point I double checked that everything was being passed through to the container (eg lspci) & went through the Emby documentation (they state that it Emby should have all the drivers that it needs built in, e.g. their own FFmpeg build).

However, when I would play a file that was a direct feed it played fine, but when I tried playing a 4K HVEC or 9/10Bit/VP9/VP10 whatever they actually decided to call that,  the video would just load and never start.

 

I went into the console and VAAPI was indeed installed and showed that it was able to decode/encode the appropriate files for my cpu. Checked the log - it looked like FFmpeg was doing it's thing and transcoding the file writing stuff to the temp folder and including a transcoding rate (e.g. at one point it said it was transcoding at 66.6 x frames ).

 

I was about to post on the forum, but I really really really have been wanting to get this working. So I looked around and I found the following site, or rather series of files from the VAAPI sites:

 

https://github.com/intel/media-driver

 

The genesis before that link was basically that VA API needed some extra libraries/intel media SDK to operate depending on the OS/CPU. So that link is for an addition driver that has links to two other libraries that are needed first (libva & GmmLib) along with their dependencies/reqs to build them. 

 

Follow the links and cit clone those libraries over to a build directory, make them and install them. For less experienced people the GmmLib instructions are less clear: after git clone GmmLib, make a build directory for cmake & change into it you issue the cmake command with '-DCMAKE_BUILD_TYPE=Release', the site just shows you the possible options. I left out the -DARCH=64 bit because from what I saw on the internet others didn't use it, but you DO need to reference a cmake build file for the command. That is in the root folder of the build folder you made. so either add '..' at the end or '/wherever you git cloned to/gmmlib' to the end of the cmake command and it will run.

Then you do the make -j8 command followed by the make install command.

 

Once those two guys are added I git cloned the media-driver bit in, followed the instructions, restarted the container and honestly didn't expect anything to have changed.

 

But hardware transcoding started working like a charm! 

I do NOT know if it was a combination of those libraries, if it was a dependency of libraries (or maybe just me having to reset the BMC a bunch of times b/c my fan control script was acting up, but I highly doubt that's what it was) but after the above everything worked.

 

As a final note, I don't think this is a shortcoming of the Emby team. As far as I can tell people are using hardware transcoding via the VAAPI files Emby installs just fine. Maybe it had to do with the specifics of my case - Emby running in an LXC container, the LXC container running inside of Proxmox, the fact that because I have IPMI the BMC has it's own video device that are seen in the OS'.

 

I'm just happy I got it to work (maybe can help the Emby team do some investigating) & hope this can help others save some time. Cheers!

 

 

 


  • NolesFan78 likes this

#2 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 138261 posts
  • Local time: 10:57 PM

Posted 16 April 2019 - 04:19 PM

Hi this is great, thanks for the info ! So you did this with Docker on FreeNAS?



#3 appoli OFFLINE  

appoli

    Advanced Member

  • Members
  • 33 posts
  • Local time: 10:57 PM

Posted 18 April 2019 - 11:12 AM

Hi Luke,

 

No problem!

And no - I did this with Emby running on an Ubuntu 18.04 LXC Container hosted on a server running Proxmox (which itself runs on Debian 9 I believe, & if it matters somehow zfs is involved except for the small drive Proxmox is on).

I had the physical devices (as far as I can tell) passed through the container/apparmor/etc just fine as vainfo was yielding the right info, but that wasn't enough.

It could be my version of the iGPU (Kaby Lake Xeon E3 1245 v6, so it has an Intel HD Graphics P630 'chip' on it). VA API, et al seem to be highly sensitive to that which kind of makes sense?

 

I will say that I had a user who had an issue playing a 4k HEVC movie on his Xbox & Apple TV, but worked on his iPhone app - the logs say it was due to the bitrate limit being exceeded, so that's all it could be... I do remember a couple years back that the Emby app on Xbox, or the Xbox itself, didn't like one of the codecs in 4K files so they wouldn't play on them.... I told him to adjust the settings, but haven't heard back.

 

 

 

On a side note - I have my Emby server behind a reverse proxy (HAProxy on pfSense) so that the url is both https & simple (https://emby.XXXX.ddns.net). It works for EVERYTHING except for Apple TVs that are OUTSIDE my home network (oddly enough the Apple TVs in my home use the external address just for the sake of continuity). It's the strangest thing.... I end up having to leave a port open for those guys for port forwarding that I would really like to close.

 

I just bought a domain from Amazon and plan on using Route 53 as my dynamic DNS (since it comes out to be cheaper than the cheapest no-ip option, I get my own domain, and a whole bunch of other things).

Since I have my own domain now I can run emby as a domain path (xxxx.net/emby) rather than a sub domain (emby.xxx.net), which seems to be the preferred method. 

Per chance have you heard about any issues like this/do you think the above would fix the issue? clearly the Apple TV doesn't has some sort of preference with https url structure or just an issue with url structure/using subdomains....

 

 

 

Anyway, just saw you guys pushed a fairly big update out. Really appreciate the work you guys are doing! Hope nothing breaks!


Edited by appoli, 18 April 2019 - 11:15 AM.


#4 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 138261 posts
  • Local time: 10:57 PM

Posted 18 April 2019 - 11:22 AM

Thanks for the feedback. Regarding ssl you'll need to make sure to use a certificate that Apple devices will trust.

#5 appoli OFFLINE  

appoli

    Advanced Member

  • Members
  • 33 posts
  • Local time: 10:57 PM

Posted 18 April 2019 - 03:35 PM

Yep - the certificate is done via let's encrypt like most people (using the Acme plugin on pfSense). Mind you, iPhones/iPads/MacOS' (which I guess means Safari, but I also have a number of other things hosted on the server under different sub domains covered by the same cert).

AND the Apple TVs in the house are told to use the external address. Granted, if they use the router as the DNS and/or the router NICs identify loop back traffic the traffic might not make it out, but I'm not sure how that works with a reverse proxy since it needs the actual url to determine which IP to route traffic to...

 

I'm hoping the change in the DDNS & shifting over to using separate domain paths rather than sub-domains which I think is what is recommended? at least 99% of examples on the internet seem to use paths rather than sub-domains...

As I'm assuming it's the Apple TV being causing the headache, I'm not 100% positive it's not the Emby App for the Apple TV. 

 

Since this is veering off course I'd be happy to post under a different thread, but I was curious if emby had a preference when handling this sort of thing. I have the 'secure connection mode' set to handled by reverse proxy, gave it the public url (https....) & public ports (80 & 443 w/ 80 supposed to be redirected to 443, but who knows how well that works. I have all the data cached so on my browsers so it's hard for me to test these things out.

 

 

On Topic!

the update did not seem to effect 4k viewing, maybe uses fewer resources (hard to tell, emby had 2 other processes going on at the same time) at leas this was on my laptop using Safari.

On my Apple TVI tried watching a 4k movie that had a 'TrueHD 7.1' audio file and a 'dolby digital 5.1' audio file. when viewing the file with the 5.1 file (that's all I have) everything seemed to be working fine, but the dashboard also said it was a direct streaming. So I let it play with the default 7.1 audio, the stream was still labeled direct streaming, but there was some transcoding going on.

 

Originally the video stalled a min and a half in on the first play, but I thought that was the Apple TV's fault (I hadn't restarted it since the update a few weeks ago and things were slow/weird). So I restarted and gave it another whirl. this time I got 4 min in to the movie until it locked up again, but unlike before if I was patient the Apple TV would start to respond again slowly (meanwhile transcoding on the server just kept chugging along building up more and more of a buffer lead).

Then the video cut out again 30 seconds later. I had to be even more patient, but eventually the Apple TV responded and hitting play resumed the movie. As per before, emby server kept chugging along.

 

So, maybe emby server isn't down mixing the audio enough for emby on the Apple TV or the device doesn't have the resources for the way the app is programmed or what have you.

 

I do know that you guys did make the recent change in the emby Apple TV app making it so it isn't reliant on the codecs that Apple TV supports, thus requiring less transcoding on the server side (like people had kept saying Plex did which I think just used vlc? I dunno).

But it does seem like choosing the 7.1 trueHD sound (I'm 90% sure Apple TV doesn't support that) over 5.1 yields the server to transcode the audio. But maybe it needs to be 'down mixed' even further given the hardware limitations of the Apple TV?

 

Or maybe the way I got the transcoding to work on my server is resulting in output that sucks?

 

I'll keep watching the 7.1 version and see what happens (giving it time to buffer, see if that makes any difference - maybe it can go on the ATV swap file or something). If it keeps doing the same thing I'll update my post.

if not then I won't

 

If it does keep doing the same thing then I will try watching the movie using the 5.1 audio and will let you know the outcome of that



#6 appoli OFFLINE  

appoli

    Advanced Member

  • Members
  • 33 posts
  • Local time: 10:57 PM

Posted 18 April 2019 - 04:08 PM

Yep, keeping the video at 4k with TrueHD7.1 on the Apple TV cause playback to constantly stop as if it's buffering. Checking the transcoding process on the server had it bouncing around between a lot ahead/almost done transcoding the whole video, to being just 10 min ahead, but there was always buffer of transcoded audio whenever the stuttering would happen. 

 

Switching to 5.1 audio made the video be full direct playback and that worked totally fine. 

 

I'll be curious how the 3rd party emby app on my smart tv handles 4k files... I'll give that a try later



#7 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 138261 posts
  • Local time: 10:57 PM

Posted 19 April 2019 - 11:26 AM

Yes truehd will still transcode on the server. Thanks.

#8 appoli OFFLINE  

appoli

    Advanced Member

  • Members
  • 33 posts
  • Local time: 10:57 PM

Posted 20 April 2019 - 11:52 AM

I understand TrueHD will start transcoding on the server.

 

I'm saying that the transcoding it does is incompatible with Apple TVs



#9 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 138261 posts
  • Local time: 10:57 PM

Posted 20 April 2019 - 11:56 AM

I would suggest lowering the in-app quality setting. Please see if that helps. Thanks.



#10 appoli OFFLINE  

appoli

    Advanced Member

  • Members
  • 33 posts
  • Local time: 10:57 PM

Posted 27 April 2019 - 03:34 AM

Hi Luke,

If I can figure that out it sounds like it might help.


I’ve been having another issue since I added HW acceleration so I figured I would add the details here. Originally he had tried playing a 4K video on his Xbox, then ATV, but both quit the playback and the log said the max bitrate was exceeded in the log (it worked on the iOS app however). Regardless, i maxed out the bitrate on his Xbox app and figured that would be the end of it.

However, he has been watching a TV show, high quality 1080p files, and the result has been not just the emby server app crashing, but the LXC container crashing & Proxmox crashing. When I find out about it I have to manually reboot the machine (can’t even SSH in to it). The logs originally said that while he was watching on his Xbox his parents signed in on their ATV, using the same account behind the same IP, and I figured that the max bit rate error seen in the log was due to the ATV app trying to play the same thing he was watching on his Xbox (I never tried using the same account on different devices behind the same IP...). Looking at the logs it still had an error for the Appleton saying the max bitrate was too high and I figured that caused the issue.

However, he was watching the show again today and after a number of episodes the crash happened again bringing everything down. Looking through the log I found the ‘max bitrate exceeded’ error again which is weird (since I maxed it out on his client), but even with that originally the client would just stop plying the video. The server completely crashing is new.

I couldn’t figure anything else about it from the logs (maybe some app trying to run in parallel?), so I’m including the logs and asking for some help.
Note: he watches with subtitles so he doesn’t need to pay attention and without transcoding throttling the video would totally stop playing within 30 seconds after the video was paused, for any reason. From the experimenting I did before I noticed that when a video is paused and transcoding isn’t throttled, much of the buffer would be lost so I turned throttling on and that seemed to help.

Either way it doesn’t really explain why my server is crashing. Any help would be appreciated, thanks! This happened on the 26th between 5:25 and 5:30pm

Attaching the general and transcoding logs

Attached Files



#11 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 138261 posts
  • Local time: 10:57 PM

Posted 27 April 2019 - 01:28 PM

Sounds like perhaps there was a GPU crash and this caused the container to crash?

#12 appoli OFFLINE  

appoli

    Advanced Member

  • Members
  • 33 posts
  • Local time: 10:57 PM

Posted 28 April 2019 - 07:15 PM

Hi Luke,

Thanks for the super prompt response as always.

I’m open to the idea & haven’t discounted it (although wouldn’t know exactly what I would need to do to figure that out? Any specific logs would help or forum I should contact ex. VAAPI?).

What I do find interesting is that it is always with the same user & the same device (emby app for Xbox - which the emby Dashboard ‘ironically’ says that those episodes are ‘direct play’/no transcoding streams). And the bitrate limit exceeded portion is also strange, would the log say that if there was a server side issue or only if there is a client side issue? (And I had bumped up the bitrate limit to 4K - 120mbps a few days ago on his Xbox)

When this specific user first experienced an issue that yielded those errors (the Apple TV and Xbox clients would crash, not the server, however the iOS client would play the video) there were no server issues. It was the 4K movie I mentioned before & I know Xbox/the emby Xbox app uses to have an issue with 4K files with DTS audio a couple years back.

I was wondering that maybe it had something to do with the guy constantly using different devices at the same time behind his router, thus few different clients being NATed behind the same IP? (Aside from his parents using the Apple TV, he often has the iOS app open while playing something on the Xbox app).
Yesterday the pause/restart issue (the video either crashing or lagging long enough for him to think it crashed if he would pause it for any period of time with the subtitles on) cause him to get frustrated and just stop using it.

Looking through the log I do notice what seem to be a lot of errors with the subtitle apps, no? I had disabled the addic7ed plugin because it seemed as though that caused nothing but errors.

Either way, any help would be appreciated

Edited by appoli, 28 April 2019 - 07:17 PM.


#13 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 138261 posts
  • Local time: 10:57 PM

Posted 28 April 2019 - 09:35 PM

Can we look at an example? The above ffmpeg log I don't believe is from the xbox, right?



#14 appoli OFFLINE  

appoli

    Advanced Member

  • Members
  • 33 posts
  • Local time: 10:57 PM

Posted 29 September 2019 - 02:03 PM

My friend that uses the Xbox hasn’t logged in quite a while so I don’t have any current logs from one.

I have fixed most of the issues I have had with emby:

For those running emby in a Proxmox container (and trying to get hardware acceleration to work correctly) & using ZFS I recommend turning swappiness = 0.

I had thought that might have been the issue (I had originally set it = 10 as per their instruction), but the container and the host machine had never actually ran out of memory, so I hadn’t actually tried it out.
But I randomly gave it a go when I couldn’t think of anything else to try (doing the recent upgrades got it to the point where only the emby application would crash, not the whole machine) & now everything seems to be running smoothly. No crashes so far. I’ve asked my friend to give it a try again, will post logs if anything happens - he’s a fickle one though and once he started experiencing problems he stopped wanting to use emby at all

#15 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 138261 posts
  • Local time: 10:57 PM

Posted 29 September 2019 - 02:38 PM

That's interesting, thanks for the feedback !



#16 appoli OFFLINE  

appoli

    Advanced Member

  • Members
  • 33 posts
  • Local time: 10:57 PM

Posted 29 September 2019 - 09:12 PM

No problem, apologies about not getting back with more info.

To provide some more information - before the recent update to Emby & the new version of Proxmox when clients that required a lot of transcoding (or has inconsistent connections requiring re-transcoding) and in general when there was high I/O load - it would often happen after my system would have done a backup of the containers - the machine would crash. I would be unable to access emby/proxmox/etc via the web GUI & would be unable to SSH into it. The BMC would also only be able to do a force power off.

After the update to the new version of emby & some of the various Linux updates (emby has been running in an Ubuntu 18.04 container using a few folders shared through the container which are ZFS datasets. With nesting & such enabled) things got better, but still during high I/O the emby app itself would crash. Everything else would work and I could log in to the container to restart things. I believe this included high bitrate videos that were direct played.

I tried a number of different things including trying different I/O schedulers, various ZFS property options, etc. My machine was using the iGPU from the Xeon E3-1245 v6, 64 GB ECC RAM, a ZRAID-2 pool with 5 SATA HGST He8 disks & 2 u.2 Intel DC P3700 disks for cache connected using OCuLink to a Supermicro controller in a bifurcated PCIe slot. The mobo is a Supermicro X11 SSM (I think that’s the model, only 2 or so motherboard in this range have VT-d, required for iGPU passthrough, and PCIe bifurcation).

Everything ran fine until I added the new motherboard and started using hardware acceleration. I got things to work by installing the VA-API software on the Proxmox host.
That worked alright, had some issues, but when adding the caches things got worse.

Finally I figured I would give the swappiness value a go, even though my machine never went above ~90% ram use on the host, and above 25% on the container, and all of a sudden things stopped breaking...
  • Luke likes this





Also tagged with one or more of these keywords: hardware acceleration, linux, hvec, transcoding, renderD128, VAAPI, Proxmox

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users