Jump to content

Best setup for transcoding on Linux


doonze

Recommended Posts

doonze

So, I'm running an older low end gateway duel core system as my server. I'm running Debian 9, and don't plan on switching to anything else. While this works for my day to day needs, it can only really transcode one stream at a time. And makes a lot of noise from fans to do so. What I'm trying to get my head around is what hardware Emby can take the best advantage of running under Linux for transcoding multiple streams. I would like to have a system that could transcode 3-4 streams and keep up.

 

My starting point would be a 8 Core processor with 32 GB ram transcoding on a SSD. 

 

What processor extensions do I need to look for that Emby can take advantage of? What's the best hardware acceleration to use? AMD better or Intel? Any motherboard hardware that can help? Can Emby take advantage of any GPU processing? If so, what cards are best?

 

Mainly I want to make a big bad Emby streaming monster. But not sure what Emby can best take advantage of.  Anyone who knows transcoding better then I do got any opinions? 

 

Thanks!

Link to comment
Share on other sites

mastrmind11

So, I'm running an older low end gateway duel core system as my server. I'm running Debian 9, and don't plan on switching to anything else. While this works for my day to day needs, it can only really transcode one stream at a time. And makes a lot of noise from fans to do so. What I'm trying to get my head around is what hardware Emby can take the best advantage of running under Linux for transcoding multiple streams. I would like to have a system that could transcode 3-4 streams and keep up.

 

My starting point would be a 8 Core processor with 32 GB ram transcoding on a SSD. 

 

What processor extensions do I need to look for that Emby can take advantage of? What's the best hardware acceleration to use? AMD better or Intel? Any motherboard hardware that can help? Can Emby take advantage of any GPU processing? If so, what cards are best?

 

Mainly I want to make a big bad Emby streaming monster. But not sure what Emby can best take advantage of.  Anyone who knows transcoding better then I do got any opinions? 

 

Thanks!

Any high end modern Intel CPU will do what you want, and VAAPI acceleration is well supported in Linux.  I have an old Xeon E3 1225 v3 that can easily do 5 transcodes w/o hardware acceleration.

Link to comment
Share on other sites

doonze

After researching it seems I had all the drivers already for VAAPI. I enabled it, and Emby still works. I can't tell if it's using it at all or not though. Transcoding stills pegs out my CPU at 100%.

Link to comment
Share on other sites

doonze

No vaapi in the ffmpeg transcoding logs. I've got VAAPI chosen in the transcoading menu, but here is the command line it's using in ffmpeg:

/var/lib/emby-server/ffmpeg/20170308/ffmpeg -ss 02:02:52.808 -i file:"/mnt/mybook/media/Movies/Blade Runner 2049 (2017)/Blade Runner 2049 (20
17).avi" -map 0:0 -map 0:1 -map -0:s -codec:v:0 libx264 -force_key_frames "expr:gte(t,n_forced*5)" -vf "scale=trunc(min(max(iw\,ih*dar)\,1280
)/2)*2:trunc(ow/dar/2)*2" -pix_fmt yuv420p -preset superfast -crf 21 -maxrate 2616000 -bufsize 5232000 -profile:v high -level 4.1 -x264opts:0
 subme=0:me_range=4:rc_lookahead=10:me=dia:no_chroma_me:8x8dct=0:partitions=none -vsync -1 -map_metadata -1 -map_chapters -1 -threads 0 -code
c:a:0 libmp3lame -ac 2 -ab 384000 -af "volume=2" -y "/mnt/mybook/emby/transcoding-temp/f55d77ed69c141914d83e783cf476fa6.mkv"

From what's if seen from a few other post here on the forum, it should be sending a flag to use vaapi in that string, but it's not.....??? confused....

Link to comment
Share on other sites

dcrdev

No vaapi in the ffmpeg transcoding logs. I've got VAAPI chosen in the transcoading menu, but here is the command line it's using in ffmpeg:

/var/lib/emby-server/ffmpeg/20170308/ffmpeg -ss 02:02:52.808 -i file:"/mnt/mybook/media/Movies/Blade Runner 2049 (2017)/Blade Runner 2049 (20
17).avi" -map 0:0 -map 0:1 -map -0:s -codec:v:0 libx264 -force_key_frames "expr:gte(t,n_forced*5)" -vf "scale=trunc(min(max(iw\,ih*dar)\,1280
)/2)*2:trunc(ow/dar/2)*2" -pix_fmt yuv420p -preset superfast -crf 21 -maxrate 2616000 -bufsize 5232000 -profile:v high -level 4.1 -x264opts:0
 subme=0:me_range=4:rc_lookahead=10:me=dia:no_chroma_me:8x8dct=0:partitions=none -vsync -1 -map_metadata -1 -map_chapters -1 -threads 0 -code
c:a:0 libmp3lame -ac 2 -ab 384000 -af "volume=2" -y "/mnt/mybook/emby/transcoding-temp/f55d77ed69c141914d83e783cf476fa6.mkv"

From what's if seen from a few other post here on the forum, it should be sending a flag to use vaapi in that string, but it's not.....??? confused....

 

This path "/var/lib/emby-server/ffmpeg/20170308/ffmpeg" would indicate that you are using the mono version of Emby from the repo and not the standalone .net core package.

 

You need to supply your own ffmpeg build that has been compiled with vaapi support, the build that the mono version downloads is a static build and hence doesn't support vaapi due to the drivers being located in different places on on different distros.

 

Alternatively you could switch to the .net core standalone package which is essentially self contained and contains everything you need for vaapi, right out of the box.

Link to comment
Share on other sites

doonze

This path "/var/lib/emby-server/ffmpeg/20170308/ffmpeg" would indicate that you are using the mono version of Emby from the repo and not the standalone .net core package.

 

You need to supply your own ffmpeg build that has been compiled with vaapi support, the build that the mono version downloads is a static build and hence doesn't support vaapi due to the drivers being located in different places on on different distros.

 

Alternatively you could switch to the .net core standalone package which is essentially self contained and contains everything you need for vaapi, right out of the box.

 

Well, heck..... I had wondered about that. It's seeming more and more like I need to just bite the bullet and switch back to the .net version. Maybe I'll play with that soon. 

 

I lost my settings last time, anyone know off the top of their head what I need to do not to lose all my settings again when switching?

Link to comment
Share on other sites

tdiguy

Well, heck..... I had wondered about that. It's seeming more and more like I need to just bite the bullet and switch back to the .net version. Maybe I'll play with that soon. 

 

I lost my settings last time, anyone know off the top of their head what I need to do not to lose all my settings again when switching?

I am not sure if this is everything because i have not used it myself but this link will likely be helpful: https://github.com/MediaBrowser/Wiki/wiki/Backup

Link to comment
Share on other sites

doonze

Alright, the plot thickens. I switched over to the installer version again. I wanted to try an figure out if vaapi was really working so took the command Emby issues and tried to run it from cli. I get errors when running it with the ffmpeg from the Emby path, It gives me some error about LIB*_55 not found or some such. But I could get it to run with my ffmpeg from bin. Anyway, after playing I was able to get the below to work:

sudo ffmpeg -vaapi_device /dev/dri/renderD128 -hwaccel vaapi -ss 00:10:52.000 -f matroska,webm -i file:"/mnt/m
ybook/media/Movies/Allegiant (2016)/Allegiant (2016).mkv" -map 0:0 -map 0:1 -map -0:s -codec:v:0 libx264 -force_key_frames "expr:gte(t,n_forc
ed*5)" -vf "scale=trunc(min(max(iw\,ih*dar)\,1280)/2)*2:trunc(ow /dar/2)*2" -pix_fmt yuv420p -preset superfast -crf 21 -maxrate 2552000 -bufs
ize 5104000 -profile:v high -level 4.1 -x264opts:0 subme=0:me_range=4:rc_lookahead=10:me=dia:no_chroma_me:8x8dct=0:partitions=none -vsync -1 
-map_metadata -1 -map_chapters -1 -threads 0 -codec:a:0 libmp3lame -ac 2 -ab 384000 -af "volume=2" -y "/mnt/mybook/emby/transcoding-temp/9331
08f75e3040fe72350a60f3098477.mkv"

I ran it both with and without "-vaapi_device /dev/dri/renderD128 -hwaccel vaapi". Difference was about 25 FPS. Sill had high CPU usage, about 10% lower than without vaapi, but it was doing something. 

 

However, running the exact same file through Emby produced no picture, and the log showed 2 FPS in the transcoding log.

 

Some other files transcoded, but didn't even try to use vaapi.   

 

So whatever, I'm done trying Hardware acceleration. Sad since it's headless and the GPU does noting else. Oh well.

Link to comment
Share on other sites

dcrdev

What hardware are you using? I can only envision the standalone version not working if your using hardware that doesn't support the i915 or r600 drm drivers - in which case ffmpeg likely doesn't have much vaapi extension coverage; i915 i.e. relatively modern Intel is best supported.

Link to comment
Share on other sites

doonze

What hardware are you using? I can only envision the standalone version not working if your using hardware that doesn't support the i915 or r600 drm drivers - in which case ffmpeg likely doesn't have much vaapi extension coverage; i915 i.e. relatively modern Intel is best supported.

 

Oh, I'm sure your right. I'm running an AMD Duel core system with integrated AMD/ATI  graphics. Radeon HD 7540D to be exact. The driver I'm using is radeonsi_drv_video.so. Here's my vainfo:

error: can't connect to X server!
libva info: VA-API version 0.39.4
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/gallium_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.39 (libva 1.7.3)
vainfo: Driver version: mesa gallium vaapi
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileNone                   :	VAEntrypointVideoProc

I get the xserver error because this is a headless server, so no xserver is running. 

5

Also note for whatever reason it wants  gallium_drv_video.so. That doesn't exist on Debian/Ubuntu. Seems currently only Arch builds come with it. I COULD jump through hoops to get it, and build the Arch driver package, but  that's not even the right driver for my card anyway, from all I could gather. So I just syslinked Gallium to radeonsi and vainfo was happy. 

 

And it DOES seem to work, with the repo build of ffmpeg as I showed above. I got 25 FPS increase in transcoding and a small decrease in CPU usage. That tells me it's doing something. However the exact same file is only 2 FPS using Emby. 

 

I can tell from the .configuration outputs of each that the Emby version has much more enabled in it's ffmpeg build. I'm wondering if that's causing the Emby version to try and do more than my repo ffmpeg build, and that one of those options is causing vaapi to choke. 

 

Unfortunately there is very little trouble shooting for vaapi out there. It's one of those things where once it's all installed it's just supposed to work. I'm sure it's something not installed right somewhere, but since I'm getting no errors I have nothing to try and work on to fix it. 

Link to comment
Share on other sites

dcrdev

Yep that'll be why then - driver isn't included in the .net core package.
 
You're also not seeing much benefit from your self supplied ffmpeg because gallium isn't very well supported:

AMD platforms use the gallium driver which is developed as part of the Mesa graphics library: http://www.mesa3d.org/ . It is less complete than the Intel driver, supporting fewer use cases - decode support has existed for a while and is relatively stable, but encode support was only added recently and is still in flux. Only some distributions package it - recent Debian-based distributions include it in mesa-va-drivers.
 
Source: https://wiki.libav.org/Hardware/vaapi

 

 

Newer GCN 1.3+ AMD Cards are better supported, but you really want Intel.

Link to comment
Share on other sites

doonze

Here's some more weird stuff. Just for giggles I started an xserver. Once I have an x running I get more correct vainfo. It uses r600 driver then. So that seems to be what linux thinks is my correct driver. 

 

I'm surprised intel is the leader for linux seeing as how AMD has always been pretty active with linux support. But I'll take your advise when I go to build my new server. We're talking mostly intel GPU's right? Do intel CPU's also have an advantage for transcoding on linux? Aside from my laptop, all my linux boxes are headless so graphics has never been an issue. 

 

Oh well, this goes on the back burner for now. I don't think I've going to figure it out. 

Link to comment
Share on other sites

dcrdev

Here's some more weird stuff. Just for giggles I started an xserver. Once I have an x running I get more correct vainfo. It uses r600 driver then. So that seems to be what linux thinks is my correct driver. 

 

I'm surprised intel is the leader for linux seeing as how AMD has always been pretty active with linux support. But I'll take your advise when I go to build my new server. We're talking mostly intel GPU's right? Do intel CPU's also have an advantage for transcoding on linux? Aside from my laptop, all my linux boxes are headless so graphics has never been an issue. 

 

Oh well, this goes on the back burner for now. I don't think I've going to figure it out. 

 

Yeah AMD vaapi support is very recent, for many years AMD utilised vdpau - which was decoding only. Are you sure it uses the r600 driver? That's amdgpu and your chip doesn't *officially support it.

 

Intel is the most supported platform for hardware encoding, next up is probably nvidia with nvenc. Intel vs AMD there's no distinct advantage in either.

Link to comment
Share on other sites

doonze

Duly noted. Due to your input my next streaming box will lean intel. From all I've read I SHOULD still be able to do HA. But something is hinked up on my setup. Windows had no issues on this box, but I dropped Kodi and picked up Emby because I could never get Kodi to do a decent job under linux. It always seems there was something up with my video drivers. A hack (turned out to be the teamviewer hack years ago) drove me to move my media box to linux. But now I wouldn't have it any other way. 

 

I'd do a fresh install of linux, but I've spent YEARS getting this box setup the way it is. I run a full webserver and have about 15 services that took me years to work all the bugs out of. I'm afraid I'd never be able to get it setup the same again. I've spent 100's of hours on this box. In fact I'd likely just run both boxes when I get a new one. Just move Emby and Plex to the streaming box and leave this one running everything else. I have everything from an NZB indexer to cloud storage running. It's a busy box. 

 

Anyway, thanks for trying to help!!!! I really appricate it!!

Link to comment
Share on other sites

tdiguy

I went with linux on my media box primarily because i didnt want to buy a copy of M$. I know there are a million and one ways to get a lic for free or very cheap but my media server does very little and its pretty nice having a dedicated linux box and i am really loving the ssh on it, seems really solid compared to the ssh on my router.

Link to comment
Share on other sites

  • 2 weeks later...
snorkel

I am using a older AMD FX 8350 8 core @ 4gzh with 16gb ram because I had it laying around and it seems to work extremely well with Ubuntu Server.

I just have a older 512MB nVIDIA Quadro NVS 300 PCIe card because the server is headless i.e. no desktop. 

it seems to transcode pretty effortlessly. 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...