Jump to content

Hardware for Emby (and other servers) ESXi Vs Docker - Performance


ScottRTL

Recommended Posts

ScottRTL

Hi Everyone, sorry for the long-winded post:

I have approximately, 5 remote users that may be transcoding at any given time, plus 3 possible direct streams in my house, then mobile phone/tablet usage, maybe a couple at a time max (but rare, at least for now)

I know there are a lot of posts where people are asking what kind of server to use, and if their server is good enough, my question(s) are different. First I will give an idea of my previous servers so you can see where I have been, so we can understand better, where I would like to go:

1.) I started with a Drobo FS (for Media) with single drive redundancy, and a Dell Poweredge R610 running a Plex server (this worked well for a few years), but I did need to rebuild Plex now and again, and setup all my coverart/other metadata a few times. (not fun).
2.) I moved to a Drobo 5N (with a cache drive) I initially tried the Plex server built into the Drobo 5N, but the server was too slow, so I went back to the Dell and used the Drobo just for the Media
3.) I lost all the Drobo files because a second drive failed during a rebuild. rip. So I moved to a Synology DS1815+ (maxed the RAM on it, 2 drive redundancy) I upgraded my Plex server to an HP DL360 G6 with 128 GB RAM and 2x Xeon 5650 (got it for cheap through my work) Eventually I lost all my metadata (not media) on Plex because a SAS drive failed.
4.) I created an iSCSI LUN on my Synology and put Plex on that (so I don't have to worry about losing the VMs/MetaData), tried that for a while, but it was too slow. (At this point I decided to switch from Plex to Emby for features as well)
5.) So FINALLY (sorry for the long post again) I ended up with my DL360 running ESXi, with an Emby VM, and a seperate torrent VM (always behind VPN).

Basiclly, The DL360 isn't working anymore, it is currently in my garage, and during the summer, it sounds like a jet taking off, and the fans die often. The only place I can move it is the basement, and that doesn't fit there (it's GIANT), so I need to build a PC, and I figured I would futureproof a bit, and move away from enterprise to avoid the expensive replacement equipment. (Fans, and SAS drives).

I am not sure if I should continue on the ESXi route (compatibility with non-enterprise parts seems to be annoying, and the VMWare matrix is a nightmare, at least to me), or maybe run something like FreeNAS or UnRAID (Emby Container). I don't want to go the UnRAID/FreeNAS route, only to find out that I have taken a step-back in performance (like the Drobo and the Synology server application was). However, I do like the idea of getting four M.2 500GB NVMe drives running the OS, so I never have to worry again about losing my VMs(or containers)/metadata. OR just install a linux server OS, then run all my servers as containers on that.

I see that I can set a GPU to passthrough to an Emby container in UnRAID, and get GPU transcoding. So that seems like a perk.

So, I am curious what everyone thinks about potential hardware setup, and what my best option may be for the OS to take advantage of said hardware. I am putting this post in hardware, because the hardware I am buying is going to change depending on what OS(s) can take advantage of it (give best performance).

When it's all said and done, I'll want a full replacement for the Emby Server (currently ESXi VM) and my torrent server (ESXi VM) and a couple other servers (Home assistant, Minecraft, whatever else I am playing with). With the best possible performance. How much RAM do you think? (like 32 or 64 DDR4 will be way faster than the 128GB that I hardly use currently) Intel CPU (probably, right?), Nvdia Video card seems to be a good choice (around 2070 looks fine)...? RAID on SSD or NVMe, at the OS level? Is this just going ridiculously overkill? Can anyone help with some ideas please? I would really appreciate any insight to help me make some decisions before it starts getting too hot in the garage.

Thank you!

 

Link to comment
Share on other sites

BAlGaInTl

Personally, I prefer Unraid/Docker to FreeNAS/Jails.  It's what I'm running. It seems that there are more options to getting everything running in a Linux environment vs BSD. 

I ran OMV for a while as well (Linux/Docker) and it worked quite well.  It just took a bit more tinkering. I started looking into drive pooling, and it was going to take quite a bit of work (but do-able). Unraid makes everything (relatively) simple.  You are building the hardware anyway, so you can try both Unraid (free trial) and FreeNAS.

 

Link to comment
Share on other sites

ScottRTL

How was/is the performance on UnRAID? Setup looks easy enough, not too worried there (slightly worried about qBittorrent behind VPN inside a docker container) Can you get a CLI into each docker container to do things like modify their openvpn config? I'm still getting used to the whole container idea...

Any idea on the Intel vs AMD and Nvidia vs AMD arguement? Or is it just Intel/NVidia always?

Edited by ScottRTL
Link to comment
Share on other sites

BAlGaInTl

With Unraid/Docker, your containers basically have access to bare metal.  So performance isn't really much different from just using a Linux install.  The performance will only be affected by everything else running.

So it really depends on what you want to do. A good Intel chip has great encoding/decoding built in through QuickSync.  But if you have a lot of different containers running, the multiple cores of AMD CPU are hard to beat.  You lose out on the QuickSync though.

I'm not using hardware decoding right now and my performance is great on an older Xeon processor (4 core/8 thread).

Getting hardware decoding with Docker will take some tweaking.  CPU is easy.  If I were building right now, here's how I would look at it:

-If Emby is my main application, and I expect to be transcoding frequently, I'd go with a later model (previous gen is fine) Intel and take advantage of QuickSync for free.

-If I wanted a more powerful all-round server that can also run some virtual machines, I'd go with an AMD 39XX or Threadripper for the extra cores. You can always add an Nvidia card for transcoding later if needed.   

 

Link to comment
Share on other sites

ScottRTL

Thanks for the insights.

I am seeing that a new Nvidia card can only transcode 2 streams at the same time is that true? Is it true of an Intel CPU with quicksync too?

Emby is my main applicaiton for sure, not to undersell the other applications though, but they sure don't need much power (like I said, qbittorrent, minecraft, home assistant, etc.)

So you think like a 3790x on a b450 or x570 with the 4x NVMe running UnRAID in double pairity and like a Nvidia 1660 would be okay? Or a last gen i5 (I guess I don't need Nvidia GPU if I have that intel CPU though..? I'm just finding it hard to make that final decision and start the build...

Link to comment
Share on other sites

BAlGaInTl
1 hour ago, ScottRTL said:

Thanks for the insights.

I am seeing that a new Nvidia card can only transcode 2 streams at the same time is that true? Is it true of an Intel CPU with quicksync too?

Emby is my main applicaiton for sure, not to undersell the other applications though, but they sure don't need much power (like I said, qbittorrent, minecraft, home assistant, etc.)

So you think like a 3790x on a b450 or x570 with the 4x NVMe running UnRAID in double pairity and like a Nvidia 1660 would be okay? Or a last gen i5 (I guess I don't need Nvidia GPU if I have that intel CPU though..? I'm just finding it hard to make that final decision and start the build...

I believe the trans-code stream "limit" is 2 with the standard drivers. The other option would be to get a professional level card that does not have the limit.

The full list of capabilities is here:

https://developer.nvidia.com/video-encode-decode-gpu-support-matrix

It really is tough to make a decision if you are buying new.  Hopefully some more members will drop in and give their thoughts.

I don't think there is a 3790X AM4 part from AMD.  Not sure what you meant there. There is 3700X and 3900X.  The 3950X is the top of the line right now on AM4. 

It depends a lot on your budget as well. Did you have a budget limit in mind?

Unraid doesn't run on a hard drive.  It runs on a USB drive, so no need for NVMe drives really. The most common setup is USB, SSD Cache, HDD storage. Some will do 2x Cache drives for parity, but I just run a backup routine of my appdata to my HDD pool in case it fails.

 

 

Edited by BAlGaInTl
Link to comment
Share on other sites

ScottRTL

Yeah, 3790x was a typo.

I don't want to go crazy here. I feel like my HP DL360 isn't too bad, its form factor/loudness just doesn't work in my basement.

I wouldn't mind just getting like an i5 or i7 (my gaming PCs are 3700x and 1700x) for that quick sync and saving on buying the Nvidia GPU since it's only 2 streams anyways. Right now, transcoding is rare(ish) and my Xeon 5650s aren't really cutting it.

More than anything I want to make sure I have fast, stable containers/VMs with enough CPU/RAM and a way to not lose everything if a drive or two fails. I'm thinking of just getting 4 2TB 7200RPM HDDs (2 in parity) then like a 500 Gb NVMe for cache. (just for cover art loading etc.When it lags, I rage)

But basiclly, if I go with a Ryzen, I need to get a GPU, right? So I can put that money into getting an i7, and skip the GPU... when I transcode via software, the CPU is really going to jump. I'll make sure to get a good MoBo with the fastest USB and a nice small form factor USB drive for UnRAID (it feels really weird to me that it runs off the USB)...

Edited by ScottRTL
Link to comment
Share on other sites

BAlGaInTl
5 hours ago, ScottRTL said:

Yeah, 3790x was a typo.

I don't want to go crazy here. I feel like my HP DL360 isn't too bad, its form factor/loudness just doesn't work in my basement.

I wouldn't mind just getting like an i5 or i7 (my gaming PCs are 3700x and 1700x) for that quick sync and saving on buying the Nvidia GPU since it's only 2 streams anyways. Right now, transcoding is rare(ish) and my Xeon 5650s aren't really cutting it.

More than anything I want to make sure I have fast, stable containers/VMs with enough CPU/RAM and a way to not lose everything if a drive or two fails. I'm thinking of just getting 4 2TB 7200RPM HDDs (2 in parity) then like a 500 Gb NVMe for cache. (just for cover art loading etc.When it lags, I rage)

But basiclly, if I go with a Ryzen, I need to get a GPU, right? So I can put that money into getting an i7, and skip the GPU... when I transcode via software, the CPU is really going to jump. I'll make sure to get a good MoBo with the fastest USB and a nice small form factor USB drive for UnRAID (it feels really weird to me that it runs off the USB)...

Why not try running a copy of Emby on your 3700X and see how it handles transcodes?  Just to get an idea.

Link to comment
Share on other sites

Tur0k

Some of those array re-slivering (rebuild) processes sound highly suspect as SMR failures. I would recommend using PMR and maintaining array types that support more than 1 drive failure for data you want to keep.

 

Here, I am still running my home server on ESX6.7. I also virtualize Freenas and emby on it. Hardware wise it still runs on a mix of old prosumer equipment (Ryzen 7 1700, ASRock taichi motherboard, originally 32 ECC unbuffered RAM, 800 watt gold PSU, 2x SSDs (VMHOST and vm system drives, and backup drive for vmhost, 4x8TB seagate Iron wolf NAS drives (for freenas) with some sprinkled older series enterprise equipment (LSI 9207-8i HBA for drive management. And originally an Intel i340-T4 4 port gigabit NIC. Case wise I keep it in a plinkusa 3u case. I paid the extra for variable speed cooling fans that are quiet and put the drives in fanned enclosures.

 

I have made some upgrades.

1. I switched my VM system drives to run on a 1TB NVME M.2, and have them setup to backup to a second SSD weekly.

2. I expanded to 64 GB ECC unbuffered memory.

3. I switched out the NIC to an Intel X520-DA2 2 port SFP+ 10 gigabit NIC. With a DAC cable to my core switch.

4. 4 more 8TB seagate Iron wolf NAS drives.

 

 

I throw a lot of compute and memory at the emby VM and it software transcodes without much trouble. I did have to work with a buddy on a script to purge the transcode cache at regular interval. Otherwise, It all works pretty swimmingly for me.

 

Future upgrades:

1. Split out storage to a separate system running at 10G network bandwidth.

2. Upgrade to a 9300 series 16i HBA (12 Gbits/s and more internal interfaces

3. ISCSI connections to storage across the 10Gbps network backbone.

4. Upgrade vmhost to a newer iteration of ryzen 8c16t cpu and motherboard.

I going to add in

5. I have an older 1070 I have been playing with getting fitted into my server (3rd PCIE is a tight fit so I will need an extension cable) and passing that through to my emby Linux vm. If that works I could theoretically replace with a newer card. I think there are some hoops you have to run to trick the vm to thinking it is a local install and getting the card to pass-through, then som other tricks to get more than 2 transcoding streams on a consumer card.

 

 

Sent from my iPhone using Tapatalk

Link to comment
Share on other sites

Riggs

In short, the processor is the most important thing, and the hardware acceleration will depend on your pocket and the graphics card you can afford.

After going through this forum and that of other media servers, at the end of the day, and after reading problems, I always come to the same conclusion, a good device as a client and hopefully direct playback is the best. Still, I have no problems with phones or the like that need transcoding.

Link to comment
Share on other sites

  • 1 year later...
Dizzy49
On 7/1/2020 at 7:16 AM, Tur0k said:

 4 more 8TB seagate Iron wolf NAS drives.

I throw a lot of compute and memory at the emby VM and it software transcodes without much trouble. I did have to work with a buddy on a script to purge the transcode cache at regular interval. Otherwise, It all works pretty swimmingly for me.

First off, how are those Ironwolf Drives?  I just had 3x 10TB die on me one week out of warranty :(

Secondly, care to share that script? :D

To take advantage of the 12Gbit on the HBA wouldn't you need to use SAS drives?  Is it worth it? (I ask that as I'm taking 4 days straight to copy things from drive to drive).

 

I just had 5 drives crash on me and I'm in the process of rebuilding (adding 4x 18TB Seagate Exos).  I'm looking into VMs, but I need to stick to Windows.  I did FreeNAS and Unraid a LONG time ago and it was very difficult to maintain and to do simple things.  I am not a linux person, I've tried many times over the years, but when it takes me 2 hours to figure out how to install or upgrade something, it's not worth it any more :(

I'm curious what case you are using/went with.  I currently have a Norco 4224 but they are officially out of business, one of the backplanes died, and they have issues with larger drives.

I currently have an i7-7700K with 32GB of ram.  I would have gone with 64GB, but the design on the Gigabyte Z270X-UD3 board is terrible and pretty much any CPU cooler blocks one of the ram slots.  I have my OS on a 250GB Samsung Pro SSD, I download to a 1TB Samsung Pro SSD, and I have 16 drives in my Drivepool (Stablebit).  I just added two 250GB Samsung NVMe drives, but I'm not sure the best way to use them right now.  Debating between using PrimoCache and adding one or both to help speed up the drives, or using one as a SSD Optimized in the balancer for Drivepool.  I only use onboard video right now.  I've had 4 streams transcoding at once a few times and didn't see any issues. So far the biggest issue I've seen is when the hotel I'm at has terrible bandwidth and even reencoding down to 1mb isn't enough.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...