Jump to content

Server build: multiple 4K & HD Streams


CVXDEV

Recommended Posts

Hi all,

I am playing around to optimize/configure an Emby-Media Server for multiple 4K and HD Streams. I’ve configured already a set-up, but it’s not 100% ideal for my need-case so I would like to adjust my configuration.

Therefore I would be happy if you would have some inputs for the following Scenario and help me with my questions/assumptions:

Goal:

  • 10x 4K Direct-Play (to Nvidia Shields + 4K TVs) - wired
  • 10x HD Direct-Play to HD-Clients (e.g. Laptops/PCs) - wired
  • 20x HD Direct-Play to HD-Clients (Tablets, Phones) - over WiFi

Network:

  • 10GB Ethernet from Server to 10GB-Switches / then 1GBE to Clients

Other:

  • “No” Transcoding needed
    • Files will be pre-encoded and have a 4K & HD-Version for the devices
      • HD files as H.264 - mp4
      • 4K in any format Shield can direct play
    • In case “little” Transcoding or Direct Stream (e.g. Audio) is needed, should be realized by Intel QuickSync

 

For my current config, I planned to be able to transcode via a Quadro P2000, but my tests haven’t been that successful – so I think the “non-transcoding” option is the better fit.

Current Test-Set-up:

  • CPU: Intel Xeon E5-2620v4 8x 2.10GHz
  • MB: Supermicro X10DAX
  • RAM: 32 GB
  • SSD: Samsung 860 EVO
  • GPU: Nvidia Quadro P2000
  • OS: Ubuntu

This setup gave me 15x direct streams easily (2x 4K / 13x HD), but with transcoding it stuttered quickly.

 

Removing the need of having the separate GPU, I would like to switch to a smaller form factor, ideally a Mini-ITX board (I know complete opposite to the above MB :)) and use an Intel CPU which supports QuickSync if needed.

I thought about a config like this:

  • CPU: Intel i7-8700 (6 core)
  • MB: TBD (1x PCIE x8 for the 10GB-Ethernet Card)
  • RAM: 32 GB
  • SSD: Samsung 860 EVO
  • OS: Ubuntu

 

Questions/Assumptions:

  • Will a 6 core CPU be powerful enough to do as many streams? For multi-streams, what’s more important the amount of cores or the overall thread rating?
  • Or will the SSDs be the bottleneck (“550 MB/s” read speed)
  • Which other constraints did I miss or any other input :)?

Thanks in advance!

PS: I have the Emby-Premiere Subscription and am happy to upgrade to the next level once needed...

Link to comment
Share on other sites

BAlGaInTl

That's way more power than I need.

 

My main concern would be the wireless clients I think. 20X simulatneous HD streams? What kind of wireless networking gear are you using? I'd be curious to see what others think about that.

Link to comment
Share on other sites

That's way more power than I need.

 

My main concern would be the wireless clients I think. 20X simulatneous HD streams? What kind of wireless networking gear are you using? I'd be curious to see what others think about that.

Planning to use multiple of the Unifi NanoHD:

https://unifi-nanohd.ubnt.com/

Edited by CVXDEV
Link to comment
Share on other sites

rliepins

I've been looking to set up a new server based on the I7-8700 as well.  Am thinking of the K model to overclock if I feel lucky.  As with your setup, I am hoping the Intel hardware QuickSync will improve transcoding and I won't need a GPU.  If you do go this route, please let us know and post what you build.  How do you propose utilizing your GPU and new CPU?  Can you use both hardware?  I do have an AMD FirePro S7000 lying around, though.  Anyone have experience with utilizing that GPU?

 

I'd LOVE to do 10Gb if I could.  Ubiquiti charges a pretty penny for their equipement.  A switch will cost more than the new CPU...

 

I'm also hoping to use the same rig to remux as rip as needed for DVDs and Blu-Rays.  Again, the i7-8700's hyperthreading has great ffmpeg numbers.  The next gen (9th) seems to be veering away from hyperthreading.  Has anyone looked at what that will do to setups such as these?

 

btw, awesome job on the v4 server.  Its great!

Link to comment
Share on other sites

How do you propose utilizing your GPU and new CPU?  Can you use both hardware? 

 

I'd LOVE to do 10Gb if I could.  Ubiquiti charges a pretty penny for their equipement.  A switch will cost more than the new CPU...

I would get rid of the GPU in favor of the smaller form factor (Mini-ITX).

If you want to have affordable 1/10GB-Switches, maybe have a look at MikroTik (https://mikrotik.com/).

Link to comment
Share on other sites

rliepins

I would get rid of the GPU in favor of the smaller form factor (Mini-ITX).

If you want to have affordable 1/10GB-Switches, maybe have a look at MikroTik (https://mikrotik.com/).

 

Would you expect the CPU and QuickSync to handle the load then?

 

Thanks for the recommendation!

Link to comment
Share on other sites

Would you expect the CPU and QuickSync to handle the load then?

Yes - as described above the idea would be to have mostly no transcoding - 4K & HD-files on the server and only remuxing or single transcodes for Wifi-Devices.

Link to comment
Share on other sites

vdatanet

I can transcode 4 streams from 4k to 720p 3 Mbps with an i5 8600 8th generation using vaapi. Therefore, if you do not require transcoding, there is no need for an additional gpu and you can stay in a reduced form factor.

Link to comment
Share on other sites

legallink

Yeah, I would abandon the need for 4k transcoding.

You didn't mention HDR or not.  That will impact clients, file formats, etc.

Wireless, that is going to be tough to stream that many streams, unless your files are significantly compressed.

I didn't get a sense for how much of that is/was concurrent.  Do you mean 10 4k streams, 10 HD streams wired, and 20 HD streams wireless all at the same time? Or that is max capacities?

One thing I didn't see was a secondary SSD/NVME drive for cache, metadata, etc.  I would do at least that.  You may even want to consider either some raid/drive pooling if you have drive bottlenecks depending on how many of those streams are concurrent.

@@pir8radio runs a pretty robust setup, so he may have more insight.

Edited by legallink
Link to comment
Share on other sites

pir8radio

Well, I'm in the middle of building this setup right now:  https://emby.media/community/index.php?/topic/66990-new-server-build-gpu-transcoding/

I take it to the datacenter this Friday and hopefully have some transcoding stats available for users.  That said, my current setup barely handles transcoding a 2 HDR HEVC 4k streams and its a xeon E5 2470 v2 10 core.   

 

But yea if you are not transcoding your bottleneck will be from drives to nic.   I don't think the 20 HD over wifi will be an issue if you have a few AP's with dual radios and balanced clients. But again like others have said if you are talking 40 concurrent streams, you will need an array and server that can handle the bandwidth.    

Link to comment
Share on other sites

Thanks all so far for your inputs! :)

 

You didn't mention HDR or not.  That will impact clients, file formats, etc.

 

I didn't get a sense for how much of that is/was concurrent.  Do you mean 10 4k streams, 10 HD streams wired, and 20 HD streams wireless all at the same time? Or that is max capacities?

 

One thing I didn't see was a secondary SSD/NVME drive for cache, metadata, etc.  I would do at least that.  You may even want to consider either some raid/drive pooling if you have drive bottlenecks depending on how many of those streams are concurrent.

  • Ideally, the 4K would be in HDR, according to the specs the Shield should be able to play it without issues ("Up to 4K HDR playback at 60 FPS (H.265/HEVC)" - https://www.nvidia.com/en-us/shield/specs/)
  • The mentioned streams are max capacity - but if needed it should ideally play all those streams concurrent :D
  • Thank you for the input of the SSD/NVME drive, I missed that - will definitely look into that and which Motherboards support it. Regarding Raid I am quite clueless/new to the topic - which Raid setup is the best for reading speed / max. concurrent streams?

 

Well, I'm in the middle of building this setup right now:  https://emby.media/community/index.php?/topic/66990-new-server-build-gpu-transcoding/

I take it to the datacenter this Friday and hopefully have some transcoding stats available for users.  That said, my current setup barely handles transcoding a 2 HDR HEVC 4k streams and its a xeon E5 2470 v2 10 core.   

 

But yea if you are not transcoding your bottleneck will be from drives to nic.   I don't think the 20 HD over wifi will be an issue if you have a few AP's with dual radios and balanced clients. But again like others have said if you are talking 40 concurrent streams, you will need an array and server that can handle the bandwidth.    

  • Your project looks awesome, will be a beast - enjoy the testing phase :-) As said, transcoding is crossed out for me, therefore I try to make it a "bit" smaller...
  • If we would talk about the 40 streams, do you have a suggestion on how to set-up the drive array to be able to handle the bandwidth? Thanks in advance!

 

Do you think a 6-core CPU like the i7-8700 can handle that or are more cores needed?

Edited by CVXDEV
Link to comment
Share on other sites

vdatanet

What operating system do you plan to use? I use ubuntu 18.10 with a pool drives, using ZFS, but I've only tested with 15 streams 1080p 5-8 Mbs without transcoding and results where ok.

Link to comment
Share on other sites

What operating system do you plan to use? I use ubuntu 18.10 with a pool drives, using ZFS, but I've only tested with 15 streams 1080p 5-8 Mbs without transcoding and results where ok.

I would also run Ubuntu, but the 18.04 LTS version though.

Link to comment
Share on other sites

vdatanet

I use one NVME drive for operating system, emby server, temp files... and 6 sata drives for my media, using a zfs pool. I'm using this motherboard: Gigabyte Z370 HD3

  • Like 1
Link to comment
Share on other sites

pir8radio

Thanks all so far for your inputs! :)

 

  • Ideally, the 4K would be in HDR, according to the specs the Shield should be able to play it without issues ("Up to 4K HDR playback at 60 FPS (H.265/HEVC)" - https://www.nvidia.com/en-us/shield/specs/)
  • The mentioned streams are max capacity - but if needed it should ideally play all those streams concurrent :D
  • Thank you for the input of the SSD/NVME drive, I missed that - will definitely look into that and which Motherboards support it. Regarding Raid I am quite clueless/new to the topic - which Raid setup is the best for reading speed / max. concurrent streams?

 

  • Your project looks awesome, will be a beast - enjoy the testing phase :-) As said, transcoding is crossed out for me, therefore I try to make it a "bit" smaller...
  • If we would talk about the 40 streams, do you have a suggestion on how to set-up the drive array to be able to handle the bandwidth? Thanks in advance!

 

Do you think a 6-core CPU like the i7-8700 can handle that or are more cores needed?

 

 

Stop and think about this.....    A decently encoded UHD 4k movie will be about 50-60 gigs each, encoded around 50-60 mbps.   The typical spinning hard drive read speed is 50-150 MB/s.  That said you can only get a x 4k streams from a single spinning drive.   Even with drive pooling you can have issues depending on how your movies are spread across the pool.  We are talking either an SSD array, or a high end spinning array, and you are trying to keep it small....

Edited by pir8radio
Link to comment
Share on other sites

Stop and think about this.....    A decently encoded UHD 4k movie will be about 50-60 gigs each, encoded around 50-60 mb/s.   The typical spinning hard drive read speed is 50-150 mbps.  That said you can only get a few 4k streams from a single spinning drive.   Even with drive pooling you can have issues depending on how your movies are spread across the pool.  We are talking either an SSD array, or a high end spinning array, and you are trying to keep it small....

Yes, was not thinking about spinning hard-drives - was planning on SSDs. With the input of legallink, i would currently plan with 1x NVMe SSD + 4-6x SSDs (like the Samsung Evo 860).

Based on the Spec, they should be able to theoretically read up to 550MB/s.

My math would look like this (please correct me if I mix something up):

  • 1x UHD Bluray = 100 GB with 128 Mbit/s (according to Wikipedia), which results in a datarate of 16 MB/s --> therefore 10x 4K = 160 MB/s read speed needed
  • 1x HD Video = 48 Mbit/s, which results into 6 MB/s datarate --> 30x HD = 180 MB/s read speed needed

So we would end up at 340 MB/s total.

Does this makes sense?

Link to comment
Share on other sites

pir8radio

Yes, was not thinking about spinning hard-drives - was planning on SSDs. With the input of legallink, i would currently plan with 1x NVMe SSD + 4-6x SSDs (like the Samsung Evo 860).

Based on the Spec, they should be able to theoretically read up to 550MB/s.

My math would look like this (please correct me if I mix something up):

  • 1x UHD Bluray = 100 GB with 128 Mbit/s (according to Wikipedia), which results in a datarate of 16 MB/s --> therefore 10x 4K = 160 MB/s read speed needed
  • 1x HD Video = 48 Mbit/s, which results into 6 MB/s datarate --> 30x HD = 180 MB/s read speed needed

So we would end up at 340 MB/s total.

Does this makes sense?

 

Yes you are actually correct, my math above was askew.  mbps and MB/s  ^_^  I'm still waking up....     You might be ok with drive bandwidth if going with SSD's.. You will still not want to put all of your eggs in one basket..      

 

Just to show you a few stats, here is the results from my evo SSD:

  Sequential Read (Q= 32,T= 1) :   545.595 MB/s
  Sequential Write (Q= 32,T= 1) :   104.563 MB/s
  Random Read 4KiB (Q= 32,T= 1) :   250.255 MB/s [ 61097.4 IOPS]
 
And here are the results from my spinning array:
  Sequential Read (Q= 32,T= 1) :  1585.232 MB/s
  Sequential Write (Q= 32,T= 1) :  1303.007 MB/s
  Random Read 4KiB (Q= 32,T= 1) :   271.405 MB/s [ 66261.0 IOPS]
 
It helps to have that data spread out across multiple drives evenly using raid vs a drive pool. 
Edited by pir8radio
Link to comment
Share on other sites

 

Yes you are actually correct, my math above was askew.  mbps and MB/s  ^_^  I'm still waking up....     You might be ok with drive bandwidth if going with SSD's.. You will still not want to put all of your eggs in one basket..      

 

Just to show you a few stats, here is the results from my evo SSD:

  Sequential Read (Q= 32,T= 1) :   545.595 MB/s
  Sequential Write (Q= 32,T= 1) :   104.563 MB/s
  Random Read 4KiB (Q= 32,T= 1) :   250.255 MB/s [ 61097.4 IOPS]
 
And here are the results from my spinning array:
  Sequential Read (Q= 32,T= 1) :  1585.232 MB/s
  Sequential Write (Q= 32,T= 1) :  1303.007 MB/s
  Random Read 4KiB (Q= 32,T= 1) :   271.405 MB/s [ 66261.0 IOPS]
 
It helps to have that data spread out across multiple drives evenly using raid vs a drive pool. 

 

Thanks for the stats - just for my understanding, the spinning array is a Raid 5 (according to your signature) or? How many drives do you use for the 16 TB?

How would you suggest to split the eggs between several baskets? ;-)

Link to comment
Share on other sites

vdatanet

According to my media:

 

My UHD rips are between 55000 kbps and 65000 kbps. They are remux  without cropping. We can talk about an average data transfer of 7.5 MBs. On the other hand my MicroHD rips are between 8000 kbps and 10000 kbps, so the average data transfer is 1.2 MBs.

 

(4 x 7.5) + (30 x 1.2) = 30 + 36 = 66 MBs

 

But this is my media, I don't know yours.

 

pir8radio has a better setup, but I can't afford that.

Link to comment
Share on other sites

According to my media:

My UHD rips are between 55000 kbps and 65000 kbps. They are remux  without cropping. We can talk about an average data transfer of 7.5 MBs. On the other hand my MicroHD rips are between 8000 kbps and 10000 kbps, so the average data transfer is 1.2 MBs.

(4 x 7.5) + (30 x 1.2) = 30 + 36 = 66 MBs

But this is my media, I don't know yours.

pir8radio has a better setup, but I can't afford that.

I like to plan with the "worst case" scenario and am happy if the reality works out better :-)

For the moment I have tested some sample 4K-files which have been between 52 Mbps and 108 Mbps - so the "reality" is somewhere in between ;)

Link to comment
Share on other sites

pir8radio

According to my media:

 

My UHD rips are between 55000 kbps and 65000 kbps. They are remux  without cropping. We can talk about an average data transfer of 7.5 MBs. On the other hand my MicroHD rips are between 8000 kbps and 10000 kbps, so the average data transfer is 1.2 MBs.

 

(4 x 7.5) + (30 x 1.2) = 30 + 36 = 66 MBs

 

But this is my media, I don't know yours.

 

pir8radio has a better setup, but I can't afford that.

 

 

I shop cheap...  (well, cheap compared to new prices)   all ebay and amazon!     

Link to comment
Share on other sites

vdatanet

I like to plan with the "worst case" scenario and am happy if the reality works out better :-)

For the moment I have tested some sample 4K-files which have been between 52 Mbps and 108 Mbps - so the "reality" is somewhere in between ;)

 

Ok, let's double data transfer rate for a worst scenario: 66 MBs x 2 = 132 MBs. The disk pool can perfectly provide this transfer rate. With 4 HDs of 4 Tb, we would have a pool of 16 Tb, with a cost of € 130 x 4 = € 520 (approx $ 590). A 16 TB SSD array costs a lot of money, I wish I could pay for it because it is the best solution because it can support a higher workload.

Link to comment
Share on other sites

pir8radio

Thanks for the stats - just for my understanding, the spinning array is a Raid 5 (according to your signature) or? How many drives do you use for the 16 TB?

How would you suggest to split the eggs between several baskets? ;-)

 

That test was my music drive, it is a raid 5 with 4 drives 2tb each.   the 16TB raid is now a raid 6 but it has 6 - 4tb drives at the moment with two open spots in the shelf. 

 

Split the eggs, like vdatanet said, either use a pool, like drive pooling in windows, or hardware mirror or something..   I've lost huge chunks of collections in the past, its nice to have raid on top of a backup...  this will also give you more lanes for traffic to travel, limiting some drive bottlenecks if any.. 

Link to comment
Share on other sites

That test was my music drive, it is a raid 5 with 4 drives 2tb each.   the 16TB raid is now a raid 6 but it has 6 - 4tb drives at the moment with two open spots in the shelf. 

 

Split the eggs, like vdatanet said, either use a pool, like drive pooling in windows, or hardware mirror or something..   I've lost huge chunks of collections in the past, its nice to have raid on top of a backup...  this will also give you more lanes for traffic to travel, limiting some drive bottlenecks if any.. 

Thanks - will investigate.

@all: thanks for all the input - highly appreciated.

I've ordered now the following config, will do some tests and keep you posted if interested:

  • CPU: Intel i7-8700 (6 core)
  • MB: ASRock Z390M-ITX/ac
  • RAM: 32 GB
  • SSD: Samsung 860 EVO for OS / cache
  • OS: Ubuntu
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...