Jump to content

Wanting to build NAS that can transcode. What hardware?


famulor

Recommended Posts

famulor

Well the titles says it all really. I want to build a NAS/PC thats able to transcode up to 10 1080p (not the real 40+gb 1080p tho) movies at the same time. Essentially i want my own "cloud solution" so i can share with my nearest family. What hardware would i need in order to do this? 

 

 

We all use chromecast

 

Do emby allow my family to see the .srt files i upload? it would really help :)

 

For the most part my content is: Mkv, ISO, AVI and a little bit of mp4

 

 

 

Oh one last question: What kind of upload would this need? right now i have a 20mbit upload but im not sure thats enough? 

 

 

 

Best regards. Kevin

Edited by famulor
Link to comment
Share on other sites

JeremyFr79

10 1080p streams eh?  Well for hardware you'd want a recent gen Xeon, I'd recommend a dual Xeon server chassis, 24-48GB of ram, and for upload for 10 transcoded streams, figure at a very minimum 3.5mbps for ok looking 1080p so 35mbps plus headroom etc no less than 50Mbps on the upstream.  

 

As a reference I'm running dual Xeon X5570's (2.93 Ghz Quad core, 8MB, 6.4GB QPI, Hyperthreaded) and can get pretty close to 10 transcodes out of it but that's pushing it.  All depends on the source material as well.

Link to comment
Share on other sites

JeremyFr79

One last thing to consider is that if you're building a NAS and using software based RAID, you'll need processor overhead for that as well.

Link to comment
Share on other sites

famulor

10 1080p streams eh?  Well for hardware you'd want a recent gen Xeon, I'd recommend a dual Xeon server chassis, 24-48GB of ram, and for upload for 10 transcoded streams, figure at a very minimum 3.5mbps for ok looking 1080p so 35mbps plus headroom etc no less than 50Mbps on the upstream.  

 

As a reference I'm running dual Xeon X5570's (2.93 Ghz Quad core, 8MB, 6.4GB QPI, Hyperthreaded) and can get pretty close to 10 transcodes out of it but that's pushing it.  All depends on the source material as well.

Well my biggest filesize is 7 gb max so im pretty sure its not a "real 1080p" and i could do with about 7 streams. (not everyone will watch movies at the same time but its nice to know that they can). 

 

I have low knowledge of this sort of things (bitrate etc). 

 

 

How many concurrent HD streams are my current upload able to handle? im 95% sure that 10 concurrent HD streams wont ever happen but its nice to know that its possible. as a safety net :)

Edited by famulor
Link to comment
Share on other sites

JeremyFr79

Well my biggest filesize is 7 gb max so im pretty sure its not a "real 1080p" and i could do with about 7 streams. (not everyone will watch movies at the same time but its nice to know that they can). 

 

I have low knowledge of this sort of things (bitrate etc). 

 

 

How many concurrent HD streams are my current upload able to handle? im 95% sure that 10 concurrent HD streams wont ever happen but its nice to know that its possible. as a safety net :)

Filesize isn't really a factor in the equation.  Bitrate, Compression, Resolution, and audio format of both the source file and your transcoding output are what affects how much power you'll need.  For instance using my server as an example.  A typical MKV with say 8-10mbps bitrate, h.264, with AC3/DTS audio at 1080P, will use around 10% of proccessing power while being transcoded to 4-5mbps 1080p with AAC Audio.

 

As for your upload.  My rule of thumb is always take away 20% of your bandwidth for overhead.  So if you have a 20Mbps connection then that give's you 15Mbps of reliable upload with another 5mbps of overhead.  So if you're clients are say using 4mbps for streaming then you're looking at 4 streams consecutively.  

 

HOWEVER it's also not that simple as streaming with Emby is like streaming with say netflix or any other service.  It does NOT stream at a constant plateaued 4Mbps. Instead it send's burst's of data that average to a total of 4Mbps.  For instance if you were to watch the bandwidth on my router/firewall you'd see that while a person is streaming remotely from my server there are spikes of 8-12mbps with dips of nothing while someone is watching a 4Mbps stream.  Essentially it blast's out a chunk big enough to fill the buffer on the receiving end so that if there is a dip in bandwidth, or a "hiccup" it won't typically affect playback on the receiving end.

Link to comment
Share on other sites

JeremyFr79

@@JeremyFr79 I am interested in accomplishing something similar in a server build at some point. Storage would be a separate unit. You mentioned you have dual Xeon X5570's. I was wondering how the move to 6-core or 8-core Xeons would effect the quality of the transcode stream? Are the additional cores a significant boost? 

ffMpeg is threaded, so the more core's you throw at it the happier it'll be.  Quality won't be affected per se, but transcoding speed will.  As to say if you had an x5570, and an 8 core equivelant of it (which doesn't exist just an example)  The 8 core variant would in theory be able to transcode twice as fast or twice as many the same stream(s)

 

I run separate storage as well, so my Emby server is literally just that, dedicated to Emby and transcoding  The 5570's do a fantastic job for my workload which isn't as much as other's but I've done some pretty good testing on it running quite a few devices and it still kept up just fine.

 

The biggest problem which I tried to elude to earlier is that with Transcoding you can have a transcode that use's very little resource and another that use's a lot, there's a lot factors with the source file and destination transcoded stream that come into play.

Edited by JeremyFr79
Link to comment
Share on other sites

famulor

Filesize isn't really a factor in the equation.  Bitrate, Compression, Resolution, and audio format of both the source file and your transcoding output are what affects how much power you'll need.  For instance using my server as an example.  A typical MKV with say 8-10mbps bitrate, h.264, with AC3/DTS audio at 1080P, will use around 10% of proccessing power while being transcoded to 4-5mbps 1080p with AAC Audio.

 

As for your upload.  My rule of thumb is always take away 20% of your bandwidth for overhead.  So if you have a 20Mbps connection then that give's you 15Mbps of reliable upload with another 5mbps of overhead.  So if you're clients are say using 4mbps for streaming then you're looking at 4 streams consecutively.  

 

HOWEVER it's also not that simple as streaming with Emby is like streaming with say netflix or any other service.  It does NOT stream at a constant plateaued 4Mbps. Instead it send's burst's of data that average to a total of 4Mbps.  For instance if you were to watch the bandwidth on my router/firewall you'd see that while a person is streaming remotely from my server there are spikes of 8-12mbps with dips of nothing while someone is watching a 4Mbps stream.  Essentially it blast's out a chunk big enough to fill the buffer on the receiving end so that if there is a dip in bandwidth, or a "hiccup" it won't typically affect playback on the receiving end.

So if i understand correctly streaming with emby is a bit like streaming with streamnation where your device is "buffering" 10-15 seconds and then just plays the movie? I guess emby is filling the buffer fast enough so there isnt any "lag" on the vewing end?

Link to comment
Share on other sites

JeremyFr79

So if i understand correctly streaming with emby is a bit like streaming with streamnation where your device is "buffering" 10-15 seconds and then just plays the movie? I guess emby is filling the buffer fast enough so there isnt any "lag" on the vewing end?

I'm not familiar with streamnation personally but yes it buffers on the client end to avoid hiccups in the playback due to momentary issues with bandwidth.  every streaming service I've ever used works in the same way essentially.

Link to comment
Share on other sites

aaronouthier

This won't be for everyone, but check out FreeNAS. http://www.freenas.org/

 

I wouldn't even know about Emby, Plex, or OwnCloud, if I hadn't tried out FreeNAS and its plug-ins. That's just me, however. I am running it on an HP proliant DL-380 G5, with 2x Dual core Xeon CPUs, 16 GB RAM, and 3x 1TB HDDs. It uses up to 8 laptop drives for hard drive storage, and I considered using a 2.5" SSD for the boot volume, but opted to use the internal USB port and a 32 GB mSATA with USB enclosure.

 

Anyhow, install FreeNAS on your SSD, and format and copy your media to the remaining drives. Install desired plugins, and your media folders so that they're visible to the plugins. Configure as desired.

Link to comment
Share on other sites

deepseth

Vaguely similar, but opting for focus on disks rather than CPU performance, I got an ASRock C2750D4I. 8 core 'atom' CPU running the media server. Haven't tested this myself, but someone on the Plex forums reported having 4 simultaneous 1080p transcodes working without issue. Given that Plex and Emby both just use ffmpeg to transcode, I imagine results would be very similar.

 

Pros on the C2750D4i include the number of SATA ports onboard, AES-NI extensions if you encrypt your data, very low power (20W TDP) and decent multi-threaded performance. I think I've peaked at five simultaneous streams, but I suspect only two of them involved any transcoding, the rest would have been direct play.

 

A really simple way to calculate the amount of bw streaming any given movie will use (averaged over the entire length of the movie, not accounting for peaks) is to take the filesize and divide by the length, giving you the average bytes/sec that streaming that movie will take (without any transcoding). multiply by 8 for bits.

Link to comment
Share on other sites

famulor

@@famulor - have you looked at OwnCloud?

 

You could then use Drive Bender for your disc pooling solution.

 

Just a thought.

I'm not totally sure what Drive bender is? 

 

I guess OwnCloud is the "cloud software" i would need after i build my setup in order to make everything work? (yeah im new to this)

 

This won't be for everyone, but check out FreeNAS. http://www.freenas.org/

 

I wouldn't even know about Emby, Plex, or OwnCloud, if I hadn't tried out FreeNAS and its plug-ins. That's just me, however. I am running it on an HP proliant DL-380 G5, with 2x Dual core Xeon CPUs, 16 GB RAM, and 3x 1TB HDDs. It uses up to 8 laptop drives for hard drive storage, and I considered using a 2.5" SSD for the boot volume, but opted to use the internal USB port and a 32 GB mSATA with USB enclosure.

 

Anyhow, install FreeNAS on your SSD, and format and copy your media to the remaining drives. Install desired plugins, and your media folders so that they're visible to the plugins. Configure as desired.

FreeNAS is definitely one of the programs im going to look into when i actually build this setup.

 

Vaguely similar, but opting for focus on disks rather than CPU performance, I got an ASRock C2750D4I. 8 core 'atom' CPU running the media server. Haven't tested this myself, but someone on the Plex forums reported having 4 simultaneous 1080p transcodes working without issue. Given that Plex and Emby both just use ffmpeg to transcode, I imagine results would be very similar.

 

Pros on the C2750D4i include the number of SATA ports onboard, AES-NI extensions if you encrypt your data, very low power (20W TDP) and decent multi-threaded performance. I think I've peaked at five simultaneous streams, but I suspect only two of them involved any transcoding, the rest would have been direct play.

 

A really simple way to calculate the amount of bw streaming any given movie will use (averaged over the entire length of the movie, not accounting for peaks) is to take the filesize and divide by the length, giving you the average bytes/sec that streaming that movie will take (without any transcoding). multiply by 8 for bits.

Real newbie question but i guess i need something else to run a NAS for my needs other than "just" the ASRock C2750D4? 

 

Yeah that sounds like a semi easy way to calculate it but i have +3TB worth of content so :P unless i just find the largest filesize and do the calculation based on that?

 

 

You guys are really awesome and helpful! Especially when im such a big noob on all this

Link to comment
Share on other sites

It is a nit but you are not building a NAS (which stands for Network Attached Storage).  You are building a media server (which is much better than just a NAS device) :).

  • Like 1
Link to comment
Share on other sites

CBers

I'm not totally sure what Drive bender is?

 

Drive Pooling software.

Link to comment
Share on other sites

Gernash

If you using windows 8 or 8.1 then Windows Storage Spaces. was brought over from WHS when they canned it (Windows home server) in just START --> search --> Storage Spaces

 

But really you need to make a decision PC or NAS as backend.

 

NAS PATH:

1. Build a computer More Cores more streams for wireless clients (Wired clients should be configured for unc passthrough -no load)

2. Install Xpenology (howtos and question on their forums)-the network throughput is the reason for this (no I don't want to debate it)

3. Install Emby for synology http://emby.media/community/index.php?/topic/22921-synology-setup/

4. Enjoy

 

Network:

-to streaming 10x1080p you will need a layer 2 switch you can bind ethernet ports to the NAS or PC

 

-Wireless

-you will need a Gigabit AP that is capable of offloading traffic rather the queuing, if you find you need more then 1 AP to run all 10 streams at once.(WARNING:this can be quite complicated and causes hair loss)

 

Throw your cromecast away get android tv for static points(where you leave it plugged in all the time) http://www.banggood.com/buy/Android-TV.html this will reduce your transcoding needs. Use chrome cast for mobile needs

 

MINIX-NEO-X8-H-Plus

seems to be the ducks nutz (there are faster but with um Unique HW)

Unless in the US then nvidia shield for gaming goodness??

Edited by Gernash
Link to comment
Share on other sites

JeremyFr79

If you using windows 8 or 8.1 then Windows Storage Spaces. was brought over from WHS when they canned it (Windows home server) in just START --> search --> Storage Spaces

 

But really you need to make a decision PC or NAS as backend.

 

NAS PATH:

1. Build a computer More Cores more streams for wireless clients (Wired clients should be configured for unc passthrough -no load)

2. Install Xpenology (howtos and question on their forums)-the network throughput is the reason for this (no I don't want to debate it)

3. Install Emby for synology http://emby.media/community/index.php?/topic/22921-synology-setup/

4. Enjoy

 

Network:

-to streaming 10x1080p you will need a layer 2 switch you can bind ethernet ports to the NAS or PC

 

-Wireless

-you will need a Gigabit AP that is capable of offloading traffic rather the queuing, if you find you need more then 1 AP to run all 10 streams at once.(WARNING:this can be quite complicated and causes hair loss)

 

Throw your cromecast away get android tv for static points(where you leave it plugged in all the time) http://www.banggood.com/buy/Android-TV.html this will reduce your transcoding needs. Use chrome cast for mobile needs

 

MINIX-NEO-X8-H-Plus

seems to be the ducks nutz (there are faster but with um Unique HW)

Unless in the US then nvidia shield for gaming goodness??

10x1080P will stream just fine on a single gigabit connection...........Binding won't do any good anyways as it will be used as a fail-over not an increase of bandwidth.

Link to comment
Share on other sites

deepseth

Real newbie question but i guess i need something else to run a NAS for my needs other than "just" the ASRock C2750D4? 

 

Yeah that sounds like a semi easy way to calculate it but i have +3TB worth of content so :P unless i just find the largest filesize and do the calculation based on that?

 

Re: filesize, yeah, good idea to know your upper limit

 

The C2750D4I is a motherboard + cpu, you need to add a case, psu, ram, disks, and an OS. I use Debian and roll my own, but I've invested years into this so I'm relatively comfortable with a lot of it now

Link to comment
Share on other sites

famulor

It is a nit but you are not building a NAS (which stands for Network Attached Storage).  You are building a media server (which is much better than just a NAS device) :).

Ah my bad! Well then yeah im building a media server then :D

 

If you using windows 8 or 8.1 then Windows Storage Spaces. was brought over from WHS when they canned it (Windows home server) in just START --> search --> Storage Spaces

 

But really you need to make a decision PC or NAS as backend.

 

NAS PATH:

1. Build a computer More Cores more streams for wireless clients (Wired clients should be configured for unc passthrough -no load)

2. Install Xpenology (howtos and question on their forums)-the network throughput is the reason for this (no I don't want to debate it)

3. Install Emby for synology http://emby.media/community/index.php?/topic/22921-synology-setup/

4. Enjoy

 

Network:

-to streaming 10x1080p you will need a layer 2 switch you can bind ethernet ports to the NAS or PC

 

-Wireless

-you will need a Gigabit AP that is capable of offloading traffic rather the queuing, if you find you need more then 1 AP to run all 10 streams at once.(WARNING:this can be quite complicated and causes hair loss)

 

Throw your cromecast away get android tv for static points(where you leave it plugged in all the time) http://www.banggood.com/buy/Android-TV.html this will reduce your transcoding needs. Use chrome cast for mobile needs

 

MINIX-NEO-X8-H-Plus

seems to be the ducks nutz (there are faster but with um Unique HW)

Unless in the US then nvidia shield for gaming goodness??

When you say i need to make a decision wether to build a NAS or a media server/pc i really cant tell you because i dont know the pros/cons of either. 

Ive decided to reduce my "demands" to a max of 4 concurrent streams at once. How much upload would i need for that? 

 

I can't tell the family to throw away there chromecast since i doubt they are willing to buy something else :P

 

Edit: Oh and btw no matter what system i choose to build i would like it to be as low on power comsuption as possible :) 

Edited by famulor
Link to comment
Share on other sites

legallink

When you say i need to make a decision wether to build a NAS or a media server/pc i really cant tell you because i dont know the pros/cons of either

 

The difference between the two is the purpose of the build. While a media server/pc can also serve as file storage, a true NAS will struggle to meet the demands of a media server/pc. A NAS is historically really just what the title says.....network attached storage. A media server/pc is intended to transcode/serve/distribute media. The latter generally speaking requires much more power than the former, as transcoding media files can suck up a few cycles on a cpu.

 

My media server/pc doubles as my NAS in that it has about 16TB of storage on it, all with network shares beyond those used for emby. However, the real differentiator between the two will be the cpu you choose to run it on. A NAS can get away with a much much weaker cpu. You can get away with an atom cpu on a NAS, whereas to truly to emby some justice, I would be looking at an i5 or even a Xeon. You can get ones with lower tdp if electricity is your biggest concern (read lower tdp can equal more $$$), but the extra features are going to make a difference in transcodes. That being said, if transcoding isn't an issue, then very lightweight cpu's can serve the need, you just have to have the connection to serve the data.

Link to comment
Share on other sites

famulor

The difference between the two is the purpose of the build. While a media server/pc can also serve as file storage, a true NAS will struggle to meet the demands of a media server/pc. A NAS is historically really just what the title says.....network attached storage. A media server/pc is intended to transcode/serve/distribute media. The latter generally speaking requires much more power than the former, as transcoding media files can suck up a few cycles on a cpu.

 

My media server/pc doubles as my NAS in that it has about 16TB of storage on it, all with network shares beyond those used for emby. However, the real differentiator between the two will be the cpu you choose to run it on. A NAS can get away with a much much weaker cpu. You can get away with an atom cpu on a NAS, whereas to truly to emby some justice, I would be looking at an i5 or even a Xeon. You can get ones with lower tdp if electricity is your biggest concern (read lower tdp can equal more $$$), but the extra features are going to make a difference in transcodes. That being said, if transcoding isn't an issue, then very lightweight cpu's can serve the need, you just have to have the connection to serve the data.

 

Well i would need to be able to transcode 4 concurrent 1080p streams at max (rarely gonna happen but if worst comes to worst i need it to be able to handle it).

 

As for nas vs media server/pc i really have a hard time seeing the pros/cons on both systems with my needs. None the less the system needs to be able to "sleep" while not used and of course wake up when someone needs to watch something

Link to comment
Share on other sites

By definition you are building a Media Server.

 

Although some of the boxes marketed as NAS have adequate processing power for other things, the definition of NAS is just storage handling.

Link to comment
Share on other sites

xnappo

Agreed.  Four concurrent 1080p transcoding streams ain't a NAS - it is a media server.

 

Just built one myself - already had the 12TB of HD space.  Here is what I built:

https://secure.newegg.com/WishList/MySavedWishDetail.aspx?ID=28259986

 

I keep my machines for many years - this is overkill but will last a really long time.

Edited by xnappo
Link to comment
Share on other sites

  • 2 weeks later...
famulor

By definition you are building a Media Server.

 

Although some of the boxes marketed as NAS have adequate processing power for other things, the definition of NAS is just storage handling.

Alright well guess im building a Media Server then. Would it be easier if i ran the client software on it aswell for my own needs or should i just build another machine for that? I guess the transcoding option is a "on/off" kinda thing so i cant say like my client machine transcodes by itself but the chromecast needs to be transcoded for them? 

 

Agreed.  Four concurrent 1080p transcoding streams ain't a NAS - it is a media server.

 

Just built one myself - already had the 12TB of HD space.  Here is what I built:

https://secure.newegg.com/WishList/MySavedWishDetail.aspx?ID=28259986

 

I keep my machines for many years - this is overkill but will last a really long time.

I cant see the machine without logging in :/ the benefit with a media server (home build) is that i can upgrade over time as the money come in :)

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...