WHS 2011 / Emby Server Rebuild to free Bottle Necks

[mrees 6]

Hi all,

 

Hope you are all well

I was hoping you could help me with some much appreciated advice:

 

Due to the fact that my server is apparently not upto scratch for Emby (slow response times), I'm looking to ramp things up a bit to kill off any bottle necks

 

What I currently have in my server rig is as follows:

 

WHS 2011

Gigabyte P5-UD4P (rev 2) w/ Intel i7 860 Quad Core CPU (http://www.gigabyte.com.au/products/product-page.aspx?pid=3435#sp)

Norco RPC-4224 4RU Chassis w/ 24 x 3.5 HDD hotswapable bays (utilising SAS connectors and chasis backplane)

12GB RAM (Due to WHS limitations only 8GB can be used)

Areca 1261 Raid PCiE card running 2 x Raid6 Arrays WD NAS PRO (18TB + 15TB - Backups, Music, Home Movies)

Onboard 6CH Sata (BIOS Hacked for greater than 2TB volume support) running Raid5 Array WD NAS PRO (30TB - TV & Movie location)

Onboard 2CH Sata running Main OS - 7200 RPM standard drive in RAID chipset configuration

 

I'm looking at options to see where I can speed the system up

 

One thought is to upgrade the MB, Memory, system HDD & CPU

Obviously quite pricey. Ive just out layed $2600AUD in HDD upgrades, so WAF on this wouldn't be huge

 

The second is to look at a total rebuild and consider the following changes:

Increase RAM to 16GB (Maximum this board can support)

Install Server 2012 R2 (to maximise RAM above) and get a clean build. The system hasn't been rebuilt since it was installed in 2011

Utilise SSD for OS drive and Emby cache (RAID 0? with daily backups to USB)

OR

Install SSD for OS drive, Install secondary SSD for emby cache (not in raid)

Utilise same CPU and MB / RAID Card / Arrays

 

The question is, will the system above be able to handle concurrent transcoding sessions?

Ultimately Id like to be able to provide 5 concurrent sessions for family / friends and in house devices + one direct play device (I only utilise direct play at home via MCE / Emby for WMC)

I would imagine most external users would be connecting via iPhones or Apple TV's (just in case that gives more of a indication on whats required on the transcoding side)

 

Id love to hear everyones thoughts

 

I absolutely love Emby and Im keen to give her (?) what she needs to run at her optimum

 

Thankyou

[Tharnax 49]

@@mrees - This - "Install SSD for OS drive, Install secondary SSD for emby cache (not in raid) Utilize same CPU and MB / RAID Card / Arrays"  

 

I'm also running EMBY on WHS2011 but using DriveBender as my storage pool and not RAID.  My main hardware isn't much different, slightly new processor on an ASRock Rack Motherboard with 8GB of RAM.  

 

The reason for my reply is in my opinion you will get a significant performance boost just by moving your OS drive, maybe even your EMBY cache onto an SSD.  I have three of the servers above all running WHS2012 and all running EMBY.  Two of the servers have a WD Black as the OS drive, where as one server has a 256GB SSD as the OS drive.  There is an night and day difference in response time, load time, etc. from EMBY based on this one difference alone.

 

The difference is so significant I've been attempting to move the two other WHS2011 servers from the WD Black to an SSD drive for the OS as well but cloning programs just haven't been working well for me and I really don't want to do a full re-install.

 

If you can just start with the SSD upgrade I'd do that first.  The processor and ram should be good enough for 5 simultaneous transcoding + direct play. 

[dcook 299]

Your motherboard SATA is your bottleneck, use SSD for OS and another SSD for cache and metadata

[bfir3 117]

So you guys use the option to store all metadata in a central location on an SSD instead of having it in each library item's folder? If I have existing metadata in each item's folder will this cause any problems or does Emby just prioritize the central location? I may try doing this with a new installation this week as it sounds interesting, especially if it will yield noticeable performance improvements.

[Happy2Play 9780]

So you guys use the option to store all metadata in a central location on an SSD instead of having it in each library item's folder? If I have existing metadata in each item's folder will this cause any problems or does Emby just prioritize the central location? I may try doing this with a new installation this week as it sounds interesting, especially if it will yield noticeable performance improvements.

Having metadata with media means less downloading on new/clean installs.  If you don't use this feature the server has to reaquire this information and will store it in a central location (metadata folder).  Issues you run into is having a large enough SSD for your needs.  I have my metadata with media and my Emby server folder is 60GB do to 52GBs being my People folder alone, so I can't say how much more space you will need if metadata was co-located also.

[bfir3 117]

Having metadata with media means less downloading on new/clean installs.  If you don't use this feature the server has to reaquire this information and will store it in a central location (metadata folder).  Issues you run into is having a large enough SSD for your needs.  I have my metadata with media and my Emby server folder is 60GB do to 52GBs being my People folder alone, so I can't say how much more space you will need if metadata was co-located also.

What the hell, your people folder is 52gb just from the people images? That seems insanely high to me...

 

I'm barely above 1.5gb with 32k+ unique people in my database - of which nearly 20k have an associated image. If I'm hitting 1.5gb with 20k person images, you would need like 800k person images, or like 60k at a high 1mb per image average. Is this really the case? Jesus, that's crazy hahaha.

 

Either way, the SSD I'm running the OS off is 250gb with 130gb+ free space, and it's the only one I've got in the server at the moment. I am interested in installing one of the newer server versions this week or weekend anyway, and starting from scratch is the way I will go. I think I will try the centralized metadata if you think it will yield improvements, unless the 130gb is too little space.

[mrees 6]

Thanks for everyones advice so far

If there is an issue with performance, should I consider having library items (images etc) in media folders, or should I look to move to a SSD?

I don't mind storing on a SSD and buying one if there is performance gains

 

At the moment all images and backdops are being stored in the media folders

But to be honest, its a little messy for my liking so I wouldn't mind moving to a separate drive for that data

[Chillout 110]

I concur that moving metadata, cache, and transcoding to a SSD was the single greatest speed improvement to running the Emby server.  I upgraded to an i7-3770 and 32gb ram and nothing solved my lag until I moved metadata out of the media folders and into a SSD.

 

Images would take a while to load, or not load at all due to all the different spin up times.  I found files were getting fragmented constantly with all the read/writes.  Transcoding would studder when I was using the drive for other tasks.

 

On an SSD, images load up immediately and transcoding runs smoothly.

 

Wear and tear on an SSD?  After 2 yrs the drive still has 98% life left so it should last 97 years before its consumed.  SSDs are dirt cheap and all the metadata can be repopulated automatically in the event of a failure.  No permanent data loss.

[Happy2Play 9780]

What the hell, your people folder is 52gb just from the people images? That seems insanely high to me...

 

I'm barely above 1.5gb with 32k+ unique people in my database - of which nearly 20k have an associated image. If I'm hitting 1.5gb with 20k person images, you would need like 800k person images, or like 60k at a high 1mb per image average. Is this really the case? Jesus, that's crazy hahaha.

 

Either way, the SSD I'm running the OS off is 250gb with 130gb+ free space, and it's the only one I've got in the server at the moment. I am interested in installing one of the newer server versions this week or weekend anyway, and starting from scratch is the way I will go. I think I will try the centralized metadata if you think it will yield improvements, unless the 130gb is too little space.

Some of it is using Readon's IMDB IBN People Downloader so some the IMDB image can be a lot larger (up to 15MB) than TVDB or TMDB images (20-100KB).  Sure I could probably find a script and resize them.  I have about 215K people folders.

 

As for metadata/Images with media or localized it is personal preference since overall the server is using the cache folder for operation viewing.  Me it is a matter of a on stop shop having them with media.

[dcook 299]

Another important note, is your stripe size, you can set it very small for the metadata drive instead of a larger value which you would have for your media library drives.

[swhitmore 781]

I concur that moving metadata, cache, and transcoding to a SSD was the single greatest speed improvement to running the Emby server.  I upgraded to an i7-3770 and 32gb ram and nothing solved my lag until I moved metadata out of the media folders and into a SSD.

 

I just bought a new SSD for my server, and it has made a huge difference to load times. I still currently have all my metadata saved to media folders, and I do noticed that images can lag a little. How did you go about moving all your metadata to the SSD? Did you just let Emby re-download it all again? I remember asking about this in the past and being told it wouldn't have much of an impact.

Edited July 4, 2016 by swhitmore

[mrees 6]

Another important note, is your stripe size, you can set it very small for the metadata drive instead of a larger value which you would have for your media library drives.

 

Great tip

 

Thanks!

[swhitmore 781]

I just bought a new SSD for my server, and it has made a huge difference to load times. I still currently have all my metadata saved to media folders, and I do noticed that images can lag a little. How did you go about moving all your metadata to the SSD? Did you just let Emby re-download it all again? I remember asking about this in the past and being told it wouldn't have much of an impact.

 

Hi @@ebr I think I remember having this discussion with you before, but I can't seem to find it. Would you recommend moving metadata and images onto the SSD? It's 250GB, and I'm using it for the OS too. I think there should be enough room. Just on that, does anyone know of a way to find out how much room files are taking up on a drive, while filtering out other files by type (e.g. .mkv and .avi)? Thanks.