Library SIze - Limitations / How big is your library?

[lorac 118]

I've had drives fail but never more than one at a time. I use the LSI megaraid 9260i. I suppose some kind of tape backup would be good. I'd be interested to hear about your little cost backup solution though.

 

Sent from my STV100-3 using Tapatalk

[mrees 6]

I'm going to need to eventually migrate off WHS at some stage

Are there any recommendations for a reasonably decent array size? (approx 60TB of usable space)

I will also look to upgrade the hardware as the ccurrentlimitation on my MB of only 12GB of RAM won't cut it if I want something like a RAM drive for cache

Just from the technical logistics, Im assuming RAM drives are significantly faster than something like a SSD? The question would be if that really makes a difference on a large library

 

I don't really care for the domain capability, I just want a fast server (for media retrieval), web accessible for file retrieval and most importantly, something that can easily handle backups of windows clients with large backup capabilities (one of my windows clients needs 10TB of backups on its own with ever changing content as its a video editing PC)

Preferably not unix for the server. Im going to maybe look at that option for a backup later down the path (thanks for the idea @spootdev)

I dont want to use acronis or the like, just doesnt gel well with me. I absolutely love WHS's backup facilities, they (appear) to work flawlessly, even on large clent backups, and have saved my bacon time and time again. The restore process works perfectly (in my experience)

 

It seems a lot of people here run Windows Server

Im guessng there must be a few MSDN subscriptions available as some of the options (like data centre edition) can be pretty pricey for home use

[spootdev 56]

Have you looked at FreeNas?  It's basically a webgui on top of Freebsd 10 and ZFS, samba, etc

[JeremyFr79 228]

Just curious.....   hosted server, and two offsite backup companies taking on 18TB each plus the traffic between backup sites. How much are you paying per year to host and backup emby? Sounds interesting.  

 

I mean I get what your saying super protected, but I'v been using a raid 5 with hot standby for many years...  I also have a 16TB drobo setup that will allow two failures, I have not lost a file since I used to store things on individual drives in the 90's. That said, I DO replace my drives every few years, by swapping them out now and then at random times when I have extra cash, plus I use enterprise SAS drives (now a days), except in my drobo, they are SATA Hitachi Ultrastar drives, Just like them, have not had one fail.... YET... 

 

I do recommend Drobo products...  Not great for commercial work (high traffic) but great for media storage..  easy to expand without having to know "raid stuff"..   Good for the general user..      I use the Drobo B810i now and its been great.    You don't need super speedy drives for your media collection...   

 

If you want a snappy emby server keep the cache, and other goodies on a fast drive or RAM Disk.

 

 

A decent cheap unlimited backup service:  https://www.backblaze.com/

Sorry I led to some confusion with my post, when I said hosted I meant on a server I own in my residence not offsite.  As for the backups at the end of the day I could care less about my media.  While I could back it up to my backup provider at no difference in cost than what I pay now, my piddly 35Mbps upload on my connection just wont allow that.  Now when I upgrade to a gig connection I may change that.  Right now I keep about 350-400GB of data offsite in real-time, it's important stuff like docs, pictures, VM and Machine image backups. etc.  I use Code42 Crashplan as my main backup provider. I've been with them for years now.  I pay $60(us) a year for unlimited storage with them.  

[JeremyFr79 228]

Just curious.....   hosted server, and two offsite backup companies taking on 18TB each plus the traffic between backup sites. How much are you paying per year to host and backup emby? Sounds interesting.  

 

I mean I get what your saying super protected, but I'v been using a raid 5 with hot standby for many years...  I also have a 16TB drobo setup that will allow two failures, I have not lost a file since I used to store things on individual drives in the 90's. That said, I DO replace my drives every few years, by swapping them out now and then at random times when I have extra cash, plus I use enterprise SAS drives (now a days), except in my drobo, they are SATA Hitachi Ultrastar drives, Just like them, have not had one fail.... YET... 

 

I do recommend Drobo products...  Not great for commercial work (high traffic) but great for media storage..  easy to expand without having to know "raid stuff"..   Good for the general user..      I use the Drobo B810i now and its been great.    You don't need super speedy drives for your media collection...   

 

If you want a snappy emby server keep the cache, and other goodies on a fast drive or RAM Disk.

 

 

A decent cheap unlimited backup service:  https://www.backblaze.com/

Ultrastars are Enterprise drives as well, matter of fact you'll find them used quite a bit in EMC products in many data centers around the world.

[mrees 6]

Have you looked at FreeNas?  It's basically a webgui on top of Freebsd 10 and ZFS, samba, etc

Ill prob look at that for the backup of the main system, just not for the actual Server OS

Im not a huge unix fan for day to day stuff. I need everything and anything to be able to run on the box at the drop of a hat

[dcook 299]

Windows 10 is free until end of July I think.

 

 

I'm going to need to eventually migrate off WHS at some stage

Are there any recommendations for a reasonably decent array size? (approx 60TB of usable space)

I will also look to upgrade the hardware as the ccurrentlimitation on my MB of only 12GB of RAM won't cut it if I want something like a RAM drive for cache

Just from the technical logistics, Im assuming RAM drives are significantly faster than something like a SSD? The question would be if that really makes a difference on a large library

 

I don't really care for the domain capability, I just want a fast server (for media retrieval), web accessible for file retrieval and most importantly, something that can easily handle backups of windows clients with large backup capabilities (one of my windows clients needs 10TB of backups on its own with ever changing content as its a video editing PC)

Preferably not unix for the server. Im going to maybe look at that option for a backup later down the path (thanks for the idea @spootdev)

I dont want to use acronis or the like, just doesnt gel well with me. I absolutely love WHS's backup facilities, they (appear) to work flawlessly, even on large clent backups, and have saved my bacon time and time again. The restore process works perfectly (in my experience)

 

It seems a lot of people here run Windows Server

Im guessng there must be a few MSDN subscriptions available as some of the options (like data centre edition) can be pretty pricey for home use

[mrees 6]

Windows 10 is free until end of July I think.

Thanks but I personally wouldn't use Windows 10 as a server

No native backup server capabilities or remote file access capabilities, and its just not designed to be a server

When you have a large set of data, you want the system to be pretty rock solid. And forced updates don't gel well with me at all in that environment

 

Plus I really hate the interface for some reason. Don't get me wrong, it performs well. I just feel everytime I use it that its either set out for a child, or that Im entirely lost in the new design and I cant find anyhing without searching for it (which makes me feel kinda stupid . Im one of those people that talks about 'the good ol days'

[drashna 47]

Personally, I use Windows, and StableBit DrivePool.  

 

All of the data is mirrored. Yeah, it's a lot of space "wasted"... but not really.  Additionally, any number of disks may fail... and the remaining data is still accessible. That's because everything is stored on NTFS volumes, as normal files. 

 

As for content: 

• TV Shows: 16TB  (360 shows)
• Movies: 8.5TBs (1500 movies) 
• Anime: 3.6TBs (381 shows/movies)
• Music: 350GBs (670 artists) 
• "Linux ISOs": 12.5TBs
• And some misc stuff I haven't 

And that's the total, BEFORE duplication/mirroring. 

 

The Emby server process takes about 1.5GBs of RAM at any given time. Namely because of the size. 

 

 

Thanks but I personally wouldn't use Windows 10 as a server

No native backup server capabilities or remote file access capabilities, and its just not designed to be a server

When you have a large set of data, you want the system to be pretty rock solid. And forced updates don't gel well with me at all in that environment

 

Completely disagree.  

 

Windows 10 is more than capable of being ran as a server.  

 

No native backup?  Umm, you're dead wrong. It includes Windows Backup, which is the client version of Windows Server Backup. THe difference? Some of the UI elements for managing it, and some of the options. But otherwise identical. 

 

No remove file access?  Just install the IIS role, create a website, and share it.  You can do this super basically, or you can get more complicated (such as using SSL certs, authentication, etc). 

Also, it supports "Incoming connections" for VPN support. May not be great, but it works. 

But SeaFile or the like may be better. 

 

As for updates, you can disable them. Simple. Just turn off updates. If you're super paranoid, run "services.msc" and disable the Windows Update service. 

 

 

Plus I really hate the interface for some reason. Don't get me wrong, it performs well. I just feel everytime I use it that its either set out for a child, or that Im entirely lost in the new design and I cant find anyhing without searching for it (which makes me feel kinda stupid . Im one of those people that talks about 'the good ol days'

 

I don't know, I love it. Much cleaner than 8's.  And if you really want older... there are plenty of mods/tools to do that. Such as Start8, or the like. 

 

As for searching, I do that for convenience, and it works really well. Rather than clicking 10 times to get to the specific option I want.

Edited July 3, 2016 by drashna

[mrees 6]

 

Completely disagree.  

 

Windows 10 is more than capable of being ran as a server.  

 

No native backup?  Umm, you're dead wrong. It includes Windows Backup, which is the client version of Windows Server Backup. THe difference? Some of the UI elements for managing it, and some of the options. But otherwise identical. 

 

No remove file access?  Just install the IIS role, create a website, and share it.  You can do this super basically, or you can get more complicated (such as using SSL certs, authentication, etc). 

Also, it supports "Incoming connections" for VPN support. May not be great, but it works. 

But SeaFile or the like may be better. 

 

As for updates, you can disable them. Simple. Just turn off updates. If you're super paranoid, run "services.msc" and disable the Windows Update service. 

 

 

 

Sorry, I wasn't real clear there

I meant that WIndows 10 has no backup server capabilities

 

It's imperative that my main server has the capabilities to backup all the clients on the network to a bare metal image.

Im not a fan of Acronis or the like

I've LOVED WHS's client backup feature and its something thats really negotiable as its saved my bacon multiple times

I just want it to work each and every time

 

I also have no intention these days of messing around with IIS and setting up web servers

The WHS 2011 web interface for file retrieval and RDP proxy capability is fantastic. I just want something to work without a lot of messing around

I used to get knee deep in things 10 years ago, these days I'm more interested in it just working straight out of the box and spending my time on watching the content rather than managing it

 

As far as the updates are concerned. I still like to update, but I like to choose what is installed and when

From my understanding of Windows 10, you don't have that capability unless you end up setting up a WSUS box to deploy updates, but i could be wrong

[eblue 1]

I'm using a product called FlexRAID running on Windows Server 2012. I have 20 SATA drives with a total of over 80TB. FlexRAID is interesting in that while it treats the drives as RAID with parity, they remain in Windows NTFS format. It also supports different sized drives within the same array. In the event that you lose a drive and it can't be recovered using parity the worst case is that you lose the contents of that single drive. I have actually had that happen before and only had to re-rip 40-50 movies. I've recently started migrating to 8TB drives which can hold 120+ blu-ray rips. Haven't lost any of those yet and I can assure you that I'm not looking forward to it. Backing up 80TB of data to any media is simply not practical. I consider the loss of one drive of movie rips to be an acceptable risk. I probably wouldn't feel the same if the content was stuff that was downloaded and hard or impossible to replace.

[Never_More 19]

I would like to first say thank you to the emby team for the great programs.

I guess I can chime in..  Seems like more and more people are moving their large collections to Emby recently..   I have been using emby for quite a while, I don't have a HUGE collection, but it is reasonably large when it comes to items databased.   Adding chunks items to my library, I found once you hit about the 100k items mark, emby starts to show minor slow down issues when moving around the web GUI... It gets progressively worse as you add items. 

 

I have about 140,000 MP3's, 13,000 Music Videos, and around 460 uncompressed movies.

 

In my setup I have given Emby every possible advantage I could..  My Emby Cache is stored on a RAM Disk that saves to the OS disk when shut down.  My OS disk is a set of mirrored SSD's, My media is stored on a RAID using mechanical disks, in Raid 5, the raid controller has Flash Backed Write cache modules installed all disks are SAS.  I have excellent disk performance.   I have 48Gb of 1600Mhz ECC ram. 20 cores (hyper-threading) worth of Xeon processor...   No bottle necks for Emby...    So the minor slowing is probably due to how Emby deals with the database, I think they are working on making things more efficient..    

 

 

 

Feel free to test my server if you want..    guest.notallmine.net       login: guest   pw: LetMeIn!  

 

My guest account is not always online, so don't get used to it people..  

Wow Nice server setup - it respond very fast, well done.

 

My setup is basic, and still my server runs great - I have 8 family members hitting it 24-7 all at one time - I have had 4 try the same movie at the same time and still had no problem - right now I'm sitting at over 4k worth of movies. I have some TV and very little music. I think if the newer users would follow the setup directions and use the server computer for nothing but a server, A lot of problems wouldn't keep showing up - . Keep up the maintenance/update the OS, run scans, keep, not only the server but all the other software up to date.

 

 

.