Library SIze - Limitations / How big is your library?

[mrees 6]

Hi all,

 

First up, love this product. Thanks so much to each and everyone that contributes to this

Im a super new user, but from what Ive seen so far, I love it. Im migrating over from My Movies and the process is quite a tedious one

 

Some brief observations and quick questions

 

Ive only stated the indexing process and so far Ive elected to catalog mainly my TV series data

Post that I will be looking at movies, music videos, home videos, then photos and music

 

Are there any limitations on library sizes that I should be aware of?

Im hitting small issues after the TV series indexing where Im receiving 'No Items' on quite a regular basis. Existing out of emby for WMC and re-opening seems to fix the issue, but it could happen again a short time later. Im also getting spinning circles when I initially open my IOS app. Sometimes they stop and the data is displayed, other times the app needs to be closed down 3 or 4 times before it works correctly. This happens remotely or on the LAN

Server resources should be able to handle it. I7 chipset, 12BG RAM, mainly sitting around 5GB

Im concerned once I have cataloged all my data, the system may not be able to handle it

 

Some stats on the files I am wanting to index in Emby

TV Series - Approx 15TB

Movies - Approx 10TB

Home Movies - Approx 5TB

 

So off the cuff (before growth), my library size for Emby to catalog is initially going to be around 30TB (excluding photos, music etc), with more to come

 

Is this going to be OK?

Whats the typical library sizes of emby users out there?

 

With the data I have above, are there any best practices to adhere to, or things to avoid?

 

Cheerz

\m/

Edited June 18, 2016 by mrees

[Spaceboy 2573]

30 odd terabytes here and I don't do anything particularly special, just follow the recommended file structure in the wiki

[mrees 6]

Awesome stuff

Do you experience any performance issues?

How responsive is your system?

 

Im struggling at the moment with libraries not responding well etc and Im trying to track down the root cause

[Happy2Play 9780]

Lot depends on your setup.  For me primarily Movies and TV everything pretty menu opens within a second or so.

[mrees 6]

Lot depends on your setup.  For me primarily Movies and TV everything pretty menu opens within a second or so.

 

Thanks for your reply

There must be something wrong with my setup of Emby then

I only have around 14TB indexed at the moment and its quite problematic

(WMC interface regularly fails to load content. IOS devices mainly don't load exiting from app 3 or 4 times. Slow web app performance in browsing library etc.)

[ebr 16171]

Size in bytes would be fairly irrelevant.  The more important number would be number of items.  For instance, a library of 1000 10GB movies vs. one of 100,000 100k audio files.

[the1legend 45]

10,000 TV Episodes

4,200 Movies

116,000 Audio Tracks

 

TV and Movies are no problem but like ebr mentioned, navigating through the music is slow (and sometimes crashes the client). I currently catalog the music but use Subsonic for most of the playback because it's lighting fast  even with 116,000 Songs. I'm not sure what makes the two so different but if Emby could process as fast as Subsonic, I would be in media heaven. As far as the client options and look of the interface Emby wins hands down.

[bfir3 117]

Yeah, it's hard for me to use Emby for my music playback when Subsonic works so smoothly regardless of collection size. And obviously the Subsonic music apps are much more mature and provide a lot of nice functionality that the Emby apps don't have like proper queue management and auto syncing played items to the device.

 

Also with the latest versions of Emby server and my large library (65TB combined movies+TV) I also experience the slowness in the web client as you've described, and it is unusable. If you would like to try an unsupported older version of Emby server, the one I recommend is 3.0.5782.0. Keep in mind that if you use this version, you won't be able to use the latest Emby theater app nor will you be able to access the Emby theater client from your browser. You can still access the server via the normal web client, and using Kodi. I use this server version and have no problems accessing my collection, even the music navigation is fast (not Subsconic level of smoothness, but acceptable at least.

[pir8radio 1312]

I guess I can chime in..  Seems like more and more people are moving their large collections to Emby recently..   I have been using emby for quite a while, I don't have a HUGE collection, but it is reasonably large when it comes to items databased.   Adding chunks items to my library, I found once you hit about the 100k items mark, emby starts to show minor slow down issues when moving around the web GUI... It gets progressively worse as you add items. 

 

I have about 140,000 MP3's, 13,000 Music Videos, and around 460 uncompressed movies.

 

In my setup I have given Emby every possible advantage I could..  My Emby Cache is stored on a RAM Disk that saves to the OS disk when shut down.  My OS disk is a set of mirrored SSD's, My media is stored on a RAID using mechanical disks, in Raid 5, the raid controller has Flash Backed Write cache modules installed all disks are SAS.  I have excellent disk performance.   I have 48Gb of 1600Mhz ECC ram. 20 cores (hyper-threading) worth of Xeon processor...   No bottle necks for Emby...    So the minor slowing is probably due to how Emby deals with the database, I think they are working on making things more efficient..    

Edited January 26, 2017 by pir8radio

[mrees 6]

Thanks so much for everyone's feedback

It gives me a good starting point to how to structure my media

I just need to work out whats causing these current slownless issues. I have about 60k of TV series files currently, so from what most people are saying the system should be pretty snappy at this stage of the collection

 

I think for the time being I will avoid cataloging music and photos. WMC can do that anyway and most of the time i 'try' to play it via Sonos (all though their system is total garbage on local music storage as their library tops out anywhere north of 40,000 tracks)

 

@@pir8radio, dude you are intense!

Your setup is crazy eyes. I gives me a brilliant example of how the application runs when you throw the best hardware situation at it.

 

Thanks everyone for taking the time to respond

[lorac 118]

I have 4200+ movies (of which only 3,700 are shown) and find the ET app choppy to browse them unless I go into a specific genre. TV works quite well because even if you have lots of shows they are broken up into seasons which makes a big difference.  All told I have about 50TB worth of media. 

 

It does take some time to get everything indexed but it's worth it.

[Koleckai Silvestri 1154]

My library is smaller than most here. 

 

1000 movies; 5000 episodes; 40,000 songs. Currently have 20 TB of storage. Don't really have issues with the database.

[boro9449 2]

I have a similar issue. I have to restart in WMC a couple times before the files show up. It might be how I have everything setup but I would be intrested in any other feedback thanks for your help.

[dcook 299]

I tried your guest account and even via the browser the music section is noticeably much slower than the movie section.

I think the devs need to take a break from adding new features and to work on performance issues and bugs

 

 

 

 

 

I guess I can chime in..  Seems like more and more people are moving their large collections to Emby recently..   I have been using emby for quite a while, I don't have a HUGE collection, but it is reasonably large when it comes to items databased.   Adding chunks items to my library, I found once you hit about the 100k items mark, emby starts to show minor slow down issues when moving around the web GUI... It gets progressively worse as you add items. 

 

I have about 140,000 MP3's, 13,000 Music Videos, and around 460 uncompressed movies.

 

In my setup I have given Emby every possible advantage I could..  My Emby Cache is stored on a RAM Disk that saves to the OS disk when shut down.  My OS disk is a set of mirrored SSD's, My media is stored on a RAID using mechanical disks, in Raid 5, the raid controller has Flash Backed Write cache modules installed all disks are SAS.  I have excellent disk performance.   I have 48Gb of 1600Mhz ECC ram. 20 cores (hyper-threading) worth of Xeon processor...   No bottle necks for Emby...    So the minor slowing is probably due to how Emby deals with the database, I think they are working on making things more efficient..    

 

 

 

Feel free to test my server if you want..    guest.notallmine.net       login: guest   pw: LetMeIn!  

 

My guest account is not always online, so don't get used to it people..  

[mattykellyuk 18]

wow how do you guys backup and store all that data?

[spootdev 56]

wow how do you guys backup and store all that data?

To another server hosting a ZFS array and sending snapshots.

[lorac 118]

I don't backup but the data is stored on 12 * 6TB raid 6 array. Even if a drive fails the array can be rebuilt in less than a day.

 

Sent from my STV100-3 using Tapatalk

[JeremyFr79 228]

I don't backup but the data is stored on 12 * 6TB raid 6 array. Even if a drive fails the array can be rebuilt in less than a day.

 

Sent from my STV100-3 using Tapatalk

And still run a very high risk of another drive or more failing during the highly stressful rebuild process killing your array completely.  RAID should never be relied upon for preventing data loss EVER.

[JeremyFr79 228]

My current library is 1700 movies, 6000 episodes and 7000 songs 18TB, it is small in comparison to many others here.  It's hosted on a 24 drive RAID 50, Emby is running on my VM host, it's drives are sitting on a 4 drive RAID 5 array of SAS 10k disks.  Performance and speed is rock solid on my setup.

[lorac 118]

I'll keep that in mind.

 

Sent from my STV100-3 using Tapatalk

[JeremyFr79 228]

I'll keep that in mind.

 

Sent from my STV100-3 using Tapatalk

The problem is that losing more than 1 drive simultaneously is not uncommon, I've seen scenarios where someone has lost 3 of 8 drives all at the same time.  There's no recovering from that with RAID 5/6 your array is bye bye forever unless you want to ship off and have a SUPER expensive recovery done.  Even if you're lucky enough to have 1 drive fail and are able to get a rebuild going your chances of losing 1 or more additional drives during the rebuild are extremely high, especially with drives as large as you're using.  RAID 5/6 was fine with smaller drives (read sub 1TB) and its actually ok to use with SSD (no spindles no problem) but outside of that it should never be looked at as a form of protecting data from loss.  And really no form of RAID ever should you always have to expect the worst because when you don't and you aren't prepared, that's when the worst happens to you.  I have worked in and managed data centers for a large part of my life and let me tell you there's a very good reason why every company on earth still backs up every last bit of data to tape on a daily basis.

[Deathsquirrel 745]

wow how do you guys backup and store all that data?

 

External drives.  I have backups of all my rips plus the original discs.  When I replace a drive in my server to gain space the old one gets turned into a backup drive.  I connect it to a SATA to USB adapter and fill it up.  The old drives are smaller than the new ones so they fill up faster.  When that i an issue I just buy a drive specifically for backup.  Easy to manage and not too much money since more than half my backups are on drives I've otherwise retired.  I've spent about $200 on external 5 tb drives to supplement the retired drives in the last 4 years or so.

 

Some consider a full backup overkill.  They're either downloading their content or have never ripped and encoded TV.  Rip all ten seasons of stargate sg-1.  They episodes aren't in order on the discs and don't have titles shown in the episodes so figuring out which episode is which is just miserable tedium.  You won't willingly do it twice

[dcook 299]

RAID is not backup, it is to protect you from a physical drive failure, if you have a RAID setup, you should also have a backup

[JeremyFr79 228]

RAID is not backup, it is to protect you from a physical drive failure, if you have a RAID setup, you should also have a backup

I prefer backup(S) myself I maintain backups with 2 different offsite vendors, they also maintain multiple copies of said files so I can go back through file changes if I need.  All said and done cost me very little money to boot.

[pir8radio 1312]

My current library is 1700 movies, 6000 episodes and 7000 songs 18TB, it is small in comparison to many others here.  It's hosted on a 24 drive RAID 50, Emby is running on my VM host, it's drives are sitting on a 4 drive RAID 5 array of SAS 10k disks.  Performance and speed is rock solid on my setup.

 

 

I prefer backup(S) myself I maintain backups with 2 different offsite vendors, they also maintain multiple copies of said files so I can go back through file changes if I need.  All said and done cost me very little money to boot.

 

 

Just curious.....   hosted server, and two offsite backup companies taking on 18TB each plus the traffic between backup sites. How much are you paying per year to host and backup emby? Sounds interesting.  

 

I mean I get what your saying super protected, but I'v been using a raid 5 with hot standby for many years...  I also have a 16TB drobo setup that will allow two failures, I have not lost a file since I used to store things on individual drives in the 90's. That said, I DO replace my drives every few years, by swapping them out now and then at random times when I have extra cash, plus I use enterprise SAS drives (now a days), except in my drobo, they are SATA Hitachi Ultrastar drives, Just like them, have not had one fail.... YET... 

 

I do recommend Drobo products...  Not great for commercial work (high traffic) but great for media storage..  easy to expand without having to know "raid stuff"..   Good for the general user..      I use the Drobo B810i now and its been great.    You don't need super speedy drives for your media collection...   

 

If you want a snappy emby server keep the cache, and other goodies on a fast drive or RAM Disk.

 

 

A decent cheap unlimited backup service:  https://www.backblaze.com/  

Edited June 24, 2016 by pir8radio