Would smaller, but more, media libraries bring better performance?

[theusedversion 35]

So I have posted several times about the less than desirable performance that I get from Emby.  I think a lot of it comes down to me having a large collection.  And I'm wondering if it would do better if I broke my media libraries up into smaller sections.  Right now I use just two libraries: Movies and TV.  And my collection is super large.  If I grouped some of the movies into different sections like genres, would I see better performance?  So instead of having just Movies and TV media libraries, I would have my 8 main genres: Action, Animation, Comedy, Documentary, Drama, Fantasy, Horror, Science Fiction plus TV because my TV collection isn't too large, I think.  Btw, I can do this because I go with what I consider the old video store method in regards to classifying genres.  That means my movies are only in one main genre right now so movies wouldn't appear in two sets of libraries if I did this.  Could I possibly see better performance if my media libraries were broken up into smaller groups like that?  I could also rearrange files on my hard drives so that genres typically are stored on one or two drives instead of having them scattered about.  It would be a huge undertaking but the pay off may be worth it if it significantly increases performance for me.  Or would it not make a difference in how Emby handles things if I had smaller libraries?

Edited May 16, 2016 by theusedversion

[JeremyFr79 228]

Breaking up your library will have little to no effect on performance, it's all 1 database no matter what. And I think the dev's will agree when I say there is pretty much no such thing as "too large a library" there are many of here with HUGE collections of mixed content that have 0 performance issues.  

 

Before responding I went ahead and read some of your other post's etc to see if I can get some history on your issue.  I saw that in another thread you were having poor search performance due to using External USB drives and allowing them to sleep and storing your data with the movies.  As was pointed out in that thread to you several days ago your best option would be to change your metadata settings so that images and metadata is stored on a drive that is not constantly going to "sleep"  

 

Personally I'm in the camp that one of the worst things you can do to a drive is let it constantly sleep/wake up.  I have literally 50+ hard drives in my system here and not a single one ever sleeps.  quite a few of them are 5+ years old and have 0 issues.  But I digress.  You're best option for your issue is to heed the advice provided to you in your other thread and use a central folder for all images/metadata instead of storing locally.  Also as the person who advised you of that stated you don't have to upload any images Emby does that automatically so changing the location where they are stored will add no extra work to you.  Honestly that is how it should be done in 99% of usage scenarios anyways.   

[theusedversion 35]

Breaking up your library will have little to no effect on performance, it's all 1 database no matter what. And I think the dev's will agree when I say there is pretty much no such thing as "too large a library" there are many of here with HUGE collections of mixed content that have 0 performance issues.  

 

Before responding I went ahead and read some of your other post's etc to see if I can get some history on your issue.  I saw that in another thread you were having poor search performance due to using External USB drives and allowing them to sleep and storing your data with the movies.  As was pointed out in that thread to you several days ago your best option would be to change your metadata settings so that images and metadata is stored on a drive that is not constantly going to "sleep"  

 

Personally I'm in the camp that one of the worst things you can do to a drive is let it constantly sleep/wake up.  I have literally 50+ hard drives in my system here and not a single one ever sleeps.  quite a few of them are 5+ years old and have 0 issues.  But I digress.  You're best option for your issue is to heed the advice provided to you in your other thread and use a central folder for all images/metadata instead of storing locally.  Also as the person who advised you of that stated you don't have to upload any images Emby does that automatically so changing the location where they are stored will add no extra work to you.  Honestly that is how it should be done in 99% of usage scenarios anyways.   

Thanks for the reply.  That's exactly what I wanted to know.  I had a sneaking suspicion that it wouldn't make a difference which is why I posted here. 

 

As for the hard drives, I have reluctantly set them to never spin down now.  There are a great many of opinions as to whether to let them spin down or not.  But I've always let them spin down and have had incredible luck with hard drives so I've always stuck with what worked.  We'll see how things go from here.  I really need to get a raid setup so I won't be so scared of hard drive failure and data loss.  But at close to 40TBs, it's not going to be cheap.

 

As for the images, I've grown to really like having them all saved with the video files.  I like having everything in a neat little folder, self-contained.  I almost wish I could save the images to both places.  A default place (such as my system drive) as well as saving everything with the video files.  That's an unnecessary redundancy but I think saving with the video files potentially offers the best possibility of compatibility with other (and future) programs and probably easier to use and make changes than having everything saved in one location.  For instance, today I noticed that my Game of Thrones collection was displaying an incorrect image.  So I just hopped over to its location and was able to easily spot the problem.  A poster.jpg filed had been downloaded and was overriding my folder.jpg cover.  Quickly deleted the poster.jpg and the problem was solved. 

[theusedversion 35]

But I want to point out, that setting my drives to never spin down does fix the problems that I have.  At least as far as I can tell.  I don't have any long-term usage under those conditions to verify.  But I think Emby must have some kind of problem with waking up drives and it shouldn't.  So I think it's something that should be fixable, right?  And therefore, there would be nothing wrong with my setup.  In many cases before, I could open up Emby and getting a spinning circle and then I could just go search for the movie manually and find it quicker and startup the movie before Emby would ever show my library to me.

[Luke 37174]

There's no problem with waking up drives because it's all handled automatically by the file system. It's just that at certain times a drive might have to spin up in order to satisfy a request and that delay leads you to think that something isn't working.

[theusedversion 35]

There's no problem with waking up drives because it's all handled automatically by the file system. It's just that at certain times a drive might have to spin up in order to satisfy a request and that delay leads you to think that something isn't working.

 I really don't think that's the case though.  There have been times where I have gone to each and every drive in File Explorer (down into the folder structure to where the video file, nfo, and images are saved) while waiting on the loading circle in Emby.  I did it this morning in fact. I was able to go into all drives will waiting for the loading circle on Emby.

[theusedversion 35]

I downloaded fraps and camstudio to catch a video of Emby performance. I could take a video and upload it of me doing this.

[Luke 37174]

ok, there's obviously more to satisfying a request than just file system access so those are not really comparable actions

[theusedversion 35]

ok, there's obviously more to satisfying a request than just file system access so those are not really comparable actions

I obviously have no idea how it all works. XD   So is it safe to say that storing metadata with the video files is not a recommended action?  And possibly shouldn't even be included in Emby?  Because it was specifically recommended for me when I started here. 

 

And also, why didn't I ever get these issues when using Emby through Kodi?  Does Kodi cache all the images and whatnot on the system drive?

[Luke 37174]

No, I wouldn't say that. The decision is use-case specific. There are pros and cons to either approach.

[Luke 37174]

The main argument against not having it is the same argument that can be applied to any optional feature. more ways of doing the same thing = more complication = more people thinking that something's wrong. But that of course is just a generalization and not specific to this one thing.

 

If you are now leaving your drives on 24/7, than the spin up/down, and image storing location is no longer of much concern.

[theusedversion 35]

Yeah, it has resolved my issues.  I was just hoping I didn't have to let them stay spinning all the time.

Edited May 16, 2016 by theusedversion

[Luke 37174]

if that's what you want, then what you should do is open a feature request topic asking that we store a cloned copy of all images underneath the server's program data folder. then we can use the cached versions while your drives stay sleeping until a full library scan or playback.

[JeremyFr79 228]

For my personal use I keep metadata/images on a separate array so as not to chew up storage space on my storage arrays, since they're all tuned for large file storage with larger striping sizes tons of "little" files can quickly use up a lot of space since say a single 4KB file could potentially use an entire say 1MB stripe in an array.  as you can see if you have 20,000 to 50,000 of these "little" files that could quickly chew through and waste a ton of space.  Instead they're stored on an array specifically tuned for small files which speeds up performance but also prevents small files from wasting space it doesn't need to.  

 

Storage tuning can become quite a task if you choose to delve into it but at the same time the payoff's can be quite beneficial.

Edited May 16, 2016 by JeremyFr79

[theusedversion 35]

For my personal use I keep metadata/images on a separate array so as not to chew up storage space on my storage arrays, since they're all tuned for large file storage with larger striping sizes tons of "little" files can quickly use up a lot of space since say a single 4KB file could potentially use an entire say 1MB stripe in an array. as you can see if you have 20,000 to 50,000 of these "little" files that could quickly chew through and waste a ton of space. Instead they're stored on an array specifically tuned for small files which speeds up performance but also prevents small files from wasting space it doesn't need to.

 

Storage tuning can become quite a task if you choose to delve into it but at the same time the payoff's can be quite beneficial.

I guess that's something I will have to learn about when I eventually go with a raid option. I wouldn't have even known to consider something like that. But I have zero experience with a raid setup so it's to be expected.

[Guest asrequested]

I have my drives set to spin down after 4 hours. It seems to work pretty well. But I don't have a super large collection, only about 600 movies plus a lot of music and shows

[theusedversion 35]

I have my drives set to spin down after 4 hours. It seems to work pretty well. But I don't have a super large collection, only about 600 movies plus a lot of music and shows

I have almost 4,700 movies and just checked my episodes and have almost 4,000 of them which is much larger than I thought. All of that scattered across 15 external drives and 1 networked drive. Quite a lot of files to bring up. XD

[Guest asrequested]

I have almost 4,700 movies and just checked my episodes and have almost 4,000 of them which is much larger than I thought. All of that scattered across 15 external drives and 1 networked drive. Quite a lot of files to bring up. XD

 

WOW! You should get a NAS system with RAID to consolidate and protect your collection. Very impressive collection. I would have a lot more, but I've had to rebuild a few times

Edited May 16, 2016 by Doofus

[speechles 1921]

Not bragging, just stating I have 5,000+ movies, 100,000+ episodes, 170,000+ mp3 and have no speed related issues.

 

You would be better off upgrade your ethernet to giga speeds. Having media drives networked across ethernet is slower than having the media drives directly connected to the box running the server via sata/usb. Running your media drives on the same box as the server speeds it up.

 

Sent from my Nexus 7 using Tapatalk

Edited May 16, 2016 by speechles

[Guest asrequested]

I'm starting to get a little performance anxiety, lol

[dcook 265]

How are your 15 external drives connected?  Are they all connected individually to your computer?

 

Its possible that you could be having a bottleneck on the USB bus as well with that many drives.

[theusedversion 35]

How are your 15 external drives connected?  Are they all connected individually to your computer?

 

Its possible that you could be having a bottleneck on the USB bus as well with that many drives.

I don't think so since my issues are resolved once I keep the drives from spinning down.  Regardless, my setup is like this:  I have only two usb 3.0 ports on my pc.  So to those two ports I have powered usb 3.0 hubs that can support the power consumption of the drives.  So with those two ports, I have 9 drives connected (I have spots for expansion).  Then I have 4 more usb 2.0 ports on the back of my pc which 4 more drives are connected directly.  Then i have two usb 2.0 ports on the front of my pc.  Attached to those two ports are non powered usb 3.0 hubs which both have two drives connected.  It's not ideal.  But it's how I slowly built my collection.  Buy a hard drive here and there and before you know it, I have a ton of hard drives.  I am moving back to the US in 3 months so I will look at getting a raid setup once I do.  Niche computer parts are super expensive where I am at.  The same usb hub that's on Amazon for $20 is $80 here.    But luckily, things like hard drives are priced normally.

[dcook 265]

If you look in Device manager are all the devices under the same USB root?  

 

Just because there are separate ports on the front and back of the computer does not mean they are separate USB bus, most likely all your USB2 is together on one bus and all USB3 is together on another.  Then you add more USB hubs which just degrades the performance even more.

 

When you go looking for something, it has to go through the entire USB bus and look through all those drives

 

As mentioned already and I think you also agree, you are much better off to use local disk or a proper NAS, I suspect the cost of a NAS would not be more than the cost of 15 external drives and usb hubs.

[theusedversion 35]

If you look in Device manager are all the devices under the same USB root?  

 

Just because there are separate ports on the front and back of the computer does not mean they are separate USB bus, most likely all your USB2 is together on one bus and all USB3 is together on another.  Then you add more USB hubs which just degrades the performance even more.

 

When you go looking for something, it has to go through the entire USB bus and look through all those drives

 

As mentioned already and I think you also agree, you are much better off to use local disk or a proper NAS, I suspect the cost of a NAS would not be more than the cost of 15 external drives and usb hubs.

No, it has 3 root hubs.  I imagine one for the 3.0 ports, one for the front 2.0 ports and one for the 4 2.0 ports on the back.  I don't think it's a problem there because I have zero performance issues when the drives aren't spinning down.

 

Yeah, I'm sure it would have been cheaper in the long run.  But I never intended to have a collection this large so I just bought a hard drive here and a hard drive there.  Long before I even needed to think about a raid enclosure. 

[Guest asrequested]

I've only ever had one external drive, (I have since made it an internal). It's quite slow. I can actually hear it spinning up. That could possibly be your issue.