Lets Talk Library Scans...

[iamspartacus 40]

Since setting up Emby just a few days ago I'm seeing some fairly long library scan times relative to what I see with Plex.  Furthermore, it seems during these library scans the performance of my media playback is affected.  I've got some pretty large libraries.  

 

Once all metadata has been downloaded and updated, shouldn't scans go rather quickly?  With Plex my entire TV library updates would take less than 1 minute, sometimes less than 30 seconds.  Shouldn't the scans only be looking for changes?

 

Is there any way to omit certain libraries from scans (like Music libraries for example) or setup a scheduled job to only scan a certain library (ie. TV show library typically needs to be scanned regularly between the hours of 8pm-12pm each night).

 

My libraries have the following settings enabled:

• Download artwork and metadata from the internet
• Save artwork and metadata into media folders
• Download images in advance
• Enable real time monitoring
• Automatically merge series that are spread across multiple folders

Edited September 26, 2017 by iamspartacus

[PenkethBoy 2068]

Usually after the first few full library scans the metadata and artwork is fully populated

 

However Scan time depends on a lot of things

 

Has all your metadata/artwork been fully downloaded/populated , responsiveness of metadata and artwork providers, how many media files you have - there are some pauses to not overload the third party providers with too many requests in a short time.

Data is local to the Emby server not on network drives.

 

my whole movie and TV library takes less than a minute - but i may have less media than you

 

i would suspect that if you turn off the merge series option things will speed things up

 

You cant control individual library scans via a scheduled job. But what i do is just have on the real time monitoring and that picks up all changes i make with both extract image chapters options selected for all my libraries.

 

IIRC Plex has(or did) its own copy servers of metadata providers which probably gives it a bit of an edge on initial population but after that Emby only checks for new data with new or changed media - unless you set a library to update the data periodically. 

[Jdiesel 1431]

Are you generating chapter images during the library scan? 

[mastrmind11 722]

Disabling downloading of images in advance will help as well.  With that disabled, emby will only go get the images of stuff you're looking at (as opposed to downloading every single image for every actor, etc in your library)

[iamspartacus 40]

Disabling downloading of images in advance will help as well.  With that disabled, emby will only go get the images of stuff you're looking at (as opposed to downloading every single image for every actor, etc in your library)

 

I prefer to keep that enabled.  I'm hoping that will make things more responsive for my clients.  If that means it will take a few days for things to speed up so bet it.

 

 

Are you generating chapter images during the library scan? 

 

I am not.

 

Usually after the first few full library scans the metadata and artwork is fully populated

 

However Scan time depends on a lot of things

 

Has all your metadata/artwork been fully downloaded/populated , responsiveness of metadata and artwork providers, how many media files you have - there are some pauses to not overload the third party providers with too many requests in a short time.

Data is local to the Emby server not on network drives.

 

my whole movie and TV library takes less than a minute - but i may have less media than you

 

i would suspect that if you turn off the merge series option things will speed things up

 

You cant control individual library scans via a scheduled job. But what i do is just have on the real time monitoring and that picks up all changes i make with both extract image chapters options selected for all my libraries.

 

IIRC Plex has(or did) its own copy servers of metadata providers which probably gives it a bit of an edge on initial population but after that Emby only checks for new data with new or changed media - unless you set a library to update the data periodically. 

 

I seem to have sped things up to where it now takes 5 minutes.  My storage is stored on the network (though connected via 10Gb so speed shouldn't be an issue) but I was pooling two sets of NFS mounts for redundancy.  Changed the paths to single NFS shares instead of the pools seems to have sped things up.  5 minutes I can deal with and maybe that will even speed up some as more and more of the metadata gets populated.

[PenkethBoy 2068]

i have a 10gb network as well but local drives make a big difference a lot more than i was expecting

[iamspartacus 40]

i have a 10gb network as well but local drives make a big difference a lot more than i was expecting

 

I see, that's disappointing.  Unfortunately there's nothing I can do to change that.  I run Emby in a docker inside a Linux VM which is on an ESXi cluster (I know...inception).  I do this for maximum uptime/redundancy.  I could run Emby in a docker on my UnRAID server(s) but that would defeat the entire purpose of the redundant network/storage I've spent the past 2 years configuring.

[PenkethBoy 2068]

i use windows server 2012r2 for Emby with Dive Pooling to keep it simple! 

 

some of your performance maybe related to docker>linux>esxi>unraid talking to each other although not a massive amount

 

anyway 5 minutes is not long and you control when that happens - if you have realtime monitoring on then its only a catch all for the odd file that might get missed (real time does not work while a scan is also running)

[iamspartacus 40]

i use windows server 2012r2 for Emby with Dive Pooling to keep it simple! 

 

some of your performance maybe related to docker>linux>esxi>unraid talking to each other although not a massive amount

 

anyway 5 minutes is not long and you control when that happens - if you have realtime monitoring on then its only a catch all for the odd file that might get missed (real time does not work while a scan is also running)

 

I had zero issues with this with Plex but maybe Emby's backend being different has an affect.  Like I said nothing I can do about that.  I'll never go back to local storage, too much flexibility with my current setup.

[kjp4756 41]

I fought this same issue for almost a year.  What I had to do was put each movie in to its own folder and have emby save metadata in with each movie.  This reduced my library scan from over 30 minutes to 13 seconds.  I am on the dot net core emby-server.  Using the mono version I think scan times were just under 30 seconds.

[PenkethBoy 2068]

Ah yes i have done that for all my media in the recent past so that might be why i was surprised local media made so much difference over pulling from the network - will have to test when i get time

 

One other thing if you have lots of photos in the extrafanart directories - i made the mistake of adding lots(100+) of photos to each directory for my movies - this will significantly slow down library scans as well - i have reduced these to 10 photos each and and library scans came back to normal times

[iamspartacus 40]

I had Emby put the metadata in each folder but since my media is accessed over the network I'm wondering if that was a poor decision. I wonder if I should try having the metadata stored on the same disk as my Emby server since thats how I had it setup with Plex.

 

Anyone know how would accomplish this based on the fact that all my current metadata is stored with the media?

[mastrmind11 722]

You'd have to do a rescan w the setting disabled.

 

But saving the metadata w/ the media is the preferred approach anyway.  Your media is accessed over the network already, so why would your metadata storage location affect anything?  I'd argue the opposite.

[rhodges 49]

If you store metadata with the media, are the images served from there? If so, an option to store with media but cache and serve from another location would be nice.

Edited September 27, 2017 by rhodges

[iamspartacus 40]

You'd have to do a rescan w the setting disabled.

 

But saving the metadata w/ the media is the preferred approach anyway. Your media is accessed over the network already, so why would your metadata storage location affect anything? I'd argue the opposite.

Will rescan with it disabled move the metadata or just leave it with the media and create new copies in the new location?

 

I don't agree with the notion though that if the media is accessed across the network than why not the metadata. You want the metadata to be very quick access so that when you are flipping through different screens in the Emby client the images load right away. It's frustrating to change screens and have to sit at a black screen for a few seconds while the images load. The actual media on the other hand is only accessed when you hit play and at that point waiting 3-5 seconds for the video to start is no big deal. I just hadn't realized how much of an effect storing the images on a network resource would have.

Edited September 27, 2017 by iamspartacus

[mastrmind11 722]

Will rescan with it disabled move the metadata or just leave it with the media and create new copies in the new location?

 

I don't agree with the notion though that if the media is accessed across the network than why not the metadata. You want the metadata to be very quick access so that when you are flipping through different screens in the Emby client the images load right away. It's frustrating to change screens and have to sit at a black screen for a few seconds while the images load. The actual media on the other hand is only accessed when you hit play and at that point waiting 3-5 seconds for the video to start is no big deal. I just hadn't realized how much of an effect storing the images on a network resource would have.

They get cached.  I have my metadata stored w/ my media on a NAS and I have 0 lag when navigating around.

[iamspartacus 40]

They get cached.  I have my metadata stored w/ my media on a NAS and I have 0 lag when navigating around.

What client are using?

[mastrmind11 722]

What client are using?

ATV, Android, Chrome.

[iamspartacus 40]

ATV, Android, Chrome.

Maybe the issue is my Samsung Smart TV app. I have a shield TV that I'll give a shot soon but for now I'm relegated to my bedroom until my kitchen/living room renovation is done.

[Happy2Play 9780]

The only reason to save metadata and images with media is for faster rebuilds.  This allows the server to read existing information without having to redownload everything.

[iamspartacus 40]

The only reason to save metadata and images with media is for faster rebuilds.  This allows the server to read existing information without having to redownload everything.

 

But if the images are cached as @@mastrmind11 says, what reason would you have NOT to store the metadata with your media?

Edited September 27, 2017 by iamspartacus

[Happy2Play 9780]

But if the images are cached as @@mastrmind11 says, what reason would you have NOT to store the metadata with your media?

 

Really it is personal preference as some people think it is clutter in there media folders,  But storing with media is the recommended method for faster rebuilds if needed.

 

 

Yes everything gets cached also.

Edited September 27, 2017 by Happy2Play

[iamspartacus 40]

What would kick off a library scan other than "Scan Media Library" scheduled tasks?  Currently I only a scheduled task to scan my media libraries at 5:00am and 5:00pm.  Yet I've seen at last 2 library scans happen during the day today.  Does something else trigger a scan of all libraries?

[Luke 42077]

We'd have to look at the server log to answer that. you can learn how to attach it here:

 

https://emby.media/community/index.php?/topic/739-how-to-report-a-problem/

 

Thanks.

[Coxeroni 26]

But if the images are cached as @@mastrmind11 says, what reason would you have NOT to store the metadata with your media?

As an example I mount my media storage read-only because I have to (plexdrive). So I can't store the metadata alongside the media.