Why does a scan take so long if the data exists and is with the media

[Gilgamesh_48 1240]

This is mainly a case of me wanting to know more.

Recently I have had a need to setup a new server. The library is actually stored on a network share and another server, much older, points at it. I have that server set up to store all data with the media so much, if not nearly all, data is already downloaded and available.

On my second server i set up a library pointing to that same location and configured it exactly the same as on the first server. However the scan is taking every bit as long as the original scans on the first server so it seems there is no advantage to storing data alongside the media as it seems to save no time.

1. Am I misunderstanding what storing the data alongside the media does?
2. What actually is the advantage of storing the data alongside the media?
3. What do I need to do to remove the local data and store the data the original way, should I choose to do so?

Thanks

[Ronstang 294]

10 hours ago, Gilgamesh_48 said:

1. Am I misunderstanding what storing the data alongside the media does?

Possibly but your experience is the same as mine.  I always store all files with my media including nfo, poster and other images and bif files.  This way all my data is always the same wherever I move it or however I access it.  But a while back I was having trouble on one of my servers when something kept getting corrupted and I would simply delete the database files and let it rebuild.  Even though all the data is there it took many hours to rebuild the database from scratch.  Now a rescan seems to pick up changes quickly but from nothing it takes forever.

 

10 hours ago, Gilgamesh_48 said:

2. What actually is the advantage of storing the data alongside the media?

I assume yet may be wrong but consistency.  My poster art is unique in that many of my images are original art from 50+ years ago that I have used paint.net to restore because I like original art with all writing even that in the white border margins and that can be hard to find in decent condition so I have spent a lot of time on artwork.  Plus, with my music collection I have a lot of older albums that were once hard to find artwork for 15-20 years ago and once again I used any source I could and manipulated it if needed in photoshop or paint.net.  Some are pictures of my album covers (vinyl) that have been reworked and resized.  Either way many of my images are unique to my server and I don't want to loose them so they stay with my movies and music and I have them backed up.

 

10 hours ago, Gilgamesh_48 said:

3. What do I need to do to remove the local data and store the data the original way, should I choose to do so?

I'm not exactly sure what you are asking here but if you have unique data that  you have handpicked or edited then I suggest you leave the data with  you media and accept long scans building the database.  Now if rescans are taking forever and that is your complaint then it may have something to do with your network setup but I don't know enough to help you there....gotta know your limitations 

[GrimReaper 4740]

11 hours ago, Gilgamesh_48 said:

The library is actually stored on a network share and another server, much older, points at it. I have that server set up to store all data with the media so much, if not nearly all, data is already downloaded and available.

On my second server i set up a library pointing to that same location and configured it exactly the same as on the first server.

You shouldn't, you should have only one server writing artwork/NFOs and have the other one only read them, disable all providers/fetchers on second server. 

11 hours ago, Gilgamesh_48 said:

However the scan is taking every bit as long as the original scans on the first server so it seems there is no advantage to storing data alongside the media as it seems to save no time.

Because Emby is still reaching out to internet, as said above disable all providers (or kill internet connection altogether, as I do when rebuilding) before rebuild, if you already have all data stored with media, that will improve initial scan speeds immensely. 

11 hours ago, Gilgamesh_48 said:

1. Am I misunderstanding what storing the data alongside the media does?

I'd say not. 

11 hours ago, Gilgamesh_48 said:

2. What actually is the advantage of storing the data alongside the media?

Preservation of custom edits and faster library rebuild. 

11 hours ago, Gilgamesh_48 said:

3. What do I need to do to remove the local data and store the data the original way, should I choose to do so?

You'd need to delete all existing artwork/NFOs on OS level and disable saving them with media, refresh metadata>replace all. Note that you'll lose all custom edits. 

Edited October 3, 2023 by GrimReaper

[ebr 16184]

Hi.  Saving metadata with the media preserves your customizations, makes the media more portable and can help speed up new library builds but probing for media information, extracting thumbs etc. are very time-consuming operations so a complete rebuild can still take some time.

[rbjtech 5284]

On 03/10/2023 at 07:16, GrimReaper said:

You shouldn't, you should have only one server writing artwork/NFOs and have the other one only read them, disable all providers/fetchers on second server. 

Because Emby is still reaching out to internet, as said above disable all providers (or kill internet connection altogether, as I do when rebuilding) before rebuild, if you already have all data stored with media, that will improve initial scan speeds immensely. I

The reply from @GrimReaperis spot on.

As an interesting exercise - It's actually pretty easy to see how long a scan with the optimal settings takes by using a decent log viewer where you just filter by MediaProbeManager.

On a test brand new emby spinup - with a single movie library of 1760 Movies (some with extras) - pointing to my main emby library (local HDD storage) with only NFO Reader enabled and nothing else - it took my system from 09:13 to 09:31 to do the main ffprobe and build the library - that's 18 minutes ...

Start ...

End ...

This will get you a full populated library - and you use from this point.

However - The 'Scan' is still going - as 'People' are still being populated - creating NFO files locally.

I'll update this thread with those timings when that finishes and the scan media library task is 100% complete...

edit/update

ok - so it looks like it's the People Providers that are slowing it down.    Remembering that there are no Providers selected - so no 'People' metedata nor images are being downloaded here - it still took from 09:30 to 10:33 - for 54,448 People. (!)  (note, 2 lines per query in the log)

Start ...

End ..

As expected - when viewing a Movie, the Cast/Crew is empty.

So in summary -

Task	Duration
Item build using ffprobe	18 mins
People provider (no metadata collected as it's turned off)	62 mins
Total	81 mins or 1Hr 20 mins

 

This corrolates with the task time in the main Scheduler.

Conclusion

Scanning the existing items with NFO, Images, BIF(thumbs) data held locally is not the long task, the long task is building the 'People' part of the database and populating the local metadata cache, because people are not held with the media.

Remember that even with the People metadata NOT being fetched - it still took over an hour to scan through them.

Next Steps 

I'd like to repeat this when I have time, and turn ON the providers - to see what the duration differences are.   For the ffprobe, I would hope there is not much difference, as all the metadata is already with the media, there is no need to download anything.   But for the People metadata, then it's going to need to download all NEW metadata/images - as they do not exist.   This is likely going to be highly dependent on the speed of your internet connection and any throttling done by the providers.  Also would like to see how BIF's are processed.

Questions -

• Why is People metadata attempting to be retrieved if there are no providers selected ?   The scan would be 'complete' in 18 mins if you added this logic.
• Is there any benefit in keeping the 'people' data with the Media - rather than saved locally ?

@ebr @Luke

 

Edited October 4, 2023 by rbjtech

[rbjtech 5284]

So same test, this time with the standard metadata providers enabled - fresh install.

 

17:39 - 18:00

21 minutes this time, so 3 minutes slower than with no providers enabled.

So for each movie, the logging is saying it is querying the Providers, despite already having the data with the media, but I do note that nothing is actually being downloaded.    So maybe it's just doing a lookup on what is 'available' and then checking if it's already available locally - if it is, then it simply moves on.

Maybe @ebror @Luke could verify the logic here ?

The full scan has now finished - and surprisingly, it is 1hr 22 mins and 60 seconds (lol) - so 1hr 23 mins.

People is identical - 18:00 - 19:02 - so 62 mins again.   So I don't think it's downloading anything at this stage.

So in summary -

Task	Duration
Item build using ffprobe	22 mins
People provider (no metadata collected as it's turned off)	62 mins
Total	84 mins or 1Hr 24 mins

The extra 3 minutes needed for the comparison to the local data - I think.

Any thoughts ?

 

Edited October 4, 2023 by rbjtech

[Luke 42078]

This is something I noticed when doing testing of a new install recently. Supporting for reading person nfo's and saving them is on by default and can't be turned off. When it's one person here or there, it is mostly neglible, but on a new install it could be thousands of people and the cost of that is just too high.

Because of this, it's going to have to be turned off by default and require opting into it - even reading existing nfo files because it's  just a lot of checking for file existence, timestamps for changes, etc.

I'm pretty sure this all got turned on by default for the power audience here in this forum, but it's something that the vast majority will never care about.