Please, for the love of all that is holy, PRIORITIZE NEW MEDIA!

[twhiting9275 10]

I cannot believe that I have to say this, but, apparently I do, because a MEDIA application in 2023 has no proper priorities, at all. This is as ridiculous as it gets, honestly

The problem:
I have a rather large media library. Like ginormous. 75k episodes, 3k(ish) movies, etc. It takes a while for Emby to go through an "scan all files", do whatever it does, blah, blah, blah, blah. Like if I restart the server, it takes better than a day to go through and do all of this.

The issue comes when I add new stuff (which I do on a daily basis). I cannot ACCESS this until Emby has completely finished it's scan and whatever processing it does when told to 'scan library files'.
Do that processing differently. PRIORITIZE new media, instead of lumping everything (subtitles, images, nfo, meta, etc, etc, etc) . Scanning files should, quite literally be just to scan for file changes and new files. 

 

[Luke 37269]

Hi, that’s what the library scan does. I would also check that you’ve enabled the realtime monitor for each of your libraries.

[twhiting9275 10]

1 minute ago, Luke said:

Hi, that’s what the library scan does. I would also check that you’ve enabled the realtime monitor for each of your libraries.

Except it’s not.

 

library scan does FAR more than just scan for new files.  It triggers checks for subtitles, image creation, etc….  It , quite literally slows down the processing of anything

 

library scan should do JUST that.  Scan the library, look for new files and make them available.  Nothing else .  
 

Yes, real time monitor is enabled .  It’s useless though, if a scan is taking place , which , again, does far more than it should 

[Luke 37269]

We can add an option for subtitle similar to thumbnails so that you can configure whether or not it runs as part of the scan.

[twhiting9275 10]

Still not the appropriate answer here.

I'm not new to media storage, or display (obviously). I've been doing this for quite a few years.

if I put something into my library, within 5 minutes, it should be picked up and ready to play. EVERYTHING else should be done in the background. That's how it's been all along, and that's what it should be.

With Emby, it's not... For example, I restarted a library scan when this thread was started. Right now? The scan is only 93% complete. If I added more media to the library during this, then it would be ignored, until I , once again, restarted a scan, because your current process prioritizes things incorrectly

Get the necessary data in, then spawn off other processes to handle everything else. Keep the process in repetitive motion 

The way you have things , as they are right now, it locks everything up while the scan is in place, and yes, with larger libraries that is going to take a while. That process needs to be simplified

[Luke 37269]

Quote

 If I added more media to the library during this, then it would be ignored

This has been resolved in the upcoming 4.8 release. 

[twhiting9275 10]

24 minutes ago, Luke said:

This has been resolved in the upcoming 4.8 release. 

I'm curious as to how? Was it a bug, or just a better workflow?

[Riggs 300]

2 hours ago, twhiting9275 said:

I'm curious as to how? Was it a bug, or just a better workflow?

Yes it was a bug, and it is already fixed. The issue was because to the new huge changes in the server.

That feature was in Emby a long time ago.

Please mark this post solved.

Thanks

[Luke 37269]

4 hours ago, twhiting9275 said:

I'm curious as to how? Was it a bug, or just a better workflow?

Previously the real-time monitor was deactivated during the library scan and that is no longer the case.

[twhiting9275 10]

4 hours ago, Riggs said:

Please mark this post solved.

Once I've verified the solution is working for me, I will, absolutely do so.
Cannot do that until the release is made, however.

[Spaceboy 2507]

7 hours ago, Luke said:

Previously the real-time monitor was deactivated during the library scan and that is no longer the case.

Interesting. So the issue of tv recordings that start during a library scan not showing up until the scan is complete has also been resolved?

[Q-Droid 670]

@Luke Are release notes available describing how media scanning has changed in 4.8? With all the forum complaints about scanning big libraries and "getting stuck" doing so I've wondered if the process could benefit from multiple passes and deferred actions (multiple workers and queues). This could speed up adding media and make it available immediately in the UI then defer detailed metadata and other collections and processing. I don't know if the updates will already do something along these lines.

[Luke 37269]

On 4/29/2023 at 9:15 AM, Q-Droid said:

@Luke Are release notes available describing how media scanning has changed in 4.8? With all the forum complaints about scanning big libraries and "getting stuck" doing so I've wondered if the process could benefit from multiple passes and deferred actions (multiple workers and queues). This could speed up adding media and make it available immediately in the UI then defer detailed metadata and other collections and processing. I don't know if the updates will already do something along these lines.

There's a number of changes, although the release notes just generalize and say library scan improvements. But to be more specific:

• fewer or faster database queries
• removal of unnecessary operations
• adjustment of database indexes to make updates and deletes quicker
• making the realtime monitor a lot lighter on it's feet
• and others that i'm probably forgetting right now

[rbjtech 4337]

1 hour ago, Luke said:

There's a number of changes, although the release notes just generalize and say library scan improvements. But to be more specific:

• fewer or faster database queries
• removal of unnecessary operations
• adjustment of database indexes to make updates and deletes quicker
• making the realtime monitor a lot lighter on it's feet
• and others that i'm probably forgetting right now

Does it defer the big resource tasks such as Thumbnail creation, Intro Detection etc - or does it still try and do those sequentially ?

[Luke 37269]

5 minutes ago, rbjtech said:

Does it defer the big resource tasks such as Thumbnail creation, Intro Detection etc - or does it still try and do those sequentially ?

That's based on your library options.

[rbjtech 4337]

Just now, Luke said:

That's based on your library options.

Sure, but given most people are going to select all these items (totally ignoring the footnote about it impacting scan times ..) .. is there not a way to defer those items until after the initial 'base' scan as if they had not selected those optional items ?

[Luke 37269]

3 minutes ago, rbjtech said:

Sure, but given most people are going to select all these items (totally ignoring the footnote about it impacting scan times ..) .. is there not a way to defer those items until after the initial 'base' scan as if they had not selected those optional items ?

The scan always does a quick scan first followed by a second pass for each folder level, so that's sort of already there. Meaning it won't stall other new items in the same folder from showing up. it just might stall their retrieval of internet metadata.