processing time for large library rescan?

[guill69f 17]

Hi,

I have a large movie library, 8000+ files.

I have renamed them all outside Emby, using Tiny Media Manager, and i was counting on Emby identifying the changes and doing a proper remapping. I have forced a library scan, but now it has been several hours that the status bar has been stuck on 5.9%.

Should i think there is an issue and that Emby is stuck, or should i just continue to wait?

Or is there a better approach (if so, which?) like wipe all movie data and library in Emby, and then rebuilt libraries from scratch?

[Killface69 96]

Emby has to rescan/rescrape the media files entirely, event thought only the file name changed.

It's normal for Emby to seem stuck after a while, it starts scanning the folders and then slows down for processing the metadata. It might be faster without external scrapers like TMDB and relying on the NFO files only, but I haven't noticed a speed increase.

You can "watch" the progress by library if you go the currently processed media folder and filter by "Missing TMDB id". This should show you the unprocessed files.

Processing 8,000+ files could actually take a day or two.

Edited January 4, 2025 by Killface69

[guill69f 17]

Thanks, then I will wait, i understand it will take some time, just got a bit surprised that in the dashboard the task library scan has been stuck at 5.9% for at least 2 hours, but fine, i'll wait

In the Library view i have another indication, but doesn't seem to move either:


Anyway, i'll rest my case for now and will continue to do the monitoring. I'll post here again when scan completed to confirm all went well, or if it is still stuck by mid-week

[Neminem 1518]

You can also follow progress in logs.

Then you can refresh and scroll down and seen where it is in the process.

[guill69f 17]

Fantastic, i did not know that. I have checked a couple of logs now and i see at the bottom of the logs that the most recent ones always have new movies that the other logs don't, so i guess things are progressing as they should. Thanks!

[guill69f 17]

Hi, coming back to this one, the majority of the movies have been rescanned i guess because they are now displayed with posters and other metadata, but a few of them still remain to be analyzed by Emby it seems as they are not displayed with this information. I see also that movies recently added to the library folder are not (yet?) showing up either.
When i check the logs, what i see for now is a ling list of "Error SubtitleManager". Will eventually Emby come over those and continue the analysis of the remaining movie files? thanks

[Killface69 96]

Id' rescan the libraries. With so many items, there are sometimes timeouts from TMDB or other issues.
If you're using video thumbnails, you could set the genereation to 'scheduled task' instead of 'during library scan' to speed things up.
If the library is also set to real time monitoring, there might be missing entries, as manual+real time can sometimes lead to not recognized entries.

Not using the SubtitleManager, could be missing entries on OpenSubtitles.org. There are options to rescrape subtitles up to 180 days AFAIK.

[guill69f 17]

Thanks, will check later today.

Can you clarify what you mean with rescanning the library please? On my install it looks like scanning is still active (progress bar at 6% but i am not sure it is really reliable, as it took 2 days to move from 5.9 to 6.00, while i could monitor in the logs more and more movies where analyzed), and on the library itself it shows the scan is 93% completed (here again, doubts on the reliability of the info, took 2 days to increase from 92 to 93).

Should i stop the emby app to kill all processes, and then manually retrigger the task "scan library", after I have done the other suggested verifications? Thx

Edited January 6, 2025 by guill69f

[guill69f 17]

Scan Status on library just updated to 94, so it's not frozen. Will check the logs again to see what they say

[Killface69 96]

It depends on the amount of folders you have in the library. 

For each main folder in the library, Emby will repeat the go from 0-100%, mainly stopping at 90% and going slow towards the 100% (the metadata part I assume).

If you're logged in locally, the progress should be correct, if your server is not busy with other things. When signed in from remote, the progress can be unreliable.

You can check the progress in the library view, see my comment with the missing TMDB ids.

[Carlo 4560]

Think of the percentage shown as the number of overall steps that have been done vs overall number needed to complete.
Do not think of the percentage shown in a time format.

For example, 50 steps could be completed in 5 to 10 minutes and account for 90% of the steps needed to complete the scanning job. 
The last 5 steps (10%) could take 1 or 2 days to complete. Some steps take longer than others and might be best to turn off or differed until after the initial scanning is complete.

I normally turn off these two options on all libraries until they have been initially scanned into the library.
Video Preview Thumbnails & Subtitle Downloaders

Checking the log file IMHO is the best method to see what Emby Server is doing while scanning. It's also a great way to quickly scan through the log to see if any errors are showing.

Carlo

[guill69f 17]

Thanks, make sense. 95% now

[guill69f 17]

daily reporting : 96% as of now, at this pace the scan will be done in 4 days I can wait no problem, the logs are indicating anyway that progresses are being made, one movie at a time, i'll just think it twice next time i'll rename the files outside Emby.

Just to comment also on first answer above from Killface69 (if i got the last line correct): when i go to the movie library and i apply the filter missing metadata and select MovieDb ID, i get a count of 35 movies, which does not include the movies I see without poster nor metadata when i have no filters set. Was just thinking to report this in case this is unexpected behavior

[Killface69 96]

1 hour ago, guill69f said:

Just to comment also on first answer above from Killface69 (if i got the last line correct): when i go to the movie library and i apply the filter missing metadata and select MovieDb ID, i get a count of 35 movies, which does not include the movies I see without poster nor metadata when i have no filters set. Was just thinking to report this in case this is unexpected behavior

I guess it's no issue to report. It stemmed from my experience, mainly with new files. 

[Luke 42077]

Hi, initial scan time all boils down to what features are enabled. If you can provide an Emby server log then we can probably offer suggestions to speed up your scan.

[guill69f 17]

Hi all, thanks again for patience and answers. Problems are now solved.

In the end i deactivated tasks related to thumbnails and subtitles, and scanning completed during the night.

But the movies i kept my eyes on still were missing posters and other tmdb info, and when i inspected those entries more attentively i saw they all had the option "split versions apart", but I could positively tell I only have one version of each of these movies.

I did then separate metadata refresh for these movies and that did the trick, posters and all other movie info is now available.

I will now reactivate the thumbnails creation and the sub downloads, and everything will be as before.

Last (but not a big deal for me), i still have 9 movies displayed when i activate the filter "missing metadata" (movies id), but when these movies are inspected they have posters, summary etc. Just fyi, nothing i really need to solve.

Thanks all one more time

[Luke 42077]

Quote

In the end i deactivated tasks related to thumbnails and subtitles

I wouldn't deactivate the tasks but rather just disable the features in library options. 

What might happen later is you might forget about this, then decide to enable the feature, only to get the impression that it isn't working because the scheduled task is never running.

[Luke 42077]

Quote

Last (but not a big deal for me), i still have 9 movies displayed when i activate the filter "missing metadata" (movies id), but when these movies are inspected they have posters, summary etc.

Exactly what filter? Let's assume it's the filter for missing moviedb id. You said this:

Quote

but when these movies are inspected they have posters, summary etc.

But that is not the criteria for the filter. The criteria is whether or not it has a movie db id assigned.

[Killface69 96]

1 minute ago, Luke said:

Exactly what filter? Let's assume it's the filter for missing moviedb id. You said this:

But that is not the criteria for the filter. The criteria is whether or not it has a movie db id assigned.

That came from my previous answer to the topic. I suggested this filter as as an indicator during library scans. 
You're right, some films don't have a tmdb id, as the movie doesn't exist on tmdb or it might be labelled as TV. 

[guill69f 17]

2 hours ago, Luke said:

I wouldn't deactivate the tasks but rather just disable the features in library options. 

What might happen later is you might forget about this, then decide to enable the feature, only to get the impression that it isn't working because the scheduled task is never running.

Yes, you are right, this is what i meant actually

[guill69f 17]

2 hours ago, Luke said:

Exactly what filter? Let's assume it's the filter for missing moviedb id. You said this:

But that is not the criteria for the filter. The criteria is whether or not it has a movie db id assigned.

OK, my misunderstanding then