Scan stuck at 90%

[Gilgamesh_48 1240]

I know I have seen a number of posts reporting a scan stuck at 90% but I searched for "90%" and for "90" and I had zero hits so i will ask a redundant question.

I have my scans of one library getting stuck (or slowing down enough to appear stuck) at 90%. When it does progress it does so very very very slowly. I deleted the library involved and rebooted the server and recreated the library under a different name (just in case) and it still sticks at 90%.

I can live with it if this is normal but my movie libraries all take less than 10 minutes to completely scan but my TV library is very very slow.

I will attach my latest log but I do not really get the why behind this slow scan.

 

embyserve 06-20-2024r.txt

[Luke 42078]

HI, I would just be patient and allow it to finish. For faster scans, consider disabling the video preview thumbnails feature.

[Gilgamesh_48 1240]

13 minutes ago, Luke said:

HI, I would just be patient and allow it to finish. For faster scans, consider disabling the video preview thumbnails feature.

But every single item in that library already has "video preview thumbnails: created and stored in the directories with the media. I do not really have any reason to disable them as they already exist. That is unless every scan creates the bif files even if they already exist. I do not really use bif file for TV so I think I will disable them anyway. 

Edit: I disabled the "video thumbnails" creation and the scan still sticks at 90%. What is your next guess?

Edited June 20, 2024 by Gilgamesh_48

[Lessaj 467]

It's copying the existing BIF files into the cache directory as it scans. Overall the scan is chugging along nicely but sometimes ffprobe takes 15-25 seconds to get a result on a file, such as between 15:30 and 15:40. Your disks might have been a little busy during those periods. There were a couple outliers that I removed data points for to better view the graph. You'll just have to wait for it to finish, as long as there is activity in the log it should be progressing even if the % is not reflective.

[Gilgamesh_48 1240]

17 minutes ago, Lessaj said:

It's copying the existing BIF files into the cache directory as it scans. Overall the scan is chugging along nicely but sometimes ffprobe takes 15-25 seconds to get a result on a file, such as between 15:30 and 15:40. Your disks might have been a little busy during those periods. There were a couple outliers that I removed data points for to better view the graph. You'll just have to wait for it to finish, as long as there is activity in the log it should be progressing even if the % is not reflective.

...

I have now deleted all bif files in my two TV directories. 
The scan speed does not seem improved, however it is no worse and, since I do not use bif files in TV libraries, I will just leave it where it is. In spite of what appears to be the "clog" being removed from all processing it still appears like it is a drain badly in need of a dose of Drano. 

Edit: And, yes, i have disabled all generation of bif files for TV libraries.

Edited June 20, 2024 by Gilgamesh_48

[Lessaj 467]

The BIF files really didn't have anything to do with it, they were being copied into the server cache directory which yes does add a little time but overall the slowest part is ffprobe which seems to slow down at certain points. Just be patient and let it finish.

[Gilgamesh_48 1240]

4 minutes ago, Lessaj said:

The BIF files really didn't have anything to do with it, they were being copied into the server cache directory which yes does add a little time but overall the slowest part is ffprobe which seems to slow down at certain points. Just be patient and let it finish.

Is there anything I can do to increase the speed. Taking 30 or more minutes to progress from 93% to 94% seems less than ideal. Also ffprobe should have very little to do as almost all my files have been scanned in the past and all metadata and all artwork is stored alongside the media files which means little to nothing must be done for those TV series. But it still takes a long long time.

I am waiting but the wait seems excessively long but if this is the best I can hope for then it will have to do.  

BTW: The scan goes from 0 to 90 in about three-four minutes then the scan seems to stop and only progress about one percent an hour or more. 
It is very frustrating to do everything I can to speed up scans and have the same old slow scans continue. 

One more time: my Movie libraries are about 5000 files in total and they all scan in less than 20 minutes total while my TV library only has about 900 shows and it takes hours to move from 90-100%. Something seems to be slowing down the TV scan and that slowdown seem to be excessive. There should be almost nothing to do during a scan since all metadata and all artwork is already stored with the media.

I may be dense but it just does not make sense. I will let the current scan finish. I hope it does not continue into the early morning hours.  

[Lessaj 467]

Importing media into the library is a single threaded operation, even if nfo files already exist I believe it still needs to probe the media files. 900 shows is a lot, I don't know how many episodes that may be but a conservative estimate of 50,000 episodes at a conservative estimate of 2 files processed per second works out to about 7 hours.

EDIT: I also think you're looking at the percentage as absolute but I think it's actually relative. It's not, for example, "I am scanning 100 files, every file is worth 1%", I think it's more "I have several operations to perform as part of this task, I have executed most of them, and now I have to scan the files" which is what that last 10% actually is. Obviously I'm not the dev so I don't know that, it's just an observation.

Edited June 20, 2024 by Lessaj

[Gilgamesh_48 1240]

28 minutes ago, Lessaj said:

Importing media into the library is a single threaded operation, even if nfo files already exist I believe it still needs to probe the media files. 900 shows is a lot, I don't know how many episodes that may be but a conservative estimate of 50,000 episodes at a conservative estimate of 2 files processed per second works out to about 7 hours.

EDIT: I also think you're looking at the percentage as absolute but I think it's actually relative. It's not, for example, "I am scanning 100 files, every file is worth 1%", I think it's more "I have several operations to perform as part of this task, I have executed most of them, and now I have to scan the files" which is what that last 10% actually is. Obviously I'm not the dev so I don't know that, it's just an observation.

If that is the case that having an existing NFO file does not speed up the scans then why is it offered at all? I just don't understand the need for redundant scanning.

But I did not write this beast so I may be failing to understand something that forces a rescan of everything every time a library is scanned. It seems that "storing metadata and images with the media is simply not an option that needs to be used as it does nothing to improve the speed of scans. But, since everything is already in the form I will just leave it as is. 

I am disappointed that Emby does not actually use the data stored with the media but that will in no way keep me form using Emby. I just don't like it when obvious possible improvements are not being used and options are given that don't really do anything to improve performance. 

Emby is the first program I have ever used that is continuously changing making it nearly impossible to use in the best way possible. 

Is there any place where all the options are discussed with the effects, pros, and cons listed? I have looked at a lot of the documentation and most of what I find is undocumented features or feature whose descriptions do not jive with the actual functionality.

I am not think the percentage is an actual indication of progress but that part is poorly written as it takes just a few minutes to go from 0 to 9- and hours and hours to complete that last 10%. If the percent is not accurate then its progress indicator needs to be rewritten to reflect what is actually happening. 

I've done everything I can to increase scan speeds but I am more than a little disappointed with the result. That is NOT disappointment in it taking a long time it is more disappointment that the interface gives false indications of progress so I have no idea when the process will finally finish.

Edited June 20, 2024 by Gilgamesh_48
incomplete first post

[Happy2Play 9780]

Initial scans are slow and caching existing info makes it even slower.  I really don't know what else can be done except better visibility of what is actually happening as the percentage indicator confuses to many people that see it jump to 90% and appear to hang as more tasks happen in that last 10% and they are just too impatient.

[Luke 42078]

Quote

If that is the case that having an existing NFO file does not speed up the scans then why is it offered at all? I just don't understand the need for redundant scanning.

It's a starting point and then internet metadata is used to supplement the nfo's after that. 

[bakes82 167]

Yeah plex gives you a bit more indication of what’s happening and you get more details on the alerts page, but you could also just goto your logs in emby to see what’s happening.  I’m not sure if there are “events” or hooks into the processing like meta refresh where you could then get a plugin to write those to the events/alerts on the emby dashboard the issue is it becomes busy.  Doesn’t emby support local nfo/images like plex?

[Luke 42078]

Quote

Doesn’t emby support local nfo/images like plex?

Yes, but that doesn't' disable internet metadata because many people have partial or incomplete nfo files...or nfo files that were produced by other software and don't have additional data provided by emby server.

[darkassassin07 652]

Just reading: something definitely seems off to me.

 

My tv shows library contains 33853 video files. Thumbnail extraction and chapter markers are set as seprate scheduled tasks. NFO reading+writing is enabled. Artwork is saved to the media folders as well as cached in the servers metadata folder, with download images in advance enabled.

Triggering a scan just now took less than 2min to complete and typically takes around 5-10min when actually importing new media (usually 1-5 episodes at a time, sometimes a full season/series).

 

 

I don't know what's causing your scans to be so slow, but they definitely seem abnormally slow to me.

Edited June 21, 2024 by darkassassin07

[Gilgamesh_48 1240]

8 minutes ago, darkassassin07 said:

I don't know what's causing your scans to be so slow, but they definitely seem abnormally slow to me.

Which was the point I was trying to make. It is not normal that my TV scans take so long but it seems that most people at Emby think I am doing something wrong but either i am not doing wrong or I am lying to this board just to shake things up.

Although I do not really expect "belief" to change I can assure everyone that it is Emby that is screwing up not me. 

But Emby is consistent for me. I also have a second server and it is looking at the same set of files (Server #2 does not write anything at all to the directories) and it is not much faster but it is a little faster and server #2 connects through the network while #1 has everything local. 

I can live with the ploddingly slow scans but I believe they "should" be a lot faster. 

I will also address a few other comments that people have made.
I will never ever again use Plex because their "people" are simply horrible to deal with.
Why is it that the nfos are not used effectively. If they are complete Emby does not need to scan for the same info again but it does? That means that nfo files are NOT being used because, if they were, then there would be no reason to rescan from the internet. It just seem foolish to write nfos and not use them. 
Is it possible that my system has something wrong that slows both of my server way down?
 

[darkassassin07 652]

Nobody's calling you a liar @Gilgamesh_48. We're just not sure what's different between your software environment and one that's performing better. Figuring that out isn't always straightforward.

Don't assume malice where there is none.

 

 

A few thoughts:

Things are slow for both servers; so think about what the two share in common (configuration, resources, etc). Your issue is likely something the two servers share.

 

Storage speed; not just raw speed, but latancy? Perhaps your drive pool is taking lots of time in between data fetches? Is there other software accessing those drives while emby tries to scan? 

 

Network; when emby does have to grab metadata, is that being hampered by network congestion or some other factor making each fetch slow?

 

Cache; is it being cleared for some reason? External cleanup of some sort, so emby has to repopulate its cache during scans? 

 

 

 

You've got two servers to play with. Experiment with one and see if you can find a change that speeds things up for you.

[Happy2Play 9780]

5 minutes ago, darkassassin07 said:

Cache; is it being cleared for some reason? External cleanup of some sort, so emby has to repopulate its cache during scans? 

No a per library cache options for with media and server.  So duplication.

So initial process will be time consuming for every item.  No different then a brand new install initial scan.

[Gilgamesh_48 1240]

So nobody knows even what to check and how to do it. I "guess" my inclusion of my latest log did not give any clues at all and was just a waste of time and space for me.

If i seem bitter I am a bit. I kind of expect that the inclusion of log files would give someone a clue but they did not. 

I am not going to fight about this any more as it seems very very unproductive to do so. My servers are working, albeit slowly, and i see no important reason to further try to find a problem that Emby seem to think is my fault.

[Happy2Play 9780]

Well everything has been said you changed options and addition processes are added to library scan it will be slow until it completes. 

All the log really tells us is Emby is working hard to copy cache images/bifs for every items directly to their /metadata/library/xx/xxxxxxxxxxxxxxxxxx folders.  So a SLOW time consuming process.

 

[Gilgamesh_48 1240]

15 minutes ago, Happy2Play said:

Well everything has been said you changed options and addition processes are added to library scan it will be slow until it completes. 

All the log really tells us is Emby is working hard to copy cache images/bifs for every items directly to their /metadata/library/xx/xxxxxxxxxxxxxxxxxx folders.  So a SLOW time consuming process.

 

After reading about the bif files causing the slowdown I deleted all TV bif files and restarted the server but that made zero difference. I then was told hat the bif files were not slowing my scans down now I am being told that the bif files, which no longer exist and are disabled, are part of the problem. Why should I trust that anything I am told is true? 

I no longer have or use bif files for my TV library but the scans are still taking the same long time so, unless they can effect scans when they do not exist and are not created the bif file cannot be the problem.

There actually may not be a problem but if there is not then why do scans take so extremely long for me while others with similar size libraries scan very fast. 

I believe that when this new scan finally completes the next scan may be faster but the non-answers I have been presented seem to be little more than bad guesses. 

I don't mind slow, much, but i like to KNOW why.

[Happy2Play 9780]

All I can say is what the original log shows.

example

2024-06-20 15:40:11.702 Info App: Caching file from P:\Active TV\Criminal Minds\Season 01\Criminal Minds - s01e14 - Riding the Lightning-320-10.bif to C:\Users\jt\AppData\Roaming\Emby-Server\programdata\metadata\library\8d\8dc817a303c5386b5bb04057493e8db7\thumbnailcache\Criminal Minds - s01e14 - Riding the Lightning-320-10.bif
2024-06-20 15:40:11.805 Info ProviderManager: Caching file from P:\Active TV\Criminal Minds\Season 01\Criminal Minds - s01e14 - Riding the Lightning-thumb.jpg to C:\Users\jt\AppData\Roaming\Emby-Server\programdata\metadata\library\8d\8dc817a303c5386b5bb04057493e8db7\imagecache\Criminal Minds - s01e14 - Riding the Lightning-thumb.jpg

Quote

Keep a cached copy of images in the server's metadata folder

If your media and images are stored on another machine, keeping a cached copy of images in the server metadata folder will reduce network access and may improve performance. The cost of creating this cache is that library scans will be slower and the server metadata folder will consume significantly more disk space.

Until this process completes scan will be slower.  As your posted log only got to C for TV Shows copying almost 4500 images/bif files from library to /metadata to that point.

Edited June 21, 2024 by Happy2Play

[Gilgamesh_48 1240]

20 minutes ago, Happy2Play said:

Until this process completes scan will be slower.  As your posted log only got to C for TV Shows copying almost 4500 images/bif files from library to /metadata to that point.

But most if not all images were cached weeks ago and the server has been running with the cached images quite well. Are you saying those images have to be recached every time there is a scan? That seems quite counter productive.

But I am quite tired of discussing this as it is clear that none of my questions actually get answered in such a way so I know what to do to fix anything so I guess I am quitting fighting about  It is quite unproductive and time wasting to do so so i will leave this here and simply accept that nobody really understands Emby's scans. I find myself getting angry for no real reason except frustration. That is not good for my health and I am sure that others also are getting frustrated either for me or with me. 

I think I may try and find another, simpler media manager for video and pictures as that is all I use Emby for. I do NOT need fancy I just want simple and as easy as possible. 

I do not think there are any alternatives that would work for me but I feel that Emby has made their program much too complex so nobody, even the developers, know how it works any more.

Edited June 21, 2024 by Gilgamesh_48

[Luke 42078]

Quote

But most if not all images were cached weeks ago and the server has been running with the cached images quite well. Are you saying those images have to be recached every time there is a scan?

No they don't, but you said you deleted the library and added it back, so many things are happening for the first time all over again. 

[rbjtech 5284]

8 hours ago, Gilgamesh_48 said:

But most if not all images were cached weeks ago and the server has been running with the cached images quite well. Are you saying those images have to be recached every time there is a scan? That seems quite counter productive.

But I am quite tired of discussing this as it is clear that none of my questions actually get answered in such a way so I know what to do to fix anything so I guess I am quitting fighting about  It is quite unproductive and time wasting to do so so i will leave this here and simply accept that nobody really understands Emby's scans. I find myself getting angry for no real reason except frustration. That is not good for my health and I am sure that others also are getting frustrated either for me or with me. 

I think I may try and find another, simpler media manager for video and pictures as that is all I use Emby for. I do NOT need fancy I just want simple and as easy as possible. 

I do not think there are any alternatives that would work for me but I feel that Emby has made their program much too complex so nobody, even the developers, know how it works any more.

Going by the log, as that is all the information we have - using a decent log tool to search for the tasks - it has

1593 ffprobe's - that's files being read, analysed and stored in the emby db.

3386 files being cached - files being copied from external USB drives (if I recall from previous threads) to the operating system drive.

That is simply going to take time if you have 'reset' the libraries.

The comment about nobody understanding how Emby works/scans is a little disrespectiful imo - people are spending their personal time trying to help you.

[Sammy 790]

16 hours ago, Lessaj said:

It's copying the existing BIF files into the cache directory as it scans. Overall the scan is chugging along nicely but sometimes ffprobe takes 15-25 seconds to get a result on a file, such as between 15:30 and 15:40. Your disks might have been a little busy during those periods. There were a couple outliers that I removed data points for to better view the graph. You'll just have to wait for it to finish, as long as there is activity in the log it should be progressing even if the % is not reflective.

What did you use to create this graph?

I have 17TB of Movies and Shows and my scans are usually pretty quick and complete without artificially hanging at 90%.