Library scan crawling / freezing / getting stuck from 68% onwards

[iamahuman 0]

Hi,

 

I've just come to Emby yesterday, after experiencing subtitle sync issues with the Plex app on my Samsung TV. How nice to see that Emby has just released an app for Samsung TVs in the UK!

 

Unfortunately I'm having trouble with my library scan completing. Details are as followed:

 

• Windows 10 x64 with media on 2 x USB harddrives connected via USB as drives M: and N:
• Emby Portable
• Movie folders contain metadata (jpg/nfo) in "Kodi" format.
• Emby is setup to 'Download artwork and metadata from the internet', and 'Download images in advance'. Nothing else is selected (e.g. chapter image extraction is disabled).

The initial import of movies was quite quick for the first 1200 (or thereabouts) - I had one hard drive connected at this point as I just wanted to import the first drive. I suspect I had to restart Emby once during that as it appeared to have stopped scanning. I don't know with 100% certainty whether it completed scanning all films on that drive, but I then added the second drive to the same library ("Movies").

 

Since hitting around 68% it's come to a crawl and appears to stop completely very often (I consider "stopping" /movies.html showing no change in total count for around 10 mins or more).

 

I have to keep restarting Emby and initiating a new scan, whereby it adds a small number of movies being stopping again.

 

When clicking "scan all libraries" on /library.html the progress bar moves up very quickly to what I would guess is around 68% (it doesn't show a value). Dashboard.html shows the same progress bar under "Running Tasks" with the percentage. It doesn't seem to update itself - I have to reload the page or visit other links before returning to it).

 

As a test, I just restarted Emby with 1340 movies in the collection. It added 8 movies and stopped. I've attached the log file.

 

Any ideas? Thanks

 

 

 

 

server-63643517072.txt

Edited October 13, 2017 by iamahuman

[Luke 42078]

Hi, try unchecking 'Download images in advance'. This will cause the library scan to take considerably longer.

[iamahuman 0]

Hi Luke,

 

Thanks for taking the time to reply. I'm aware that's what the setting says but it doesn't address my issue really (perhaps my fault for not being succinct enough). The library scan doesn't "take longer" (which that setting suggests) - it stops completely. I've been out of the house for over an hour now and nothing has changed.

 

To my knowledge, this setting queries the TMDB API for data (text,image,other) and pulls it down when it can. In the event it can't get data, it should say something to that in the logs, pause and retry right? I've some experience working with APIs and don't understand why this setting would cause Emby to stop scanning, period. I'd like images to be downloaded, just like Kodi, just like Plex.

 

Thanks

[Luke 42078]

I realize you believe that it has stopped, but I don't think that it has. I think if you just be patient it will finish. Thanks.

[dee1 3]

I have a question, I want "scan for new and updated files" and I'm running the same OS win10 64 with two wd mycloudss ex with kodi format (Save artwork and metadata into media folders) only item checked.

 

How do I make sure that the scans stay as "scan for new and updated files"??

 

How do I go forward lock setting in? that in?  Is there a script I can set?

 

I also setup the RokuTV (elements from amazon) and loaded the Roku plugin, just to test, see the difference.

So lets say I buy the subscription for a month and then change my mind and want to go lifetime can I do that then??

[Luke 42078]

The regular automated library scan always has this behavior:

scan for new and updated files

So there's nothing you need to do. To convert from monthly to lifetime you would simply cancel your monthly recurring and then purchase lifetime.

 

Have these answered your questions?

[Sammy 790]

My last library scan after restoring my server settings stalled at 73.x% for a long time but I let it run, with it slowly creeping along. Then I stopped watching water boil, put the lid on and came back a while later and it was at 98% then wrapped up pretty quickly. I have a pretty large library but that scan took about 4 hours. I'm thinking I'll have to go through this again pretty soon when. .net core comes about but it will be worth it for the increased speed.

[Happy2Play 9780]

Yes the percentage can be deceiving on a initial library build, looking like it stalled in the 70+% range but you just have to let it run.  Only way to know if it has actually stalled is watching the log.

[Sammy 790]

This was an "issue" when rolling into the Emby Version first using .net 4.6.2 or something like that so I expect it again when going to .net core. Not really an issue actually.

[dee1 3]

The regular automated library scan always has this behavior:

scan for new and updated files

So there's nothing you need to do. To convert from monthly to lifetime you would simply cancel your monthly recurring and then purchase lifetime.

 

Have these answered your questions?

Just making sure, I'm at the wall.. I like what I'm seeing in HTML getting the notices on the devices.

[SDHoover 1]

I'm having this problem again and it's getting really aggravating having these complicated issues while trying to do something that should be simply done and only recently started being a issue with Emby. Due to an issue with Windows update I had to wipe my os and reinstall windows. but in attempts to try and avoid these issues I just installed a new drive, installed the latest version of windows Version 1809 (OS Build 17763.107) and attempted to migrate my Emby (Version 3.5.3.0) files over to the new drive hoping to avoid these issues that Emby has trying to upload my Library. Migrating the files over seemed to work at first however, I realized some of my TV shows were not listed. I selected the Scan Library and they still would not show up on the list. This was only noticed with my TV shows, My movies appear to all be listed. So I removed the TV Shows Library and tried to set up my TV Shows Library from scratch. I have 108 TV Shows in my Library spread across three drives. I scanned my files pretty quick until it got to 68% Then 6 hours later it managed to get to 92.1% then another 4 hours it managed to get up to 93.1%. So being aggravated wasting so much time troubleshooting something that should be automatically done as it previously has back when I first started using Emby when it actually worked. So at this point I completely uninstalled the program, deleted all files/folders and items related to Emby then re installed it. I followed the suggestions including disabling the Downloaded Images in Advance. Instead of attempting to sync it up with my entire Library at once I attempted to just add one drive at a time, the one holding the smallest amount of files which was like 11 TV Shows thinking It would make a difference only syncing it up with a few shows at a time instead of my entire collection but I was wrong. Apparently regardless of the number of files or file types It still made it Just as far. I quickly runs up to 68% then slows down to a crawl. After a day It made it to 93.1% with 11 TV Shows. Frankly I'm tired of this. I use to support Emby not to get any support in return when Issues occurred. Last time I had this same issue I followed the suggestions I received or seen on here but after having this issue every time I set up the server and after this being a issue for me and others going on two years now and apparently no one is attempting to do anything to resolve this. when I initially intended on seeking help after explaining and seeing my self all the troubleshooting/time I've put into just attempting to set up my Library plus when I have previously managed to get my entire library uploaded I'd have to go through and correct where it would take one show and list each season as if it was separate tv shows,  I'd have to edit the pictures to the correct show. It's a waste of my time having to do something manually when that's the entire reason I started using this software was so I wouldn't have to spend so much time managing my media it's kinda pointless. I'll just stick to using PLEX, It doesn't give me any problems. Good luck guys. 

[Happy2Play 9780]

I'm having this problem again and it's getting really aggravating having these complicated issues while trying to do something that should be simply done and only recently started being a issue with Emby. Due to an issue with Windows update I had to wipe my os and reinstall windows. but in attempts to try and avoid these issues I just installed a new drive, installed the latest version of windows Version 1809 (OS Build 17763.107) and attempted to migrate my Emby (Version 3.5.3.0) files over to the new drive hoping to avoid these issues that Emby has trying to upload my Library. Migrating the files over seemed to work at first however, I realized some of my TV shows were not listed. I selected the Scan Library and they still would not show up on the list. This was only noticed with my TV shows, My movies appear to all be listed. So I removed the TV Shows Library and tried to set up my TV Shows Library from scratch. I have 108 TV Shows in my Library spread across three drives. I scanned my files pretty quick until it got to 68% Then 6 hours later it managed to get to 92.1% then another 4 hours it managed to get up to 93.1%. So being aggravated wasting so much time troubleshooting something that should be automatically done as it previously has back when I first started using Emby when it actually worked. So at this point I completely uninstalled the program, deleted all files/folders and items related to Emby then re installed it. I followed the suggestions including disabling the Downloaded Images in Advance. Instead of attempting to sync it up with my entire Library at once I attempted to just add one drive at a time, the one holding the smallest amount of files which was like 11 TV Shows thinking It would make a difference only syncing it up with a few shows at a time instead of my entire collection but I was wrong. Apparently regardless of the number of files or file types It still made it Just as far. I quickly runs up to 68% then slows down to a crawl. After a day It made it to 93.1% with 11 TV Shows. Frankly I'm tired of this. I use to support Emby not to get any support in return when Issues occurred. Last time I had this same issue I followed the suggestions I received or seen on here but after having this issue every time I set up the server and after this being a issue for me and others going on two years now and apparently no one is attempting to do anything to resolve this. when I initially intended on seeking help after explaining and seeing my self all the troubleshooting/time I've put into just attempting to set up my Library plus when I have previously managed to get my entire library uploaded I'd have to go through and correct where it would take one show and list each season as if it was separate tv shows,  I'd have to edit the pictures to the correct show. It's a waste of my time having to do something manually when that's the entire reason I started using this software was so I wouldn't have to spend so much time managing my media it's kinda pointless. I'll just stick to using PLEX, It doesn't give me any problems. Good luck guys.

 

Without more information on what options you selected when building the library and a server log, it hard to say. Download images in advance and Extract chapter images during the library scan are the biggest culprits.

[SDHoover 1]

I'm not sure what information would be relevant since it doesn't matter what changes I make to Emby or if I leave everything set how it comes automatically. it's a fresh install of Windows I ended up doing a fresh install of Emby doing nothing more than following the step by step settings upon installation and setting up my library. And I actually made an error earlier, looking back on my earlier posts on here this has been a constant issue for me going on three years now of encountering issues trying to set up my Library every time I've gone through the setup process or try to add additional folders/drives to my Library. Which I build a new computer or Upgrade a component where I end up re installing everything every 3 - 6 months. So I dont see how this issue with every version of Emby that's been released in the last three years consistently happening on different computers without making any changes to the settings in Emby is a issue outside of Emby itself. 

[SDHoover 1]

[SDHoover 1]

that's screen shots of all the settings under Library and from when I first posted on this issue tonight which was at 6:47pm and it is now 9:00pm and attempting to add only 11 tv shows throughout the day that have previously been added to Emby prior to me installing a new hard drive for my Operating system, and installing operating system so those files should already have the metadata, pictures, backdrops whatever Emby adds to the files are already stored in with those files. Emby should not have to do anything other than access those drives/folders and subfolders and add to library. no changes have been made to my media files since I replaced my C:\ drive. so I should not have to sit here, waiting all day while between 6:47pm - 9:00pm Emby goes from 93.3% - 93.4% on adding 11 of my TV shows to my Library. I have lots more I'd like to move on to adding. My files are organized as Drive:\ TV Folder\ Tv Show\ Season. This has been a ongoing issue,  It has been over a week process to get Emby to complete adding all my files for the last three years. and I've tried changing every relative setting through the past three years of experiencing this issue. I've followed every recommendation that's been suggested to me on here or that I've seen given to someone else having the same issue and nothing makes a difference. I'm not the only one encountering this issue, Why are we still experiencing this after three years. I have to keep Plex on standby since Emby has been so problematic and Plex has my entire library scanned/synced in less than a hour. I just prefer Emby's layout and options on my devices I stream too. but all that's irrelevant when I'm not even able get Emby to acquire my library. I have my own business, plus I work a second job full time, I have a disabled mother I help take care of, three kids, 33 acres, and a bunch horses and dogs I have to take care of. I do not have time to waste on this. or to keep having to check back hoping for a response/ solution when all I get is people just trying to suggest making independent changes until something eventually fixes the issue. I've already tried that many times. It would be nice to have some access to someone who really knows what they're talking about. because it appears to me after making every possible change on my end, and trying a variety of computers/servers yet still continue to have this issue the only common denominator is Emby itself. and after all the troubleshooting I've done My professional recommendation would be that this issue is due to a change made in the coding approximately three years ago that has continued over the the later versions. Further evidence of this is about a year ago troubleshooting this issue I tried installing a older version of Emby I still had at that time and It gave me no issues with adding my library. however it did give me issues with my devices running later versions trying to access a outdated server so I was forced to run the latest server version. 

[SDHoover 1]

here it is 9:45 and it's only gone up to 93.7% of adding only 11 TV shows to my library that have previously been set up on Emby and contains all metadata, photo's, backdrops, etc that emby acquires when adding to library previously. 

[Luke 42078]

We'd have to look at the server log to provide more detail, but the first scan can take a little while. I would just be patient and allow it to finish. thanks !

[Luke 42078]

I've hidden your post due to profanity. Just so you're aware, the next release of Emby Server will have dramatically improved performance (although not necessarily the first scan). Thanks !

[SDHoover 1]

No thanks needed. I'm here seeking alternative methods of support seeing if anyone has found the issue, or a way to resolve it and all I find on reddit and many other forums/sites is lots of people with the same issue. so apparently asking for information on my computer is not going to resolve the issue when you know it's a problem with the program itself and insisting on wasting my time. Only reason I've spent so much time on this is because my disabled mother prefers the emby layout on her Roku. I prefer the Emby look but running the server aspect is nothing but a headache. plus when I stream my media from my server I'm not able to stream to my xbox one without paying the subscription for the premium. and I'm cheap but I dont mind paying for a good product that works. but after encountering this issue so frequently when their is no excuse for it. It's not worth my money after putting so much time into it myself trying to get it to work correctly. All the time I've put into troubleshooting this over the last three plus years I could have programmed my own media server. Their is no excuse for having to take days to set up a Library. That is pitiful and poor programming. That should be the priority fix prior to releasing any further updates. I might could understand if it's someones first time installing/using Emby and they had a massive amount of media. which I have a pretty large collection however it's not my first time using Emby. all the required metadata and pictures are already set up by MCM and from previous installations of Emby. Their is no excuse. 

[SDHoover 1]

yea, see I use profanity and someone was here instantly to block and respond to it. yet when it comes to trying to resolve the issue you know cant be resolved since it's a issue with the program itself you dont bother to respond. you state that a new update is going to be released soon with performance improvements but still may take time to acquire the library the first time. You should be resolving that issue first. new users try out this program which looks good, until they download it and realize it's going to take days before they are able to utilize the new improvements they will be less likely to continue using it and if they do probably not willing to purchase the premium package. Someone new trying this and encounter that and get the run around like the rest of us has who encountered this issue they'll loose interest before Emby allows them access to their media rendering any other improvements useless. I've suggested this Program to many of my clients over the years until recently and they contacted me complaining about having so much trouble setting it up and having access to their Library with no reasonable support It made me look bad and I had no choice but to suggest other software. and will continue to do so since it's obvious that this issue isn't a concern to the designers. that a known flaw with the main, initial set up of the program is less of a priority than visual aspects that currently works. Just Ignore what you cant easily fix and dress it up making the outside look nice. make it look nice, add new pointless features to bait customers in and when they encounter problems just string them along. 

[bgx 17]

Great to hear! Since a couple of betas I've the same problem. Will wait patiently for the next build :-)

[Spaceboy 2573]

yea, see I use profanity and someone was here instantly to block and respond to it. yet when it comes to trying to resolve the issue you know cant be resolved since it's a issue with the program itself you dont bother to respond. you state that a new update is going to be released soon with performance improvements but still may take time to acquire the library the first time. You should be resolving that issue first. new users try out this program which looks good, until they download it and realize it's going to take days before they are able to utilize the new improvements they will be less likely to continue using it and if they do probably not willing to purchase the premium package. Someone new trying this and encounter that and get the run around like the rest of us has who encountered this issue they'll loose interest before Emby allows them access to their media rendering any other improvements useless. I've suggested this Program to many of my clients over the years until recently and they contacted me complaining about having so much trouble setting it up and having access to their Library with no reasonable support It made me look bad and I had no choice but to suggest other software. and will continue to do so since it's obvious that this issue isn't a concern to the designers. that a known flaw with the main, initial set up of the program is less of a priority than visual aspects that currently works. Just Ignore what you cant easily fix and dress it up making the outside look nice. make it look nice, add new pointless features to bait customers in and when they encounter problems just string them along. 

you know you are talking that stuff you spend hours cleaning out of your horse's stables? emby has thousands of happy users that don't have the problems that you have so clearly the issue is with you. rant all you like but if you want help try providing a log

[ebr 16184]

A log will tell us a lot but what kind of drives are those media drives with letters?  Are they physically attached to the machine or are they mapped network drives?

[SDHoover 1]

Yea, here I go again. For whatever reason Emby wasn't reading my files even though the directory was correct. I had to re install Emby. Now I get to wait a week for Emby to read my library and download it's own metadata into once large consolidated folder taking forever and requiring me to have to wait forever for this process to access me media. Here's an idea. SInce I already have software that downloads metadata, backdrops, trailers, thumbnails. etc and stores it in the folders where the files are located. Why does Emby insist on installing all the same thumbnails, metadata etc in one consolidated file that's required to go through this long ass process anytime you have to install Emby, Re-install Emby, or simply re arrange your media onto different drives storing large amount of data on my C drive where I'd prefer to avoid storing large amounts of data. Just change the program to store all relevant metadata, thumbnails, backdrops, etc. that takes forever onto the video/file locations so that it can just read the files and be done without having to wait for it to re download all the BS. This way that data would remain after having to re install Emby or re arranging file locations. 

[Luke 42078]

 

 

Just change the program to store all relevant metadata, thumbnails, backdrops, etc. that takes forever onto the video/file locations so that it can just read the files and be done without having to wait for it to re download all the BS

 

Hi @@SDHoover, we already have options for this. Did you explore the options when setting up each of your emby libraries?