Scan sticks at 67.8%

[Gilgamesh_48 1240]

Due to  some rather extreme stupidity on my part I had to completely remove Emby and then reinstall it.

 

In the removal I choose "Remove all traces."

After it completed I rebooted the computer and then re-installed Emby,

During the install process I choose to setup, to start with, only one library. That library is for TV shows.

The scan started and ran to 67.8% and stuck and failed to progress any further for over two hours.

I then restarted the server and restarted the scan and it only ran again to 67.8% and stalled for over an hour.

Then I stopped the scan and rebooted the computer,

Once more the scan stalled 67.8%. I am allowing it to continue but it is still stalled.

My computer is running Windows 10 and my library is on another computer that shares th drive.

 

Before my stupidity Emby was running on the same computer and had successfully scanned all my libraries and nothing else changed on the computer.

 

I have attached my server log.

Emby_server_Logs.txt

 

Thanks for looking at this somewhat perplexing problem.

[Luke 42083]

Hi, it's not stuck, it's just moving slowly, most likely due to the fact that you installed the Roku thumbnail plugin. If you recall, when you installed that, you were warned that it would result in significantly longer library scans.

 

To improve the performance of your library scan, try uninstalling the plugin.

[Gilgamesh_48 1240]

Hi, it's not stuck, it's just moving slowly, most likely due to the fact that you installed the Roku thumbnail plugin. If you recall, when you installed that, you were warned that it would result in significantly longer library scans.

 

To improve the performance of your library scan, try uninstalling the plugin.

 

Well the stalling or extreme slowdown happened before the Roku plugin was installed. I installed it while waiting for the scan to progress. But I have now removed the Roku plugin and the progress is still EXTREMELY slow. It has progressed only 1% in the last hour.

 

Any ideas about how to get the scan running properly again? It only took about 4 hours to complexly scan my TV library on my previous install on this same machine. 

 

Edit: I rebooted the computer after I removed the Roku plugin.

Edited February 21, 2018 by Gilgamesh_48

[ebr 16187]

Did you tick any other options that also impact the scan speed like enable extra metadata providers or download all images in advance?

[Gilgamesh_48 1240]

Did you tick any other options that also impact the scan speed like enable extra metadata providers or download all images in advance?

 

Nope, at least I do not think so.

[mastrmind11 722]

Just let it run.  If you don't think it's doing anything, tail the log file and see what its doing.  There will be timestamps on each line.

[Mibok 158]

In my experience, sometimes those long scans are caused by some problematic media like a folder that the server have problems reading. That could cause that the scan always stops at the same point, when is trying to read that problematic media. Try to segment you media adding different parts to scan and see if you can find any movie that the server could've problems reading.

[Gilgamesh_48 1240]

Well the scan is moving, sort of. Since I started this thread the scan has been running nearly continuously except for three or four restarts. It has progressed from 67.8% to 72.9%. 

 

That is not much progress but it is progress.

 

The first time I installed Emby on this computer the entire scan took less than 24 hours and that was for all four libraries not just my TV library.

 

My TV library is by far my largest with almost 30,000 episodes while my Movie library has a, by comparison, paltry 3000 movies. In total size (by disk space) my TV library is about 50 times larger than my movie library. But That was just about the same on my earlier install of Emby.

 

BTW: As a point of interest I just also redid my Plex installation on my Shield TV and it took a little over 30 hours for a full scan of all libraries and the computer Emby is running on is more powerful that the Shield by about 50%.

 

I am perfectly willing to allow the scan to continue as long as it takes but this does raise some concerns as it might mean that something is seriously wrong withe the install.

[Luke 42083]

We're probing every single media file. It is going to take a little time.

[Gilgamesh_48 1240]

We're probing every single media file. It is going to take a little time.

 

I figured that but it was just taking so long this time.

 

However something happened right after the scan passed 75%, The scan completed within 15 minutes. That is after taking well more than a day to go from 0-75% it went from 75-100% in just minutes.

 

I am not complaining but rather I am just observing.

 

I do see another issue but since it is unrelated to this in that it involves watched status I will start another thread.

[Guest asrequested]

It probably just wasn't accurately representing the scan progress.

[ebr 16187]

However something happened right after the scan passed 75%, The scan completed within 15 minutes. That is after taking well more than a day to go from 0-75% it went from 75-100% in just minutes.

 

There is really no way for us to make that progression linear so, perhaps, we should just not report the pct number...

 

The process has a certain number of "things" it needs to do and not all of those things take the same amount of time - in fact, some of them will take orders of magnitude longer than others.  Add to that the fact that there is no way for us to accurately predict how long each "thing" will take and you end up with the display you are seeing.

 

The % progress shows you how many of the things have been completed compared to the total number of things to do but this does not relate to time like you might expect.

[Gilgamesh_48 1240]

There is really no way for us to make that progression linear so, perhaps, we should just not report the pct number...

 

The process has a certain number of "things" it needs to do and not all of those things take the same amount of time - in fact, some of them will take orders of magnitude longer than others.  Add to that the fact that there is no way for us to accurately predict how long each "thing" will take and you end up with the display you are seeing.

 

The % progress shows you how many of the things have been completed compared to the total number of things to do but this does not relate to time like you might expect.

 

You probably should keep the percent indicator even though it does not convey completely accurate info. It does give an indication that something is happening. You might consider augmenting it with an info box that displays something like "Currently processing xxxxxxxx." While that would convey no real useful info either it would let users know that actual work is being done.

 

Back in the day when I programmed I wrote a program to consolidate a bunch of accounting info from all over the country. This took time, a lot of time, due to the much slower data movement in that era. After it was finished and working well I began to receive complaints that it was frozen at times when it was actually just getting data. 

First I tried just displaying a "busy" indicator but that was also met with complaints. Then I provided a progress indicator but had the same problems as you have because some parts were faster than others. That met with complaints as well. Then I added a little useless text box that showed a snippet of each record as it was being imported and the complaints stopped.

The display of the text actually slowed things down a bit and provided no useful info to the users but it made them happy so I left it in place. 

I think that a little more info during scans, even if really useless, would keep mistaken complaints like mine from showing up as often.

[mastrmind11 722]

Why not list the main steps performed in the library scan, and display an indicator when each step is finished (possibly w/ text like "this step takes a while")  Or, instead of percentage just make it x step/y steps with a note that x step takes a while to complete.

[ebr 16187]

Why not list the main steps performed in the library scan, and display an indicator when each step is finished (possibly w/ text like "this step takes a while")  Or, instead of percentage just make it x step/y steps with a note that x step takes a while to complete.

 

The simple answer to "why not ..." is that would be a fairly complex piece of work that falls outside of how every other task reports its progress and would only serve to combat this problem of perception .

 

So, if we can combat the perception problem in an easier way, I think our time is best spent on other items.

 

The display of items being processed is a good one too but, again, isn't as easy to pull off as it may seem.

[speechles 2055]

As an aside, I see this happen on my system once the % reaches 72.8% it starts to drag and take ages just to crawl up each .1% could take an hour. This is music causing the slowdown in my case, music which has not had the id3 tags cleaned up and such. This causes the artists, studios, and other information to cause duplicates. Think AC/DC vs AC-DC vs AcDc. The library can grow exponentially filled with redundant data and as a result, slows things down. In my case, this is the cost of doing business and is an acceptable trade off that I don't mind.

 

You probably should keep the percent indicator even though it does not convey completely accurate info. It does give an indication that something is happening. You might consider augmenting it with an info box that displays something like "Currently processing xxxxxxxx." While that would convey no real useful info either it would let users know that actual work is being done.

 

I say keep the %, it means something and it going upwards indicates progress to users. Even if the progress isn't even among the entire range. Without it all you see is a "busy" (spinning wheel/spinning dots/etc) which isn't very intuitive to indicate to users what is going on or how long. But adding more context such as "currently processing: XXXXXXX' would actually cause the scan to take longer. Instead look at what other programs do....

 

Ever copied files (on windows) and used the "More details" (drop down) button. After you click this it starts to report everything it is currently copying to the display as a single line. This extra overhead slows down the copy since the resources must now devote time to telling the user exactly what is going on. Depending on size this time slowdown difference between "less detail" and "more detail" grows exponentially.

 

Perhaps the same could be done during a scan as windows does and have "More Details/Less Details" buttons?

Edited February 22, 2018 by speechles