Linux Library Scan

[dcook 299]

What is normal for a Library Scan on a Linux Emby Server?

 

Mine is taking over an hour which does not seem right.

Also what I don't understand, if I do a Library Scan right now, wait until the scan is completed and then without changing or adding anything to the library do a 2nd Library Scan the 2nd scan takes just as long as the 1st scan, however nothing has changed.

 

Shouldn't the 2nd scan take only seconds or minutes since there is no change?

 

My understanding is that the Initial Library Scan is normal to take long time, but the regular scans after that point should be very fast since they are only incremental?

 

Or is each time a Library Scan is run it is doing a Full Initial scan?

 

 

[Luke 42077]

Interestingly the opposite feedback here: https://emby.media/community/index.php?/topic/45402-library-scan-speed/

[mastrmind11 722]

My initial scan took a little over an hour.  Now the scan averages just over 2 mins.

[dcook 299]

So from watching the scan its pretty fast up until 60% then it sits for long time, and then goes fast again.

 

This was the 2nd scan which I started right after the previous scan finished.

 

I have attached the log

 

 

 

 

Any ideas why its taking so long to scan?

logmar10.txt

[Luke 42077]

It looks like you enabled downloading images in advance, and that library scan is downloading lots of images. That will definitely cause it to take longer.

[dcook 299]

Right but once the scan completes 100% 1st time, shouldn't it be fast when it runs in the future?

 

 

It looks like you enabled downloading images in advance, and that library scan is downloading lots of images. That will definitely cause it to take longer.

[Luke 42077]

Yes.

[dcook 299]

So that is what is not happening, I make no changes to the library and it takes about the same amount of time to complete.

 

 

 

Yes.

[dcook 299]

@@Luke

 

So I ran it a 3rd time right after the one I posted about above finished and this time it was:

 

 

Your saying the subsequent library scans should be much quicker if there is no change in the media library, however that is not what I am seeing with this version of Emby.

log-mar10-2.txt

Edited March 10, 2017 by dcook

[Luke 42077]

Try removing the genre cleaner plugin.

[dcook 299]

I removed Gene Cleaner plugin (there are no other plugins) and I started the scan again:

 

 

Clearly there is something wrong with this 3.2.7 build, the incremental scans used to be fast

 

It should not take 45 minutes if nothing has changed in the Media Library

[dcook 299]

So with exactly the same emby server settings, no plugins, and the same media library if I try a Library Scan on my Windows 10 machine:

 

 

Can you provide any insight on what has changed in the Linux version that would cause the slowness?

[Luke 42077]

Try changing the setting back related to downloading images ahead of time. i am not really sure what's happening but I do not see this on an ubuntu test machine. but looks like your scan for some reason is re-fetching metadata.

[dcook 299]

Ok, I will uncheck: downloading images ahead of time and try to run a scan now

 

However like I said I do have the same checked on my Windows 10 Server and finishes in less than 15 mins with the same media library

[dcook 299]

@Luke 

 

Windows 10 Server: Scan media library - 10 minutes

Centos 7 Server: : Scan media library - 45 mins

 

Same media library, all Emby settings the same

No plugins

Unchecked "Download Images in Advance"

 

 

The scan goes fast up until about 57-60% then it sits there and does nothing, when its at this stage of the scan, if I do tail -f log file, this is where its waiting:

2017-03-10 22:35:19.3767 Info HttpServer: HTTP GET http://mydomain:8096/emby/ScheduledTasks?isHidden=false. UserAgent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
2017-03-10 22:35:19.3780 Info HttpServer: HTTP Response 200 to 99.230.114.37. Time: 1ms. http://mydomain:8096/emby/ScheduledTasks?isHidden=false
2017-03-10 22:36:28.6265 Info HttpServer: HTTP GET http://mydomain:8096/emby/Users/1cd2d0e3722348c2a04614c9ad32f77e/Items?Recursive=true&Limit=3&Filters=IsNotFolder&SortBy=DateCreated&SortOrder=Descending&Ids=ea0955d855b475f8a7103dc3e7b7c2a1%2C296978aaab33ca7fd32e37edd2fc2f11%2Cf8f626946363ff96d600e66a480f9063%2C9e29520544ebafe5220a040e7139f452%2C61bc084dc80641dfbb69a7ee93878e75&MediaTypes=Audio%2CVideo. UserAgent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
2017-03-10 22:36:28.6309 Info HttpServer: HTTP Response 200 to 99.230.114.37. Time: 4ms. http://mydomain:8096/emby/Users/1cd2d0e3722348c2a04614c9ad32f77e/Items?Recursive=true&Limit=3&Filters=IsNotFolder&SortBy=DateCreated&SortOrder=Descending&Ids=ea0955d855b475f8a7103dc3e7b7c2a1%2C296978aaab33ca7fd32e37edd2fc2f11%2Cf8f626946363ff96d600e66a480f9063%2C9e29520544ebafe5220a040e7139f452%2C61bc084dc80641dfbb69a7ee93878e75&MediaTypes=Audio%2CVideo

Based on my observations when it gets to this stage in the scan, the percent done indicated increases by approximately 0.01 percent every 1.5 to 2 minutes

 

Any idea what its trying to do at this step?  While the percent indicator is very slowly increasing there is nothing new being written to the log file.

Edited March 11, 2017 by dcook

[dcook 299]

Here is log file.

 

After turning on debug mode, I notice some ffmpeg errors, but I am not sure what they mean?

 

 

log-mar10-3.txt

[Luke 42077]

The last scan that took 46 minutes is a good sample. There's a little metadata downloading but not a whole lot. The library scan goes recursively through every folder looking for files that have changed. I wonder if this is just how fast the I/O is in this environment.

[dcook 299]

@@Luke can you make library scans per library?  I think that would help alot

 

For example my Movie Library does not change often, but my TV library changes every day.

 

I don't need to scan my Movies often, but since Real time monitoring does not work, I would like to set my TV library to scan more often

 

 

What about the ffmpeg errors in the log?

Edited March 11, 2017 by dcook

[Luke 42077]

Yes I think that is a good idea for the future.

 

What ffmpeg error do you mean?

[dcook 299]

I will have to find the error again and let you know.

 

What kind of timeline do you think in order to break out library scan to be able to schedule it to run on a per media library basis?

 

I think this would make a huge difference for many people, as I am sure most of us have libraries that don't change often and some that do, and to be able to define which libraries to scan often would be great.

 

 

 

Yes I think that is a good idea for the future.

 

What ffmpeg error do you mean?

[dcook 299]

On my Linux Server VM I changed from 1 Cpu Core to 4 Cpu Cores everything else remained the same.

 

 

Much better now, looks like 1 Cpu Core was not enough to finish the scan quickly.

[Luke 42077]

Thanks for the report !