Slow Library Scanning

[Sireone 6]

Hi, I'm running Emby (latest version) on Ubuntu (Hyper-V VM).  All my media, ~ 1800 movies, is on a Synology NAS.  I've always noticed very slow library scan times.  There's has to be a way to speed this up.  One hour scanning is beyond ridiculous.  Isn't there an option to only scan newly added items instead of the entire library?

 

 

 

[fc7 123]

Can you please share more details about the setup?

 

Hyper-V version? Ubuntu version?

How many vCPUs and RAM did you allocate for your VM? Which network protocol are you using to access the remote library (NFS, SMB)?

Which is the network speed (100Mbit, 1Gbit)?

 

And last but not least, how big is your music library (in terms of items)?

This is important because Emby is doing a lot of throttling against Music Brainz so depending on your music library size and if you are using Music Brainz to get music metadata your scan times can vary significantly.

Just as an example my music library is around 25K items, with Music Brainz enabled it takes around 12-15 minutes to complete a library scan, with Music Brainz disabled is usually less than 3 minutes. We are talking about 5 times less the time.

 

 

Cheers.

Edited May 13, 2016 by fc7

[kjp4756 41]

How do you have your movies organized?  Are all of your 1800 movies in 1 folder?  Could try moving each movie to it's own folder.  For example; /media/movies/Movie (1999)/Movie (1999).mp4

 

I have 1200 movie in my main movie's section.  I had them in 1 movies folder and a scan was ~18 minutes.  I moved each movie in to it's own folder and it's now 18 seconds to scan all my libraries.  My wife's movies are in it's own folder with 77 movies in that folder.  It takes longer to scan her 77 movies than it does the main library with 1200 movies in their own folders.

 

I see some people post about how their scans are so quick and I am thinking the reason why is because of how their library is organized.  People never seem to mention how their libraries are organized when talking about scan times.

[Sireone 6]

Can you please share more details about the setup?

 

Hyper-V version? Ubuntu version?

How many vCPUs and RAM did you allocate for your VM? Which network protocol are you using to access the remote library (NFS, SMB)?

Which is the network speed (100Mbit, 1Gbit)?

 

And last but not least, how big is your music library (in terms of items)?

This is important because Emby is doing a lot of throttling against Music Brainz so depending on your music library size and if you are using Music Brainz to get music metadata your scan times can vary significantly.

Just as an example my music library is around 25K items, with Music Brainz enabled it takes around 12-15 minutes to complete a library scan, with Music Brainz disabled is usually less than 3 minutes. We are talking about 5 times less the time.

 

 

Cheers.

 

Hyper-V is 2012 R2 running Ubuntu 15.10 with 2GB RAM & 4 vCPU.  Synology share is CIFS using AutoFS (no password) at 1Gbps.  As far as music, I have about 1200 songs which have already been identified, so I don't think it's scanning those.  Although, it really could be scanning the whole entire library over and over again.

[Sireone 6]

How do you have your movies organized?  Are all of your 1800 movies in 1 folder?  Could try moving each movie to it's own folder.  For example; /media/movies/Movie (1999)/Movie (1999).mp4

 

I have 1200 movie in my main movie's section.  I had them in 1 movies folder and a scan was ~18 minutes.  I moved each movie in to it's own folder and it's now 18 seconds to scan all my libraries.  My wife's movies are in it's own folder with 77 movies in that folder.  It takes longer to scan her 77 movies than it does the main library with 1200 movies in their own folders.

 

I see some people post about how their scans are so quick and I am thinking the reason why is because of how their library is organized.  People never seem to mention how their libraries are organized when talking about scan times.

 

They are all in one folder, but that should not have anything to do with it.  I've posted before that when I was purely using Kodi and the WatchDog Addon, It would take less than 8 seconds to scan (including metadata) when I simply drop a new movie file on the same Synology NAS share.  What is WatchDog doing that Emby can't???

 

http://kodi.wiki/view/Add-on:Watchdog

Edited May 14, 2016 by Sireone

[fc7 123]

Hyper-V is 2012 R2 running Ubuntu 15.10 with 2GB RAM & 4 vCPU. Synology share is CIFS using AutoFS (no password) at 1Gbps. As far as music, I have about 1200 songs which have already been identified, so I don't think it's scanning those. Although, it really could be scanning the whole entire library over and over again.

I'm running on a similar setup. Also a VM on Hyper-V 2012 R2 with 3GB of RAM (dynamic) and 4vCPUs. Differences are I'm running on CentOS 7.2, using NFS to access the file server which is another VM running on the same vHost so all the networking will stay on the vSwitch.

 

You may want to try adding more RAM to the VM, at least another 1GB (you can use dynamic memory) and also switch from CIFS to NFS. In my case I got better results with NFS.

 

Regarding your library you can try to put each movie in it's own folder. This can be easily done with a script and check if that makes any difference.

 

Also posting the server log including a library scan may help to determine if there is any other problem going on.

 

BTW did you check the load on the synology nas while Emby is scanning? Just to discard any issue on that side two.

Edited May 14, 2016 by fc7

[Sireone 6]

I'm running on a similar setup. Also a VM on Hyper-V 2012 R2 with 3GB of RAM (dynamic) and 4vCPUs. Differences are I'm running on CentOS 7.2, using NFS to access the file server which is another VM running on the same vHost so all the networking will stay on the vSwitch.

 

You may want to try adding more RAM to the VM, at least another 1GB (you can use dynamic memory) and also switch from CIFS to NFS. In my case I got better results with NFS.

 

Regarding your library you can try to put each movie in it's own folder. This can be easily done with a script and check if that makes any difference.

 

Also posting the server log including a library scan may help to determine if there is any other problem going on.

 

BTW did you check the load on the synology nas while Emby is scanning? Just to discard any issue on that side two.

 

I've since added 3GB RAM to the VM, made no difference.  I think the real issue is not with the resources available, but with the scanning technology itself.  Emby should keep a database of what's already scanned and only scan for new content.  Kodi has been doing this for years!

root@emby:/var/lib/emby-server/logs$ grep "library Completed" *
server-63598521605.txt:2016-05-11 01:55:16.0128 Info TaskManager: Scan media library Completed after 55 minute(s) and 9 seconds
server-63598608000.txt:2016-05-12 01:55:16.6913 Info TaskManager: Scan media library Completed after 54 minute(s) and 58 seconds
server-63598694404.txt:2016-05-13 01:58:29.7762 Info TaskManager: Scan media library Completed after 58 minute(s) and 19 seconds
server-63598780800.txt:2016-05-14 02:00:32.0990 Info TaskManager: Scan media library Completed after 60 minute(s) and 32 seconds
server-63598814763.txt:2016-05-14 10:29:15.7104 Info TaskManager: Scan media library Completed after 58 minute(s) and 12 seconds

[kjp4756 41]

They are all in one folder, but that should not have anything to do with it.  I've posted before that when I was purely using Kodi and the WatchDog Addon, It would take less than 8 seconds to scan (including metadata) when I simply drop a new movie file on the same Synology NAS share.  What is WatchDog doing that Emby can't???

 

http://kodi.wiki/view/Add-on:Watchdog

I know it makes no sense but having each movie in it's own folder makes a HUGE difference.  I don't know why, but it does.  Likely a glitch in the emby library scan code. 

 

@@fc7 How are your movies organized?  1 folder or each movie in it's own?  

[fc7 123]

I know it makes no sense but having each movie in it's own folder makes a HUGE difference. I don't know why, but it does. Likely a glitch in the emby library scan code.

 

@@fc7 How are your movies organized? 1 folder or each movie in it's own?

Each movie in it's own folder.

[Sireone 6]

I know it makes no sense but having each movie in it's own folder makes a HUGE difference.  I don't know why, but it does.  Likely a glitch in the emby library scan code. 

 

@@fc7 How are your movies organized?  1 folder or each movie in it's own?  

 

I think what needs to happen is a complete rewrite of the scan code.  Even the VideoStation package in my Synology scans newly added video in seconds, not an hour!

[Wob76 22]

Has something changed in the latest update? my scan time has tripled since updating yesterday.

 

server-63599385600.txt:2016-05-21 02:33:06.2275 Info TaskManager: Scan media library Completed after 33 minute(s) and 6 seconds

server-63599471999.txt:2016-05-22 02:33:10.7279 Info TaskManager: Scan media library Completed after 33 minute(s) and 10 seconds

server-63599558399.txt:2016-05-23 02:32:28.4231 Info TaskManager: Scan media library Completed after 32 minute(s) and 28 seconds

server-63599644800.txt:2016-05-24 02:32:04.0465 Info TaskManager: Scan media library Completed after 32 minute(s) and 4 seconds

server-63599686944.txt:2016-05-24 13:22:18.4430 Info TaskManager: Scan media library Completed after 98 minute(s) and 27 seconds

 

Regards,

Beau

[the1legend 45]

My scan is about the same from the 21st to the 24th

 

2016-05-21 21:22:18.9175 Info TaskManager: Scan media library Completed after 20 minute(s) and 4 seconds

2016-05-24 15:41:47.0011 Info TaskManager: Scan media library Completed after 22 minute(s) and 18 seconds

 

Library is 9400 TV Shows, 4100 Movies, 10800 Songs.

 

EDIT: I'm not using Linux though, server is on Windows 10 so YMMV.

Edited May 24, 2016 by the1legend

[Sireone 6]

Welcome to my world.  No reason for this to take so long and it should be addressed.  What exactly is happening during a scan?

 

server-63599385600.txt:2016-05-21 01:55:29.8400 Info TaskManager: Scan media library Completed after 55 minute(s) and 29 seconds

server-63599472000.txt:2016-05-22 01:55:37.6298 Info TaskManager: Scan media library Completed after 55 minute(s) and 37 seconds

server-63599558400.txt:2016-05-23 01:54:42.5052 Info TaskManager: Scan media library Completed after 54 minute(s) and 41 seconds

server-63599633275.txt:2016-05-23 23:17:43.7195 Info TaskManager: Scan media library Completed after 75 minute(s) and 4 seconds

server-63599644802.txt:2016-05-24 01:57:55.5488 Info TaskManager: Scan media library Completed after 57 minute(s) and 54 seconds

[Wob76 22]

Mine has returned back to just over 30min, so it seems the first scan after the update was just excessive. Watching my logs it appears allot of my scan time is due to MusicBrainz delays, the MusicBrainz servers are very limited on how much traffic they will respond to, so I am not sure if there will be an easy fix for that.

 

It does seem excessive when only a couple of Items in my Movie\TV Library have changed since the previous scan, I am not sure why it is rescanning my Music Library at all. But I my guess is that it is some kind of housekeeping.

[mgworek 134]

I decided to move all my movies to their own folder. The more I thought about it the more I liked, keeping metadata with the movie. My scan went from an 1.5 hours to 6 minutes. I still don't see why it takes so long when they are all in one folder but I'm just glad my scans take around the same time they did when I used other software.

[Sireone 6]

I decided to move all my movies to their own folder. The more I thought about it the more I liked, keeping metadata with the movie. My scan went from an 1.5 hours to 6 minutes. I still don't see why it takes so long when they are all in one folder but I'm just glad my scans take around the same time they did when I used other software.

 

I'm not reorganizing my as I shouldn't have to.  No other media server software on the market takes as long to scan media than Emby.  This really sucks, I just added 4 movies to watch with some friends and it's embarrassing to tell them that we have to wait an hour to start watching anything.

 

What EXACTLY is Emby doing during library scans?  Can someone answer this for me?

[Sireone 6]

BTW, here's the source code for Watchdog if it'll help make things better for Emby in the future, I hope!

 

https://github.com/tamland/xbmc-addon-watchdog

[Wob76 22]

So is this just being logged under expected behaviour? It seems a bit excessive that the Library Scan takes so long each time it runs. 

 

I can run a scan, then run the same scan (via scheduled tasks) and it takes between 30-45min per run, no changes to any files in the Library. 

 

I enabled debug logging, but most of the time the scan is taking place I don't see anything in the logs, what exactly is emby doing during all this scan time?

 

When the log is moving I do see it go fetch some fanart.tv images (particularly music related), but I am not sure why, considering the scan ran just a few minutes earlier, why would it need to regrab this artwork?

 

I have loaded up plex to do some testing and the same sort of scan takes just seconds, same files are being used so Library and folder structure is the same. Both Emby and Plex are running in dockers on unraid.

 

Attached is my last scan, as described above.

 

server-63601631999.txt

[Luke 42077]

Thanks for the report. We are working on improving it. Thanks.

[Wob76 22]

Thanks Luke, let me know if I can do anything to help.

[Sireone 6]

Thanks Luke, let me know if I can do anything to help.

 

Same here Luke, willing to run any tests needed. Thanks.

[Sireone 6]

Any update on Library Scanning performance?  Still taking over an hour on each scan.  I'm trying to figure out EXACTLY what's happening here.  What is Emby actually doing during each scheduled scan???  I noticed that when I run a 'Clean Database', which is supposed to look for which file is no longer in the path and remove it from the library, only takes 10 SECONDS to run.  10 SECONDS!!!.  If Emby can find the file that's missing from the path and update the library, why can't it find a single file and just fetch metadata and update the library?  Why does it have to take 1 HOUR to do that???  Does some here know the answer to that?  Please help me to understand exactly what's going on. 

[Guest plexman]

Any update on Library Scanning performance? Still taking over an hour on each scan. I'm trying to figure out EXACTLY what's happening here. What is Emby actually doing during each scheduled scan??? I noticed that when I run a 'Clean Database', which is supposed to look for which file is no longer in the path and remove it from the library, only takes 10 SECONDS to run. 10 SECONDS!!!. If Emby can find the file that's missing from the path and update the library, why can't it find a single file and just fetch metadata and update the library? Why does it have to take 1 HOUR to do that??? Does some here know the answer to that? Please help me to understand exactly what's going on.

I'm pretty sure there is a reason for that but maybe there is other problems more important to solve. Because library scan still works well even if it takes a lot of time, however, you can do what it's said before of putting all your movies in individual folders to improve your library scan speed.

 

Until they solve this, just chill and enjoy Emby because it's not a big big problem i think.

 

Enviado desde mi Aquaris E5 mediante Tapatalk

[anderbytes 141]

Have you tried not using CIFS? What happens to scan times if your files are acessible from, let`s say, direct attached storage?

[Sireone 6]

Have you tried not using CIFS? What happens to scan times if your files are acessible from, let`s say, direct attached storage?

 

Unfortunately, my Emby server is on a VM, so I cannot attach the files directly (unless I do something like iSCSI, which is a road I'm not going down).  I don't think everyone understands what's going on here.  Every single media server I own (Plex, Kodi, MediaPortal etc.), scans files from this same CIFS share in seconds.  I'm just curious as to why it takes one hour+.  Is it actually looking at every single file (on each scheduled scan) on the share and checking to see if there is updated metadata?  If this is the case, why not have Emby scan JUST the files that were added (using inotfy or FileSystemWatcher), then use the Scheduled scans for the full library.