Slow library scan

[charles-martel 0]

Hi,

Librariess scan after initial scan are slow if I compare to Kodi (a full scan takes less than 1 minute).

I have a Synology NAS with a SMB share, in the share there are approximativly 2000 movies and 3200 TV shows.

Emby runs on Windows 10 computer (i7, 16G RAM, SSD). Entire scan takes little more than 20 minutes.

If I run an individual scan of TV shows, it takes a couple of seconds, I think the problem is in movies but I don't see anything strange in the logs.

How does the library scan process work ?

I run Emby 3.2.70.0, no plugins and no particular time consuming options (chapter picture extraction, ...)

 

PS : sorry for my english.

Edited February 1, 2018 by charles-martel

[Happy2Play 9780]

What is the folder structure of your movies?

[charles-martel 0]

Movies are in one folder.

[Luke 42077]

Hi there, please attach the emby server log, thanks.

[charles-martel 0]

Luke, I send you the log in MP.

[Happy2Play 9780]

Movies are in one folder.

 

When you say in one folder do you mean "\Movies\somemovie (year).ext"?  If so others have reported converting to "\Movies\\Moviename (year)\Moviename (year).ext" have significantly increase scan speeds.

[charles-martel 0]

Yes, my folder structure is "\Movies\somemovie (year).ext". I saw that some people have reported that, but I prefere not use one folder per movie. Why it's faster with this folder struture ?

[PenkethBoy 2068]

probably because emby has a lot less variables to account for with movies in sub folders

[Luke 42077]

The scan in your log was 22 minutes. is that about how much the scans normally are?

[charles-martel 0]

My librariess full scans weren't less than 20min. I don't know if it's normal.
 
Compared to TV show library scan (less than 1 minute) it's slow. (and full Kodi scan of the same medias take less than 1 minute also, but it's not the same software)
A priori it's because my movies are all in single folder ?
 

When you say in one folder do you mean "\Movies\somemovie (year).ext"? If so others have reported converting to "\Movies\\Moviename (year)\Moviename (year).ext" have significantly increase scan speeds.

 
My TV shows have folder structure "TV show\season\episode".
 
 
I also tried with Emby installed on my NAS, I have same issue it's just to slow, the full scan take 2 hours and 50 minutes, I think it's because it's not as powerful as my PC.

Edited February 8, 2018 by charles-martel

[Happy2Play 9780]

My librariess full scans weren't less than 20min. I don't know if it's normal.

 

Compared to TV show library scan (less than 1 minute) it's slow. (and full Kodi scan of the same medias take less than 1 minute also, but it's not the same software)

A priori it's because my movies are all in single folder ?

 

 

My TV shows have folder structure "TV show\season\episode".

 

 

I also tried with Emby installed on my NAS, I have same issue it's just to slow, the full scan take 2 hours and 50 minutes, I think it's because it's not as powerful as my PC.

The are a lot of users that have reported better scan performance with movies in there own folders.

Edited February 8, 2018 by Happy2Play

[charles-martel 0]

The scan of library with the movies in a single folder will be improved ?

[ebr 16169]

The scan of library with the movies in a single folder will be improved ?

 

There are too many variables in play to be sure but it is possible.

 

It is definitely true that Windows at least has performance problems with directories with large numbers of files in them.  Breaking that out into more of a tree definitely can improve basic file I/O performance.

[PenkethBoy 2068]

@@charles-martel

 

Just did a test to see if having lots of files in one directory vs. Each file in a subdirectory of the the library folder would make a significant difference - as i was curious

 

Turned off metadata download to avoid any variability with metadata providers

 

Library is located on a local SSD - so disc access not a factor

 

copied 557 video files to one directory - let Emby scan them in - checked they were all visible in the library home page

 

Then ran the library scan scheduled task - consistent time was 190-195 seconds

 

Next copied all the same video files to their own subdirectory (simple script)

 

Let emby rescan the "new" files (and delete the old ones) - checked the correct number of video's were present in the library with their new path's

 

Then ran the library scan scheduled task again - consistent time of 2-3 seconds!!! - this was a bit of a surprise - so shut down the emby server and restarted to see if there was any caching going on - time to scan was still 2-3 seconds

 

This was done on a win10 machine with the stable .70 server

Edited February 14, 2018 by PenkethBoy

[charles-martel 0]

It is definitely true that Windows at least has performance problems with directories with large numbers of files in them.  Breaking that out into more of a tree definitely can improve basic file I/O performance.

 

I have the same issue on Linux (Synology).

[ebr 16169]

@@charles-martel

 

Just did a test to see if having lots of files in one directory vs. Each file in a subdirectory of the the library folder would make a significant difference - as i was curious

 

Turned off metadata download to avoid any variability with metadata providers

 

Library is located on a local SSD - so disc access not a factor

 

copied 557 video files to one directory - let Emby scan them in - checked they were all visible in the library home page

 

Then ran the library scan scheduled task - consistent time was 190-195 seconds

 

Next copied all the same video files to their own subdirectory (simple script)

 

Let emby rescan the "new" files (and delete the old ones) - checked the correct number of video's were present in the library with their new path's

 

Then ran the library scan scheduled task again - consistent time of 2-3 seconds!!! - this was a bit of a surprise - so shut down the emby server and restarted to see if there was any caching going on - time to scan was still 2-3 seconds

 

This was done on a win10 machine with the stable .70 server

 

Interesting.  In addition to the underlying OS implications, when you put individual movies in their own folders it makes it much easier for Emby to identify them which probably translates into less code that has to run in the process.

[PenkethBoy 2068]

the limits in windows was a winxp/fat32 era thing IIRC - ntfs has a limit at ~4 billion or so per folder

 

yes the folder browsing might get slower at very high numbers - i.e. many tens of thousands - but its not a problem in everyday use for 99.9999999999% of users

 

might be different in a corporate server environment if the hardware is not up to the load

[charles-martel 0]

Hi,

 

I have some news.

 

Some time ago I used emby 3.5.3.0 with docker on my synology ds918+, the scans are very faster (~2 min, after the initial scan), comparatively with the others version of emby.

All of my movies (~2500) are in the same folder, like my previous posts.

 

I thinked the slow scan issue was fixed but, I maked an update to 4.0.1.0 and the scans are terribly slow (~1 hour and 43 minutes). I tried with a new install, I have the same issue.

 

Best regards.

Edited January 29, 2019 by charles-martel

[Luke 42077]

We're looking into it, thanks.

[brunovon 1]

The auto scan for new movies was working great until the latest update. Now it is a mess. It does not even scan. Even when manually scanning, it takes a bit to find the movie and then does not display it in the begining (i have sort by date added). Then search will show the movie after 5-10 minutes without meta data. I have to then identify and manually add the metadata. All my movies and metadata are in one folder.

 

What happened?

[Luke 42077]

We are looking into it, thanks.

[Gaspar Filmes 8]

The auto scan for new movies was working great until the latest update. Now it is a mess. It does not even scan. Even when manually scanning, it takes a bit to find the movie and then does not display it in the begining (i have sort by date added). Then search will show the movie after 5-10 minutes without meta data. I have to then identify and manually add the metadata. All my movies and metadata are in one folder.

 

What happened?

Yes, i can confirm it too.. Since update from 3.5.3 to 4.0.1, the time to auto scan for just 1 new content are taking about 5~10min, now, I Recently uploaded 6 new contents and are taking about 30min+...

[Luke 42077]

We're working on it, thanks guys.

[charles-martel 0]

Hi,

 

I tested the beta 4.1.0.5 and the scan returned to the normal like the 3.5.3.0 (~2 mins vs 1 hours 40 mins) . What did happen ?

 

Best regards.

[Luke 42077]

There are improvements in the beta server. Thanks for the feedback.