Jump to content

Extremely long rescans


RPro

Recommended Posts

Ever since version 3.0.6020, my library has been experiencing excessively long rescans.

 

The library is placed on an SMB share on a ZFS filesystem, running OpenIndiana (and IllumOS variant based on Solaris), under VMWare ESXi 6.

 

Emby Server runs under Windows 2008 R2 on the same ESXi machine. This was not a problem before version 3.0.6020. Scans were done in reasonable times (only a couple of minutes, with 30 TBs worth of movies and TV shows - only a thousand titles and episodes).

 

Dozens of entries like these are showing up in the log on rescans, and ffprobe and ffmpeg always run when I restart the server or manually scan the library:

2016-08-12 15:46:43.1079 Debug App: Date modified for \\HORZA\media\Movies\Test Patterns\SPEARS & MUNSIL HIGH DEFINITION BENCHMARK. Old date 1/1/0001 7:00:00 AM new date 1/1/0001 12:00:00 AM Id 831db1b9-5636-cb27-ac17-8261a390e2b2
2016-08-12 15:46:43.1079 Debug ProviderManager: Saving \\HORZA\media\Movies\Test Patterns\SPEARS & MUNSIL HIGH DEFINITION BENCHMARK to Nfo.

2016-08-12 15:46:43.1079 Debug App: Saving \\HORZA\media\Movies\Test Patterns\SPEARS & MUNSIL HIGH DEFINITION BENCHMARK to database.

I think it's related to the incorrect date being assigned to the entries, triggering ffmpeg/ffprobe to run all the time. This basically makes it impossible to use the server.

 

Things I've tried:

 

- using BulkFileChanger to force file modification time to be the same as file creation time (many of the files were copied, so file mod time was less than file creation time after the copy).

- using touch command-line command to force all files in the library (several thousand files) to have the current date and time in the file modification date.

- deleting library.db and refreshinfo.db and starting the server

- changing all permissions on every file so that Everyone gets full control. ACL permissions set the same for every single file.

- deleting all the xml and nfo files generated by Emby and starting the server to re-scan.

 

If the "new date" is not misconstrued as 1/1/0001 12:00:00 AM then it doesn't get scanned again on restart. However, all the files/directories that are given the incorrect date of 1/1/0001 12:00:00 AM are always rescanned. This takes hours, and means I can not add any new content in a timely manner.

 

Nothing seems to work! The only thing I haven't tried is moving the files off the server, onto a local hard drive, and copying back. That could take an entire day.

 

Any ideas? Or should I give up on Emby? :(
 

Edited by RPro
Link to comment
Share on other sites

Oh I forgot to add - I also went into Metadata Manager and selected Refresh. This worked a couple of times on the affected titles, but stopped working later on. Weird.

Link to comment
Share on other sites

Ok I think I narrowed the problem down.

 

Version 3.0.6060 definitely has the problem. Version 3.0.6030 behaves differently - it does not use a new date of 1/1/0001 12:00:00 AM, and instead uses the current time for the titles I mentioned above.

 

 

2016-08-12 19:40:13.4183 Debug App: Running MovieNfoProvider for \\HORZA\media\Movies\Blu-ray\Conan the Barbarian (1982)
2016-08-12 19:40:13.5258 Debug App: Date modified for \\HORZA\media\Movies\Blu-ray\Conan the Barbarian (1982). Old date 1/1/0001 7:00:00 AM new date 8/12/2016 8:18:37 PM Id 7cf801b6-5c51-f1cc-e806-0751d1516771
2016-08-12 19:40:13.5333 Debug App: Saving \\HORZA\media\Movies\Blu-ray\Conan the Barbarian (1982) to database.

 

The above title would ALWAYS have a new date of 1/1/0001 12:00:00 AM in 3.0.6060.

 

It will be a few hours before I can determine if 3.0.6030 will scan for 4 hours every time I restart the server or initiate a scan...Ugh.

Link to comment
Share on other sites

It appears to only happen on folders. I couldn't figure out a pattern to why it only happens to 100 of them, but not the reset. Resetting the modification times to the current time didn't make any difference. Also,  the creation times were normal and not in the future or before the year 2000. All the permissions are the same too. Even the file ownership is all the same.

 

Here is a (large) log - debug enabled of a scan I did last night (rar format):

https://dl.dropboxusercontent.com/u/1156664/Emby%20Logs/Log.rar

 

Thanks for looking into this!

Link to comment
Share on other sites

rui no onna

I'm experiencing the same issue on one of my libraries (TV DVD/VIDEO_TS folder rips).

 

Since this folder is pretty much static, is there a way I can exclude it from the library scans while still keeping the content in the library? I tried enabling the "Lock this item to prevent future changes" setting but that didn't do anything.

Link to comment
Share on other sites

rui no onna

Hi, this is resolved for the next release.

 

Yup, saw that. Given the next stable release isn't available yet though, I was just wondering if there's a workaround I can do in the meantime to prevent the long scan times short of removing the entire DVD folder rip library. Thanks!

Link to comment
Share on other sites

Happy2Play

Yup, saw that. Given the next stable release isn't available yet though, I was just wondering if there's a workaround I can do in the meantime to prevent the long scan times short of removing the entire DVD folder rip library. Thanks!

Only options is wait or update to the beta release.

Link to comment
Share on other sites

feardamhan

I just updated to 3.0.6070.0 and am having similar problems. What build is the fix in? Nearly every time my system wakes from sleep, so part of the library is missing (tonight it was the entire TV Series folder). Takes as long as 30 mins to scan and find all the content again, even when there is nothing new. Just wondering if my current build is meant to have the fix or if its in a pending build......

Link to comment
Share on other sites

Happy2Play

For the OP issue the fix was in that release 3.0.6070.

 

•Fix dvd & bluray folder rips refreshing on every scan

Link to comment
Share on other sites

feardamhan

Thanks. My issue is strikingly similar but has nothing to do with DVD rips....these are all current TV shows I get on a weekly basis. Time for some logs? Suggestions of what to pull? Dont know whether to open a new thread or continue with this one.

Link to comment
Share on other sites

Happy2Play

I would start a now one explaining the issue in detail and provide server log.

Link to comment
Share on other sites

It sounds like you're taking your media offline? I would just adjust your library scan schedule in server-> scheduled task to something that makes more sense for you when the content will be available.

Link to comment
Share on other sites

MikePlanet

It sounds like you're taking your media offline? I would just adjust your library scan schedule in server-> scheduled task to something that makes more sense for you when the content will be available.

Hi Luke,

honestly, I think that is something the server should be able to handle with own intelligence. We already had some threads about the behaviour of the library scanner and how it should/could behave, when the remote server is offline/sleeping. Imho the server should handle that in as follows:

- Monitor availability of the remote server(s) (select the distinct server names from the media pathes)

- When a remote server goes offline/online, the thread should put all related media items to offline/online (could be done with only one statement and is very efficient/quick)

- The libraryscanner needs a switch to know, if it should simply ignore offline sources or remove them from the library (the latter option is required to cleanup the library from entries, if a server will never come back because it is gone out of service - normally, offline media should simply ignored and NOT be removed from the library)

Edited by MikePlanet
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...