jonesc2 1 Posted March 22, 2018 Share Posted March 22, 2018 Hi Guys I have nearly 6000 movies running on a mounted GDrive. I have completed a full library scan 2-3 times with save artwork in folders, and predownload images in advance. This same library works fine on Plex, however what I'm finding is every time i re-run a scan after the 1st one has completed it takes over 24 hours to complete (Hitting 90% after about 3-5 minutes) My folder layout is like this: Movies/H264/Movie 1.mkv /Movie2.mkv All the .nfo's and images have been processed as I can see them in the same folder, and after each scan nothing new has been downloaded. I've rinsed and repeated without pre-downloading the images and that doesn't appear to make any difference. Any suggestions? Thannks Link to comment Share on other sites More sharing options...
Luke 37233 Posted March 23, 2018 Share Posted March 23, 2018 Hi, there is currently a known issue that we're investigating related to slows scans when multiple movies are placed in a single shared folder. Most likely there are some operations being repeated over and over. An immediate workaround until this is resolved is to have dedicated movie folders, for example, /Movies/Movie 1/Movie 1.mkv Thanks. Link to comment Share on other sites More sharing options...
jonesc2 1 Posted March 23, 2018 Author Share Posted March 23, 2018 Hi, there is currently a known issue that we're investigating related to slows scans when multiple movies are placed in a single shared folder. Most likely there are some operations being repeated over and over. An immediate workaround until this is resolved is to have dedicated movie folders, for example, /Movies/Movie 1/Movie 1.mkv Thanks. Cheers Luke is there any ETA? Link to comment Share on other sites More sharing options...
Luke 37233 Posted March 23, 2018 Share Posted March 23, 2018 No, sorry. Hopefully in the next release. Link to comment Share on other sites More sharing options...
Carlo 4331 Posted March 24, 2018 Share Posted March 24, 2018 (edited) I'd highly advocate using FileBot to setup you movies using the correct naming and folder names with years. Test locally with test setup. Then test a few at a time on the mounted cloud drive working your way up to large changes. Once you get the hang of it, it can make quick work of proper naming for both Movies and TV Shows. Carlo PS To make navigation and file system return info quicker use a slightly different layout with large collections such as: /Movies /Movies/# /Movies/#/#1 Cheerleader Camp (2010) /Movies/#/10 Cloverfield Lane (2016) /Movies/A /Movies/A/A Bad Moms Christmas (2017) /Movies/A/Abducted (2013) /Movies/B /Movies/B/Bachelor Party (1984) /Movies/B/Big Miracle (2012) /Movies/C /Movies/C/Camp X-Ray (2014) /Movies/C/Captain Phillips (2013) etc Keep in mind the way OS directory modification dates work. This type of layout allows programs to drill down to "changed/new" data much quicker since it's a tree like structure. Not to mention it's highly organized and allows using File Explorer and similar tools much easier! Edited March 24, 2018 by cayars Link to comment Share on other sites More sharing options...
CharleyVarrick 277 Posted March 24, 2018 Share Posted March 24, 2018 Are you sure having lettered mid folders (Movies/A/movie tile (yyyy)/movie title (yyyy).mp4) is a better idea than just Movies/movie tile (yyyy)/movie title (yyyy).mp4? Care to expand on that? Link to comment Share on other sites More sharing options...
Luke 37233 Posted March 24, 2018 Share Posted March 24, 2018 That's just a matter of preference. Link to comment Share on other sites More sharing options...
CharleyVarrick 277 Posted March 25, 2018 Share Posted March 25, 2018 Having year* in parenthesis after movie title (movie title (2017)/movie title (2017).mp4) is a must, otherwise how will Emby guess the correct movie? For instance, doing a search on movie title "The Awakening" will return 34 different movie results. * correct year as per themoviedb.org, which regularly differ from IMDB Just 1 year off (ie: 2016 instead of 2017) is enough to fool Emby into wrongly ID'ing a movie. Link to comment Share on other sites More sharing options...
MikePlanet 140 Posted March 25, 2018 Share Posted March 25, 2018 Having year* in parenthesis after movie title (movie title (2017)/movie title (2017).mp4) is a must, otherwise how will Emby guess the correct movie? For instance, doing a search on movie title "The Awakening" will return 34 different movie results. I think it is not officially supported, but having the imdb-id within the directory name will be used to identify the movie. Unfortunately, not all imdb-ids can be found via themoviedb, but for the common movies this is working and identifying 100%. Link to comment Share on other sites More sharing options...
Luke 37233 Posted March 25, 2018 Share Posted March 25, 2018 You can do [imdbid=xxx] somewhere in the file name. Link to comment Share on other sites More sharing options...
Carlo 4331 Posted March 26, 2018 Share Posted March 26, 2018 Are you sure having lettered mid folders (Movies/A/movie tile (yyyy)/movie title (yyyy).mp4) is a better idea than just Movies/movie tile (yyyy)/movie title (yyyy).mp4? Care to expand on that? It's certainly not a requirement by any means. I've just found with over 10K movies it helps separate the files for easier navigation in the directory tree. Otherwise I would have over 10K directories listed under /Movies. This makes life easier when using other tools such as File Explorer, dir command lines, adding these folders to an FTP server, etc. Any scripts I use that can traverse or work on the whole library can be setup to "narrow down" quickly things that changes since the parent directory will get a change to the "last modified date". Think of it like the TV Shows directory. You don't have all your shows in one large directory. Instead you have sub directories for each show which helps to keep the parent directory smaller. Speaking of TV Shows I do something similar with a nested directory for these as well but it's strictly for my use. I use "ended" and "ongoing" so I for example would have: /TV Shows/Ended /TV Shows/Ended/House (2004) /TV Shows/Ongoing /TV Shows/Ongoing/The Big Bang Theory (2007) So anything listed under "ended" on my system is complete and the series has ended with no additional NEW shows broadcast. Anything in "ongoing" is an active show still broadcasting NEW shows. In Emby clients when I look at a show/episode it will put the words ENDED or ONGOING on the screen. So if your not familiar with the show you can easily tell if it's ended on an active ongoing show. Obviously I can use this tree as well for my own use. For example I know to ONLY concentrate on the ongoing section to bring in new files via any means such as DVR Series setup. This directory layout works just fine in Emby, Plex and Kodi as well as many tools on might use. Hope that helps, Carlo PS just having Ended or Ongoing on the screen in the clients for shows is worth it to me even without any other reason. 1 Link to comment Share on other sites More sharing options...
CharleyVarrick 277 Posted March 26, 2018 Share Posted March 26, 2018 As for the lettered folder, I understand your point, it makes it more manageable for you. But when you navigate those in Emby, doesn't the letters show up there as well? Your Ended and Ongoing TV show folders sounds like a great idea. Link to comment Share on other sites More sharing options...
FreshDog 8 Posted April 2, 2018 Share Posted April 2, 2018 (edited) Hi Luke, As some other users here I also like to keep my movies in a single folder. Therefore I was annoyed about the slow scanning speed, too. So I've looked into the code and I think a have found the reason for the slow scans. In the "BaseItem.cs" placed in "MediaBrowser.Controller/Entities" you have a property "SupportsOwnedItems" (line 1230). Here you're checking if the item is a folder and if it has a parent (GetParent() != null). The GetParent part is the one which produces the extremely long scans. As every movie has a parent (which is the folder of the movie) this results in scanning the movies "n2 + 1" times. I have tried to change the getter of this property from "return IsFolder || GetParent() != null;" to "return IsFolder;". In my case this reduced the re-scan time for about 2300 items from about 1h 40min down to 16 seconds. I'm not really sure which reason the GetParent query has, but the folder itself is also part of the provided list and therefore it gets scanned anyway. Maybe you could change this in the next version. Edited April 2, 2018 by FreshDog Link to comment Share on other sites More sharing options...
Luke 37233 Posted April 2, 2018 Share Posted April 2, 2018 Thanks for the info, I'll look into it. Link to comment Share on other sites More sharing options...
FreshDog 8 Posted April 2, 2018 Share Posted April 2, 2018 Thanks Luke. BTW: Why is the Windows release of Emby Server which I can download from releases differ from the one I can build from the repository? Is there a new repository for Emby? If yes, where can I find it? Or how can I build the Emby Server the same way? Link to comment Share on other sites More sharing options...
FreshDog 8 Posted April 12, 2018 Share Posted April 12, 2018 Thanks for the info, I'll look into it. Any news on this? Were you already able to look into it? Link to comment Share on other sites More sharing options...
Luke 37233 Posted April 12, 2018 Share Posted April 12, 2018 Yea can you try the beta server? Thanks. Link to comment Share on other sites More sharing options...
FreshDog 8 Posted April 12, 2018 Share Posted April 12, 2018 I've tried the current beta v3.3.1.16. Unfortunately the issue still exists. The scan took nearly the same time as before. Even slightly more (1h 46min). As the code for the v3.3.1.16 beta is not in the repository yet, I had to test with the v3.3.1.15 beta. I've tried to remove the same query as before (meanwhile changed to "!ParentId.Equals(Guid.Empty)"). Without this query the scan took only 17 seconds. Link to comment Share on other sites More sharing options...
Luke 37233 Posted April 12, 2018 Share Posted April 12, 2018 It's something else, and not related to that. Link to comment Share on other sites More sharing options...
FreshDog 8 Posted April 12, 2018 Share Posted April 12, 2018 It's something else, and not related to that. The reason for the long scan time is just the amount of scans which are performed if all the movies are placed in a single folder. Because every movie has a parent, which is the folder of the movie itself, SupportsOwnedItems returns true. Then in the RefreshMetadata method the contents of the same folder are retrieved with the GetFileSystemChildren call for every movie and the same 2,300 movies are refreshed with the RefreshedOwnedItems call over and over again. Therefore instead of scanning just the 2,300 movies in the folder 5,290,000 scans (2,3002) are performed. Link to comment Share on other sites More sharing options...
Luke 37233 Posted April 13, 2018 Share Posted April 13, 2018 Right exactly, there is some processing happening over and over again that we need to look at. Link to comment Share on other sites More sharing options...
FreshDog 8 Posted April 13, 2018 Share Posted April 13, 2018 Does that mean that you will remove the named call ("!ParentId.Equals(Guid.Empty)" inside SupportsOwnedItems) in the next version? Link to comment Share on other sites More sharing options...
Luke 37233 Posted April 13, 2018 Share Posted April 13, 2018 No, the issue is elsewhere. Link to comment Share on other sites More sharing options...
FreshDog 8 Posted April 14, 2018 Share Posted April 14, 2018 I'm sorry, but I don't understand that. According to my tests this is exactly the place. The scan time difference also prove that. And I didn't notice any side effects of the change. If I'm wrong, then could you please explain why I'm wrong? Then I could at least try to analyze further as this fix is important for me (and I think also for many other). 2 Link to comment Share on other sites More sharing options...
CBers 6785 Posted April 14, 2018 Share Posted April 14, 2018 No, the issue is elsewhere. I'm sorry, but I don't understand that. According to my tests this is exactly the place. The scan time difference also prove that. And I didn't notice any side effects of the change. If I'm wrong, then could you please explain why I'm wrong? Then I could at least try to analyze further as this fix is important for me (and I think also for many other). @@Luke - if this is a bug, then surely this takes precedence over new features etc. To me, @@FreshDog looks like he knows what he's talking about, so perhaps you could expand on your comment, if only to put his mind at rest that you are actively looking into it. Just fobbing someone off with "No, the issue is elsewhere." is not good. I personally don't have this issue as I have one movie per folder, but perhaps it is affecting many people. Please provide a more constructive response. 4 Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now