tenzen 1 Posted March 16, 2016 Share Posted March 16, 2016 (edited) Hello, Looking for some help figuring out what the issue is. in Short: My library scans take hours to complete (not the initial scans) when I read it should take seconds. My significantly reduced library of just 1000 videos takes 18minutes to complete.Currently: - Running Emby Version 3.0.5903.0 but have been fighting this for months and many different versions - I have reduced my library very much to help diagnose - a single movies library with 1941 items in a folder (approx: 1/2 movies, 1/2 subtitles) - I have run ffprobe on every single item and FFProbe can successfully pull out details without error - The full scan took about 2 hours, on the 1941 items which seems reasonable to me - The re-scan after full scan (Scheduled Tasks -> Scan Media Library) takes about 18 minutes. - If i have my full catalog loaded, it takes hours. - I use disk pooling software; however, the problem exists with DriveBender, DrivePool, and even if I move my library to an independent disk and try without pooling - During scan, Emby bounces between 25-40% cpu usage. Seems reasonable to me too. - I deleted everything, cleaned the database, recreated the single library, completed initial scan, and then scanned again -- still 18 minutes - Currently I have my library watching the folders of my pools disks directly (not via the pool drive letter) but the result is similar either wayI have reviewed many articles in the forums and didnt find any solutions, such as: - http://emby.media/community/index.php?/topic/31600-library-scan-excruciatingly-slow/?hl=%2Blibrary+%2Bscan - I have a default install, no plug-ins - http://emby.media/community/index.php?/topic/31406-problem-scanning-library/?hl=%2Blibrary+%2Bscan - Default install, no plug-ins - http://emby.media/community/index.php?/topic/30628-scan-media-library-takes-more-than-2-hours/?hl=%2Blibrary+%2Bscan - Already running the beta stream - http://emby.media/community/index.php?/topic/29732-scan-takes-too-long/?hl=%2Blibrary+%2Bscan&do=findComment&comment=284947 - Removed my music libraries entirely - http://emby.media/community/index.php?/topic/28555-scan-media-library-failing-for-days-about-50/?hl=%2Blibrary+%2Bscan&do=findComment&comment=280613 - Not getting disk errors, StableBit Scanner is reporting healthy disks, Windows logs show no disk errors. - http://emby.media/community/index.php?/topic/28825-long-library-scan-question/?hl=%2Blibrary+%2Bscan - Seems unrelated.So, In all, I've reviewed my logs and to my unfamiliar eye, during the library scan the only thing being written to the logs (in debug) are my accesses to the web portal...The logs have two of those library scans taking about 18 minutes each to complete. In the log file you will see a fresh restart and single library scan. Thoughts? server-63593711515.txt Edited March 16, 2016 by tenzen Link to comment Share on other sites More sharing options...
Deathsquirrel 741 Posted March 16, 2016 Share Posted March 16, 2016 Turn off hte creation of chapter images during the scan. It's in the Dashboard under Metadata > Advanced > Chapters Link to comment Share on other sites More sharing options...
davidtjudd 37 Posted March 16, 2016 Share Posted March 16, 2016 Posting a reply since my setup is similar and I was experiencing many of the same issues. I run stablebit drivepool as well with several thousand movies/tv shows on a dedicated 2k12R2 box. What I was seeing was; Long scan times Constant scans due to new media being added Clients being unresponsive during scans No errors in logs/No disk saturation In other words the system was unusable half the time. Like you I tried several things. Removing all plugins, disabling chapter scans, removing parts of my library, running the beta. None of these worked. Ultimately I found two separate things that actually worked. If I swapped out sqlite with berkeleydb scans still took a long time but the library was accessible since berkeley didn't lock the db files. I did this on a VM with access to the drivepool for testing. This worked. Although berkeleydb has a replacement for the sqlite lib (System.Data.SQLite.dll) and I wouldn't have to keep my own build of Emby I would still have to replace the dlls every time I installed a new version of the Emby server. In addition to this, berkeley has a different db file format so it wouldn't be easy to switch back and forth. I almost went with this solution but out of whim tried something else. The other solution I tried on the day I was about to switch my server over to berkeley. Many in teh forums had suggested using a SSD. I didn't have much faith in it working since I wasn't seeing drive saturation. Not wanting to wait on amazon or invest much time I went to best buy and bought a cheap 250GB SSD. I plugged it into one of the free on-board SATA ports on the mobo. I then renamed AppData\Roaming\Emby-Server, mapped the SSD to the AppData\Roaming\Emby-Server directory and copied over everything to the SSD. I was surprised. This worked. Scan times were reduced to minutes. Clients were responsive and fast. This is a simple, cheap solution that works. Although I recommend throwing the Emby install and most importantly the db files on a SSD to solve your problem IMO it is masking the underlying issue. Sqlite write locks it's db files and if you have a large library that is constantly updated, Sqlite is going to be locking the db files a lot. Using a SSD is compensating for the lack of concurrency by getting those reads and writes done fast. It works but ultimately Emby probably needs to move to another db. Link to comment Share on other sites More sharing options...
tenzen 1 Posted March 16, 2016 Author Share Posted March 16, 2016 Thanks both of you for responses. Turn off hte creation of chapter images during the scan. It's in the Dashboard under Metadata > Advanced > Chapters It isnt enabled on mine. (I thought about custom configurations being an issue and reinstalled emby as default). Currently none of that is enabled and has never been on this installation. <snip>If I swapped out sqlite with berkeleydb scans still took a long time but the library was accessible since berkeley didn't lock the db files. I did this on a VM with access to the drivepool for testing. This worked. Although berkeleydb has a replacement for the sqlite lib (System.Data.SQLite.dll) and I wouldn't have to keep my own build of Emby I would still have to replace the dlls every time I installed a new version of the Emby server. In addition to this, berkeley has a different db file format so it wouldn't be easy to switch back and forth. I almost went with this solution but out of whim tried something else. <snip> I went to best buy and bought a cheap 250GB SSD. I plugged it into one of the free on-board SATA ports on the mobo. I then renamed AppData\Roaming\Emby-Server, mapped the SSD to the AppData\Roaming\Emby-Server directory and copied over everything to the SSD. I was surprised. This worked. Scan times were reduced to minutes. Clients were responsive and fast. This is a simple, cheap solution that works. <snip> My boot drive is a 256Gb SSD on this system. Emby is installed with defaults on the 256SSD already (and has always been since the start of my issues). -- The swapping out of sqlite with berkeleydb is interesting, although I have never done such a thing before. I'll do some googling, but I dont have much faith in this one solving my specific slow library scan issue as it didnt help you in that regard. Also, as I am already on SSD, I dont see any responsiveness issues with clients while the scan is running, my main issue is that with my full library, it is pretty much always running... More thoughts? Link to comment Share on other sites More sharing options...
kjp4756 41 Posted March 16, 2016 Share Posted March 16, 2016 My library is similar size and I have 18 minute scan times as well. I've never been able to figure it out. I am going to try an ssd in the near future. Where is your metadata being stored? If it's not on the ssd, try having it on the ssd. Metadata and emby db on an ssd should provide good results. Link to comment Share on other sites More sharing options...
tenzen 1 Posted March 16, 2016 Author Share Posted March 16, 2016 (edited) My library is similar size and I have 18 minute scan times as well. I've never been able to figure it out. I am going to try an ssd in the near future. Where is your metadata being stored? If it's not on the ssd, try having it on the ssd. Metadata and emby db on an ssd should provide good results. They are on the ssd in the default folders in %appdata% (C: in my case on my SSD) Just for stats, I moved 900 files (1/2 video, 1/2 subtitles) into the folder, and now scans take 40 minutes... Edited March 16, 2016 by tenzen Link to comment Share on other sites More sharing options...
Happy2Play 8304 Posted March 17, 2016 Share Posted March 17, 2016 They are on the ssd in the default folders in %appdata% (C: in my case on my SSD) Just for stats, I moved 900 files (1/2 video, 1/2 subtitles) into the folder, and now scans take 40 minutes... There is not a whole lot of information there but this sounds right, since metadata is not save locally. Any time a media path changes the media is rescanned and given a new itemid (so basically starting from scratch). I use Stablebit Drivepool and library scans take 2-5 minutes. Of course initial scan usually takes about 24hrs. Movies - 5305 Series - 631 Episodes - 40,030 My libraries are added by pool drive letter, (Z:\ServerFolders\Videos\Movies) and substitute path (\\WHS2011\Videos\Movies). I also store metadata with media. Not that anyone wants to but you are on a legacy install MediaBrowser-Server and may want to consider a clean install. Your log doesn't show any of the metadata or cache configurations. Link to comment Share on other sites More sharing options...
swhitmore 781 Posted March 17, 2016 Share Posted March 17, 2016 My libraries are added by pool drive letter, (Z:\ServerFolders\Videos\Movies) and substitute path (\\WHS2011\Videos\Movies). Is this any different they adding the library using the UNC path and not substituting? Link to comment Share on other sites More sharing options...
Happy2Play 8304 Posted March 17, 2016 Share Posted March 17, 2016 (edited) Is this any different they adding the library using the UNC path and not substituting? I could be wrong but wouldn't using UNC locally use more resources? Edited March 17, 2016 by Happy2Play Link to comment Share on other sites More sharing options...
swhitmore 781 Posted March 17, 2016 Share Posted March 17, 2016 I could be wrong but wouldn't using UNC locally use more resources? I'm not sure. Would it? Link to comment Share on other sites More sharing options...
Luke 37119 Posted March 17, 2016 Share Posted March 17, 2016 Yes because the server has to access the files over the network interface. there are a lot of optimizations in windows nowadays to make that perform as close as possible to direct file access but it's not exactly the same. Link to comment Share on other sites More sharing options...
swhitmore 781 Posted March 17, 2016 Share Posted March 17, 2016 Yes because the server has to access the files over the network interface. there are a lot of optimizations in windows nowadays to make that perform as close as possible to direct file access but it's not exactly the same. Wow, I never knew that. So should I rebuild my libraries as above? Use drive letters and path substitution to UNC paths? Link to comment Share on other sites More sharing options...
tenzen 1 Posted March 17, 2016 Author Share Posted March 17, 2016 There is not a whole lot of information there but this sounds right, since metadata is not save locally. Any time a media path changes the media is rescanned and given a new itemid (so basically starting from scratch). I use Stablebit Drivepool and library scans take 2-5 minutes. Of course initial scan usually takes about 24hrs. Movies - 5305 Series - 631 Episodes - 40,030 My libraries are added by pool drive letter, (Z:\ServerFolders\Videos\Movies) and substitute path (\\WHS2011\Videos\Movies). I also store metadata with media. Not that anyone wants to but you are on a legacy install MediaBrowser-Server and may want to consider a clean install. Your log doesn't show any of the metadata or cache configurations. Thanks, this is exactly why I have my thread here, I dont see anything significantly different from my setup and my scan times are much much longer. The install is completely new (removed all folders, everything) as of about 3 weeks ago when I decided to troubleshoot this, so shouldnt having anything left from any old installs. What lead you to think I have old mediabrowser-server components? Is there more I should be considering when removing emby to reinstall it? This is what I did the last time (about 3 weeks ago) - uninstalled the program - manually removed anything left in the %appdata% folder - searched %programdata% for anything left over - downloaded the latest beta, installed, added library It is good to know somebody else who is using drivepool has good results. Thanks Link to comment Share on other sites More sharing options...
ebr 14935 Posted March 17, 2016 Share Posted March 17, 2016 Wow, I never knew that. So should I rebuild my libraries as above? Use drive letters and path substitution to UNC paths? Probably not worth a rebuild but I guess you could try it given the other issues you've been having. 1 Link to comment Share on other sites More sharing options...
tenzen 1 Posted March 19, 2016 Author Share Posted March 19, 2016 Anybody have any other ideas? I did a full wipe and reinstall of emby again and it is currently doing it's initial scan... about 18 hours and still at about 70%... Can somebody maybe point me to the codebase that the scanner uses in Github, I am not a programmer but I can work my way through code pretty well and maybe I can find my issue by seeing in detail what Emby is doing. Link to comment Share on other sites More sharing options...
Ian-Highlander 145 Posted March 19, 2016 Share Posted March 19, 2016 I found that if themoviedb in particular can't find an item the scan takes a long time to time out and carry on. I found removing themoviedb from metadata downloaders for movies and series vastly increased the speed of my scans with no apparent detrimental effect on the metadata. Link to comment Share on other sites More sharing options...
tenzen 1 Posted March 20, 2016 Author Share Posted March 20, 2016 I found that if themoviedb in particular can't find an item the scan takes a long time to time out and carry on. I found removing themoviedb from metadata downloaders for movies and series vastly increased the speed of my scans with no apparent detrimental effect on the metadata. Thanks, but this wasnt it either.. I tried each of the various checkboxes by disabling and enabling.. all was about the same speed... I am trying something else that might turn into a workaround.. I am using some filebot scripts to pre-download metadata into .nfo and all the images. I know I could use Emby for this, but I store all my movies in a single folder, and it is already at thousands of files without an additional bunch of files per movie (images, .nfo, .srt, etc). So Filebot scripting also puts the movies into folders (one per movie)... It isnt a fix, but it might be a workaround by taking the metadata/image gathering out of Emby's hands... I'll post back the results and would love to hear anybody else's suggestions. Link to comment Share on other sites More sharing options...
tenzen 1 Posted March 21, 2016 Author Share Posted March 21, 2016 (edited) Well, I have processed over 1000 videos now, and pre-added the .nfo and the images, each movie in it's own folder. A library scan now takes 34 seconds.. It would be curious to see why it is so slow when after the initial scan, it has basically the same thing (.nfo and images) in a metadata directory on a SSD.. It would also be curious to let Emby store the metadata with the movies (rather than prepare the .nfo and images) and see if it was still fast. Either way, it is clear that something is weird with my setup that causes Emby on my SSD, with metadata in my %appdata% (on the ssd) to perform so poorly. For those who want to use this workaround: 1) Download filebot 2) use command prompt and use the following command to process your movies into individual folders + download images and .nfo - note, due to how the groovy script was written (more on this at the bottom) it names all the nfo files as movie.nfo instead of <moviename>.nfo which emby requires - the script (Alter the paths) filebot -script fn:amc --output "z:\Movies3" --action move -non-strict "z:\movies1\2" --log-file c:\users\lriss\amc.log --def artwork=y forceMovie=f "movieFormat=z:/movies3/{ny}/{n} ({y}) {group} {vf} {af} {vc} [imdbid={imdbid}]{' CD'+pi}{'.'+lang}" - note2: The movieFormat is the magic in the naming. {ny} = Name (year) the rest are my bits that you should alter to however you like your movies named 3) use the gui filebot and drag all the movie folders into it. do it's scans and such and it will then rename the movie.nfo to <moviename>.nfo. - note: be sure to use the same naming as you have in step 2 to not lose all your names. you probably want to do this in a folder emby isnt looking at and move the files in later to prevent library confusion Here is the groovy script link to the points where I think the bug is: https://github.com/filebot/scripts/blob/devel/lib/htpc.groovy#L306 (Note: movieFile = null ) https://github.com/filebot/scripts/blob/devel/lib/htpc.groovy#L311 (Note: hardcoded name 'movie.nfo') Also notice several other hardcoded file names... I am pretty sure that when I get some time, I can fix it and submit a pull request, or maybe fork it and offer back to filbot a variation on the amc script with different naming If anybody has questions on the workaround, let me know. I can whip together some screenshots and a much better guide to this; but to me this is pretty ugly and Emby is plenty powerful enough to really make all this unnecessary if I could just figure out what's up with my setup. Edit: Added --def forceMovie to help matching by amc script. If you are doing the same for series, use the approperiate --def (forceMovie forceAnime forceSeries forceAudio) Lance Edited March 21, 2016 by tenzen 1 Link to comment Share on other sites More sharing options...
kjp4756 41 Posted March 21, 2016 Share Posted March 21, 2016 (edited) Glad this worked for you. I was waiting for your results before I went though the trouble on mine. My library scans have always been quite slow. This weekend I tried an SSD and that didn't change anything. I'm on freenas, not windows. I've seen this behavior on os x, freebsd, linux and windows emby server. Never could figure it out. Thanks! UPDATE: Library scan down from 18 minutes to 23 seconds. Much better! Only had 1 small issue; Several of my movies show as being unwatched when they weren't before. Edited March 21, 2016 by kjp4756 Link to comment Share on other sites More sharing options...
kjp4756 41 Posted March 21, 2016 Share Posted March 21, 2016 After using your workaround last night, I discovered 1 annoying problem. I had a few movies that were incorrectly matched and therefore had the wrong art etc. I couldn't figure out how to fix that using filebot. What I ended up doing was having each of my movies in their own folder. I used filebot gui to do this. Basically I have 'Movies/{ny}/{n} ({y})' I then configured emby to save metadata in with the media files and also to use the nfo writer. Then I think I had to do a full scan which took a while. Then a full metadata refresh under metadata manager. Scantimes are now 23 seconds and I still can use emby to handle the metadata. I think the trick was to have each movie in it's own folder. Previously I had tried saving the metadata in with the media when I only had 1 Movies folder with everything in it and the scan times were horrid. 2 Link to comment Share on other sites More sharing options...
tenzen 1 Posted March 21, 2016 Author Share Posted March 21, 2016 nice update. i like that approach more. Link to comment Share on other sites More sharing options...
Solution Luke 37119 Posted December 13, 2016 Solution Share Posted December 13, 2016 The library scan will be much faster with the next release of Emby Server. Enjoy. 2 Link to comment Share on other sites More sharing options...
ranmachan 0 Posted January 22, 2018 Share Posted January 22, 2018 (edited) After using your workaround last night, I discovered 1 annoying problem. I had a few movies that were incorrectly matched and therefore had the wrong art etc. I couldn't figure out how to fix that using filebot. What I ended up doing was having each of my movies in their own folder. I used filebot gui to do this. Basically I have 'Movies/{ny}/{n} ({y})' I then configured emby to save metadata in with the media files and also to use the nfo writer. Then I think I had to do a full scan which took a while. Then a full metadata refresh under metadata manager. Scantimes are now 23 seconds and I still can use emby to handle the metadata. I think the trick was to have each movie in it's own folder. Previously I had tried saving the metadata in with the media when I only had 1 Movies folder with everything in it and the scan times were horrid. i find therenamer much easier to use. more configurable and customizable in my opinion. www.therenamer.com polls from imdb, tvdb, epsguide, and tv.com. also does movies as well. filebot is nice too but i found this one easier from the getgo Edited January 22, 2018 by ranmachan Link to comment Share on other sites More sharing options...
Fratopolis 62 Posted January 22, 2018 Share Posted January 22, 2018 (edited) EDIT: I removed this comment for a quick command to make a sub for every file and move the file to the folder due to watched status being reset. Edited January 22, 2018 by Fratopolis Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now