chjohans 32 Posted February 20, 2017 Share Posted February 20, 2017 (edited) This issue has been ongoing for a long time, I'm running Emby on a Intel NUC with Windows 10 and my Movies/TV Series is kept on a very fast Synology NAS. For file copying back and forth between the Intel NUC and my NAS I easily see speeds of around 100 MB/s. My library consists of around 2800 movies and 162 TV Shows each with multiple seasons and episodes. A library scan takes at least 90 minutes every single time and this seems rather high. Can anything be done about this? Edited February 20, 2017 by chjohans Link to comment Share on other sites More sharing options...
Luke 37229 Posted February 20, 2017 Share Posted February 20, 2017 Hi there @@chjohans, we're sorry to hear about your library scan issue. In order for us to best help you, please provide the information requested in how to report a problem. thanks ! Link to comment Share on other sites More sharing options...
chjohans 32 Posted February 20, 2017 Author Share Posted February 20, 2017 Thanks for the quick response. But since the page you are referring to is quite general, I assume that what you need in this case is a server log, correct? So if I start a library scan and let it finish and then post the server log, is that what you need? Link to comment Share on other sites More sharing options...
Luke 37229 Posted February 20, 2017 Share Posted February 20, 2017 Yes. Link to comment Share on other sites More sharing options...
chjohans 32 Posted February 20, 2017 Author Share Posted February 20, 2017 Assuming a server log is what you need, here is one that contains the whole time period a typical library scan was running. server-63623242312.txt Link to comment Share on other sites More sharing options...
chjohans 32 Posted February 20, 2017 Author Share Posted February 20, 2017 Close to realtime support, unreal! Link to comment Share on other sites More sharing options...
Luke 37229 Posted February 20, 2017 Share Posted February 20, 2017 It appears you enabled episode metadata from TheMovieDb. Every additional provider you enable will add time to the library scan. Also, as a test, please uninstall these plugins and see if the issue persists: 2017-02-21 02:51:54.0505 Info App: Loading AutoBoxSets, Version=1.1.0.6, Culture=neutral, PublicKeyToken=null 2017-02-21 02:51:54.0505 Info App: Loading CoverArt, Version=4.0.23.0, Culture=neutral, PublicKeyToken=null 2017-02-21 02:51:54.0505 Info App: Loading Emby.Kodi.SyncQueue, Version=1.4.6257.42260, Culture=neutral, PublicKeyToken=null 2017-02-21 02:51:54.0505 Info App: Loading GenreCleaner, Version=1.0.2.5, Culture=neutral, PublicKeyToken=null 2017-02-21 02:51:54.0505 Info App: Loading MediaBrowser.Plugins.StudioCleaner, Version=3.0.6192.25583, Culture=neutral, PublicKeyToken=null Thanks ! Link to comment Share on other sites More sharing options...
Chillout 85 Posted February 21, 2017 Share Posted February 21, 2017 I've been noticing scans taking longer too.... It used to do a quick 30sec scan to pick up anything new. Now it takes over 5min and acts like it is doing a complete scan each time. Log file attached. server-63623218775.txt Link to comment Share on other sites More sharing options...
chjohans 32 Posted February 21, 2017 Author Share Posted February 21, 2017 Before I made any changes my last library scan took 1h 5m. I then disabled under "Library - Display": - Group movies into collections - Display a collections view to show movie collections - Enable external content in suggestions Uninstalled the following plugins: - Auto Box Sets - CoverArt - GenreCleaner - Studio Cleaner Restarted the server. This had no effect on the library scan, the next library scan took 1h 6m I then reinstalled the following plugins: - CoverArt - GenreCleaner - Studio Cleaner I then disabled "Download artwork and metadata from the internet" for all my libraries and restarted the server Again, not real difference in the tima a library scan takes, the next scan took 1h 3m. I then uninstalled this plugin: Rotten Tomatoes Reviews Restarted the server again, and started a new library scan. Again no real difference in library scan time, this time it took 59m. I then uninstalled this plugin: Emby Kodi Sync Queue Restarted the server again, and started a new library scan. Again, no difference in the library scan time, this time it took 1h 3m. A pattern is that every time I start library scan the "percentage indicator" quickly jumps to 11.something percent done and then it just sits there "forever" and the scan takes 1+ plus no matter settings or plugins it seems. In the end I just set everything back to the way it was before I made any changes as nothing I did had any positive effect on the time a library scan would take. This is kind of ruining my "Emby experience" though. Link to comment Share on other sites More sharing options...
ebr 14958 Posted February 21, 2017 Share Posted February 21, 2017 I then disabled "Download artwork and metadata from the internet" for all my libraries and restarted the server Can you do that again, as well as uninstall all plug-ins then re run the scan and post that log? Thanks. Link to comment Share on other sites More sharing options...
chjohans 32 Posted February 21, 2017 Author Share Posted February 21, 2017 First of all, thanks for helping out with this, much appreciated! My previous library scan before making the changes below took 1h 2m. Disabled the folloving plugins: - Auto Box Sets - CoverArt - Emy Kodi Sync Queue - Genre Cleaner - Rotten Tomatoes Reviews - Studio Cleaner - Trailers These plugins are still installed: - Email Notifications - MediaPortal TV plugin - Pushover Notifications - Server Configuration Backup - Trakt Uninstalling Studio Cleaner was giving me problems, each time I uninstalled it and restarted the server it would just ree-appear. Looks like it didn't get uninstalled, even if the server said it was and asked me to restart. I have previously noticed that this particular plugin give problems, like server crashing when I try to uninstall it or not being uninstalled. In the end I just deleted StudioCleaner.dll from the "plugins" directory and that took care of the issue. I then disabled "Download artwork and metadata from the internet" for all my libraries and restarted the server. Ran a new library scan, again I noticed the "percentage done graph" quickly went to 11.something percent done and then just sat there without moving. This time the library scan took 1h 29s, server log is attached. Link to comment Share on other sites More sharing options...
chjohans 32 Posted February 21, 2017 Author Share Posted February 21, 2017 @@ebr At the moment I'm also unable to add back my deleted plugins. I just get the "spinning wheel" forever when I'm clicking on a plugin in the "catalog". Verified that I have the same issue at a second Emby installation at a different location. Both Emby installation also have the same slooooow library scanm issue. They have exactly trhe same libraries, but on different NAS servers and the installations are totally separate. Link to comment Share on other sites More sharing options...
ebr 14958 Posted February 21, 2017 Share Posted February 21, 2017 Hi. Server log was not attached... Link to comment Share on other sites More sharing options...
chjohans 32 Posted February 21, 2017 Author Share Posted February 21, 2017 Sorry sorry, I was multi-tasking, or trying, something I'm obviously not capable of Log file attached this time. And the "spinning wheel" problem when trying to add back my plugins is now gone. I'm impatient by nature so I just copied the plugin files from another Emby installation, but when I checked just now the plugins can again be installed from the server. The issue lasted for about an hour. server-63623312920.txt Link to comment Share on other sites More sharing options...
ebr 14958 Posted February 21, 2017 Share Posted February 21, 2017 All through the period of that library scan it looks like there are multiple Kodi clients requesting primary images at massive sizes (10,000px wide). What is that..? Link to comment Share on other sites More sharing options...
chjohans 32 Posted February 21, 2017 Author Share Posted February 21, 2017 (edited) I have two Kodi installations, both running Emby for Kodi. One is a standard Windows 10 PC in my living room, running Kodi 17 Krypton with the Emby plugin. The other one is a Raspberry Pi running OSMC and also with the Emby plugin. There nothing unusual about those installations. I have the "Emby for Kodi Sync Queue" plugin installed on my Emby server, and I can only speculate but I assume that a sync between Kody and Emby is somehow triggered by the library scan. I was also watching a episode of a TV Show and browsing a bit through "Emby for Kodi" during the library scan as I got tired of watching the non-moving "progress indicator" From the requests in the log file I see the following as part of the parameters that are passed: MaxWidth=10000&MaxHeight=1000 I assume that's part of a request that want something in return, but not *bigger* than 10000x1000 pixels, at least that's how I read it. I don't know the internal workings of things, and I have downloaded both the Emby server, "Emby for Kodi Sync Queue" and "Emby Kodi plugin" from emby.media But with regards to to slow library scans I think it's irrelevant as I just started to use Kodi the past few months, and I've had this slow library scan issue for much longer than that. Edited February 21, 2017 by chjohans Link to comment Share on other sites More sharing options...
Luke 37229 Posted February 21, 2017 Share Posted February 21, 2017 Yea those requests should be OK. What kind of storage is this on? the library scan does go recursively through every single folder looking for changes files. Link to comment Share on other sites More sharing options...
ebr 14958 Posted February 21, 2017 Share Posted February 21, 2017 Yeah, I just wondered why it was requesting primaries so large. To me, it looks like the whole thing is moving rather slowly (from what I can tell from log message timings) so also wonder if maybe it is hardware bound somehow. Link to comment Share on other sites More sharing options...
chjohans 32 Posted February 22, 2017 Author Share Posted February 22, 2017 Hardware bound? Granted the Intel NUC running the Emby server is not all that powerful, it has an Intel Core i3 @ 1,8 GHz and 8 GB ram. My second installation that has the same slow library scan times has a Intel Core i5 at 1,9 GHz and 16 GB of ram. If this is not enough, what does it take? I see Emby being advertised for NAS devices many times less powerful. What kind of operations do you see that is moving slowly? What exactly does a library scan do? I would think that as a minimum a basic library scan would look for new and changed files and then of course "process those" if it find some. But with few or no changes in the library this process should be pretty quick. But in my case it always takes 1+ hour, consistently no matter if there are any changes in my library or not. So it seems like there must be something else that is very time consuming going on during a library scan? Link to comment Share on other sites More sharing options...
PenkethBoy 2063 Posted February 22, 2017 Share Posted February 22, 2017 Whats the spec for you data store - as if its slow (for some reason) that would slow down the scans and whats the specification of you network - wireless by chance? Link to comment Share on other sites More sharing options...
chjohans 32 Posted February 22, 2017 Author Share Posted February 22, 2017 Thanks, I too wish is was that simple. No wireless connection for this of course. The NUC running Emby has a 1GB wired connection to a managed switch. The "data store" is a Synology DS1815+ that has 2x1GB wired connections to the same switch (using LACP). I easily get speed on or above 100 MB/sec when copying files between the server and the NAS, both ways. One thing tho, I use "Media Center Master" to manage my metadata, partially because my libraries are replicated to another location and partially because I like it that way. But I have allowed metadata to be downloaded by Emby as well, mainly for "Auto Box Sets" to work but also to download additional information, especially "people" (as in actors, directors and the likes). But metadata downloaded by Emby is not saved in the media folders but on a local HD instead. Link to comment Share on other sites More sharing options...
mastrmind11 717 Posted February 22, 2017 Share Posted February 22, 2017 (edited) Try disabling DLNA on the server. The scan of your media is quick, but take a look at the time between the last RESPONSE from any internal request to when the nest DLNA GET request takes place... they range from 3 seconds up to 30 seconds per request. Since the logs imply a single thread, this is adding a shitload of overhead. eg, 2017-02-21 22:45:25.8728 Info HttpServer: HTTP Response 200 to 192.168.3.10. Time: 3ms. http://192.168.3.105:8096/dlna/9c8f00a08d4248ddb27910bff2e2518d/description.xml 2017-02-21 22:45:55.6776 Info HttpServer: HTTP GET http://192.168.3.105:8096/dlna/9c8f00a08d4248ddb27910bff2e2518d/description.xml. This pattern repeats a lot, and appears to get slower as time goes on. Just a thought, but seems to make sense. Edited February 22, 2017 by mastrmind11 Link to comment Share on other sites More sharing options...
PenkethBoy 2063 Posted February 22, 2017 Share Posted February 22, 2017 is it not that the Get happens before the Response? Link to comment Share on other sites More sharing options...
mastrmind11 717 Posted February 22, 2017 Share Posted February 22, 2017 (edited) is it not that the Get happens before the Response? Meaning there is a delay between the previous dlna get's RESPONSE to when the next dlna GET occurs.. I can't determine a definitive pattern either. Edited February 22, 2017 by mastrmind11 Link to comment Share on other sites More sharing options...
PenkethBoy 2063 Posted February 22, 2017 Share Posted February 22, 2017 @@chjohans could it be that you have a race condition with the way you have your metadata managed? If emby detects a change in metadata - from MCM - it has to update its database etc - say info on a new actor which it needs to go to the internet for - this would slow down the overall scan If the changes are common and widespread then this might account for the slow scans - also depends how MCM deals with metadata changes Might be worth running with only one Metadat manager for a few days to see if you scans improve - as it might take a few scans to reach a steady state situation. For me a normal scan is ~20 mins (with no changes) - like you across a 1GB network - although the speed of the network is a minimal factor in this as when on wireless a while ago the speed was no different for the scan and you can see in task manager that Emby has a very small network foot print while scanning. (this is with ~ 5000 video files with Metadata) Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now