Initial library scan taking days

[Quiks 15]

Hi,

 

I recently decided to do a brand new install of emby on a new VM after having my database reset on me twice over the last few months. Instead of trying to troubleshoot what may be going wrong, I decided to start over.

 

A little background.

I have ~60 TB of tv/movies

I have allotted 16 GB of ram and 32 virtual cores to this VM.

Windows Server 2012 R2 OS.

Very bare install (just enough to install emby server with .net framework)

All VM data is stored on an SSD

tv/movies are stored on various HDD's with no RAID or anything special

all media locations are added as network paths

Host: 2x E5 2670 processors, 64 GB DDR3, and 512 GB SSD

This installation has started with the latest 3.2.7 install

The data directory of emby has grown from 300 MB to 1000 MB over the course of these 3 days

 

I am now ~72 hours into my initial "scan media library" and it is stating 29.5% completion.

another note: My previous installation had a multi-day "scan media library" when its database randomly reset which was one of the reasons I wanted to do a fresh install.

 

I have restarted emby server after ~1 1/2 days of scanning to see if it got stuck.

Looking at the logs it does appear to be getting some progress done, but this is at such an excruciatingly slow rate which previously did not take this long (many emby versions ago). My hardware has not changed. 

 

I understand that my collection is pretty large, but not even one of my libraries has completed, or even shows signs of being done, in this time 

 

attached, you will find all logs that are in my log folder of this new installation over the past 2 days.

I did see a lot of errors and I'm not sure how to make sense of all of them. Some of them look like tvdb problems where their site might be throttling my requests, but I don't think that's the root issue here. Hopefully someone can take a look and figure out what's going on.

 

debug logging was not checked inside the log area of emby, so I hope my logs will be worth something

 

 

 

server-63624736819.txt

server-63624736820.txt

server-63624738199.txt

server-63624740369.txt

server-63624787200.txt

server-63624862090.txt

server-63624873600.txt

[Luke 42080]

Hi, try removing the kodi sync and autobox set plugins and see if that improves the performance of the library scan.

 

Additionally, if you checked the box on any of your libraries to download images in advance, try unchecking that.

[Quiks 15]

I did have that checked as I wanted it to re-build everything. I will try unchecking that now as well as those two plugins. 

 

How long should I expect the initial library scan to take in this regard? I know its subjective to the number of items, but a ballpark would be appreciated.

[Happy2Play 9781]

Do you have chapter extraction enable also?

 

I know with existing metadata my 38TB server takes about 24hrs to re-build.

[Quiks 15]

I only have chapter extraction enabled for movies. ~900 movies. I ran that task separately and it completed rather quickly.

[Quiks 15]

I restarted emby after making those intitial changes.

 

My library scan started at 26% and has gotten to 28.4% in 1.5 hours. This still seems pretty slow to me.

 

Attached, you'll find the log since the restart.

server-63624915946.txt

Edited March 12, 2017 by Quiks

[Luke 42080]

You still have the two plugins installed. Also, there are lots of http request timeouts and this is undoubtedly slowing down the process.

[Quiks 15]

I don't see these plugins installed.

 

https://i.imgur.com/OAna182.png

 

 

also, are these request timeouts from tvdb or something that's my fault?

Edited March 12, 2017 by Quiks

[Luke 42080]

They are still present in the log you posted, that's why I mentioned it.

[Quiks 15]

Oh, i see

 

I think that the 20 mb log is from before I restarted. I looked at last modified instead of when it was created.

 

Only look at the smaller one. I edited the post just now.

 

Does the smaller one also have these same problems of request timeouts?

Edited March 12, 2017 by Quiks

[Luke 42080]

Yes and because it seems to be happening with every single outgoing request, that points to some kind of networking problem. it's not just metadata related requests, it's also requests to our domain, and others unrelated to metadata. It looks like there is no active internet connection although I'm sure you're about to tell me that there is.

[Quiks 15]

Correct.

 

I checked on the firewall settings of this VM and I have everything set to allowed.

 

One thing to note is that I have a port translation happening on the WAN.

 

I have a public 8097 forwarding to a local 8096, when accessing remotely. This is only until I get this second installation up and running to replace the first one, but I don't think that this would effect this.

The other installation is using the public 8096. Do you think this would cause anything odd to happen from a networking perspective or the way emby handles its requests?

 

On this emby install it's showing its Remote (WAN) access: http://public:8096, but 8096 is not forwarded to that local machine publically, it's forwarded to the original install.

 

I tested internet connectivity from inside the VM and it is currently working. 

Edited March 12, 2017 by Quiks

[Luke 42080]

OK so this is a VM. Yea for some reason it appears the server is not being granted access to the internet.

[Quiks 15]

I'm going to change the public port to 8097 inside the emby install instead of forwarding at the router level.

 

I did test a manual identify on a movie and it seemed to take forever without yielding a result. 

 

I'll report back if anything changes.

[Luke 42080]

I am not sure that will matter because unless I'm misunderstanding, that will only affect incoming traffic.

[Quiks 15]

I'm not sure what would be stopping traffic. It appears that all outgoing traffic is working fine.

[Quiks 15]

I switched the emby public port to 8097 and restarted. I'm able to correctly identify 2 movies. I'm trying the scan media library now to see if the results are different.

[Quiks 15]

I know it's only been a small amount of time, but can you verify in this log if things look different?

 

server-63624925525.txt

[Luke 42080]

yes that's better. I saw an Omdb timeout but that looks specific to Omdb because others are now succeeding.

[Quiks 15]

Interesting. I wonder how something like that would cause such a problem.

 

The only change I made was changing the LAN Port to 8097 and the WAN port to 8097 from within emby.

 

I changed my firewall rule from "8097 public to 8096 private" to "8097 public forwarding to 8097 private".

 

I'll let this go for a bit and see what progress I make. Thanks for the help, Luke.

[Quiks 15]

it looks to be working now. I'm now at 75% completed over these 9 hours or so.

[Luke 42080]

Thanks for reporting back.