emby server becomes unresponsive after adding 30K photo and videos

[Hzemby 4]

Today I tried to add some family videos and photos to emby server, then the server becomes unresponsive (cannot use dashboard even on local browser) after starting library scan, and Windows Task Manager shows that EmbyServer is conusimg 10GB memory and library scan is stuck at about 92% forever. 

My family videos and photos are about 500GB and have about 30,000 files, is this too much for emby to handle?

Thanks!

[RanmaCanada 497]

It will take a long time for it to parse that amount of data.  What are the specs of your server?

[justinrh 260]

16 hours ago, RanmaCanada said:

It will take a long time for it to parse that amount of data.

So you are saying that after Emby finishes the parsing the server resources and Emby responsiveness will return to normal?

[Hzemby 4]

17 hours ago, RanmaCanada said:

It will take a long time for it to parse that amount of data.  What are the specs of your server?

i7-4770K, 16 GB RAM, WIn 10.

It's been running for around 3 hours, and it is stuck at 92% with the EmbyServer process consuming ~10 GB RAM. Web GUI was not responsive most of the time, sometimes I got lucky to see that videos seemed to be indexed fine, but all photos have no preview. When I exited Emby from the system tray icon, the process EmbyServer didn't exit, I needed to kill it from task manager. I didn't wait further, gave up eventually.  

[RanmaCanada 497]

3 hours ago, justinrh said:

So you are saying that after Emby finishes the parsing the server resources and Emby responsiveness will return to normal?

It should yes, but adding this much content at once can be a complete overload on the system.  OP would be best to add content slowly.  I know when I had to redo my server, it took 2+ days for it to complete all my content as it got hung up on my mp3 library which sits at 345gb.  This is an issue where their lack of patience got the better of them.

[RanmaCanada 497]

1 hour ago, Hzemby said:

i7-4770K, 16 GB RAM, WIn 10.

It's been running for around 3 hours, and it is stuck at 92% with the EmbyServer process consuming ~10 GB RAM. Web GUI was not responsive most of the time, sometimes I got lucky to see that videos seemed to be indexed fine, but all photos have no preview. When I exited Emby from the system tray icon, the process EmbyServer didn't exit, I needed to kill it from task manager. I didn't wait further, gave up eventually.  

Is your Emby exe stored on an ssd or spinning rust?  30 000 files to catalogue will take some time.  Even at one second a piece, that's 500 minutes or over 8 hours.  If you're impatient, instead of doing everything at once, load them slowly.  Probably 1000 files at a time.

[Luke 42083]

There is an issue in the 4.7 server that can cause certain photos to hang the library scan. This has been resolved in the upcoming 4.8 server releaase.

[Hzemby 4]

On 10/29/2023 at 7:59 PM, RanmaCanada said:

Is your Emby exe stored on an ssd or spinning rust?  30 000 files to catalogue will take some time.  Even at one second a piece, that's 500 minutes or over 8 hours.  If you're impatient, instead of doing everything at once, load them slowly.  Probably 1000 files at a time.

It might be my impatience, or might be due to the issue of 4.7 as mentioned by Luke. I think Emby could improve here by showing the files being handled, instead of letting users guess what is happening. As a reference, I tried PhotoPrism to index the same set of photos, it did take a couple of hours to finish, but the UI shows which file is being processed, so user can know it is making progress. 

[Luke 42083]

2 hours ago, Hzemby said:

It might be my impatience, or might be due to the issue of 4.7 as mentioned by Luke. I think Emby could improve here by showing the files being handled, instead of letting users guess what is happening. As a reference, I tried PhotoPrism to index the same set of photos, it did take a couple of hours to finish, but the UI shows which file is being processed, so user can know it is making progress. 

Hi, yes I agree we should add this.