Extremely long media library scan and slow unresponsive apps

[jameskitt616 9]

Hello fellow Emby Users,

lately i encounter extreme loading times, when i open emby on web/mobile app/desktop app it takes 10 seconds easy to load all the preview. Same for searching or any click, it takes ages. Starting a video file sometimes can take up to 30-60 seconds. If i go into the Database Settings and vacuum the database on next start up and restart Emby, everything is blazing fast again as it should be. But pretty much as soon as i start a library scan it slows down pretty fast till it's not usable anymore.
A library scan takes +18 hours in this state. The VM is installed on SSDs and runs Ubuntu 22.04 minimal version. I set up a new VM and installed Emby again but i run into the same issues. I am trying to troubleshoot this for a couple of weeks now but i sorta run out of ideas. For storage i use cloud storage mounted via rclone and image extraction off.

The VM has 8GB ram and 8 cores of a i9-10850K CPU assigned which is plenty so there shouldn't be any bottleneck. I sadly can't tell when this issue starts if it came with a new Emby version or not since i've got this for a while and only occasionally bothered to check what could cause it, so i have it for a while. 

If it requires me to test different scenarios and or send more files please let me know.

Thanks for any help in advance!

embyserver.txt

[Luke 39652]

Hi, can you try removing the stats plugin, then restart the server and see if that helps? Thanks.

[jameskitt616 9]

I did, vacuumed the DB file and restarted Emby, started a library scan again and it still became slow. Sadly didn't help.

[jameskitt616 9]

I also noticed after cancelling the library scan it does not become fast again, it stays in that bugged state.

[Luke 39652]

On 6/20/2022 at 2:03 AM, jameskitt616 said:

I also noticed after cancelling the library scan it does not become fast again, it stays in that bugged state.

Hi @jameskitt616 can you please provide a new log file after making the two above changes? Thanks !

[jameskitt616 9]

Sure, here you go!

P.S. I also started *another* test run with another VM, this time with Ubuntu 20.04. The initial Library scan takes a few days so i will report back when it's done. Also i was not sure if this phenomenon instantly occurs or after certain events, i remember something when restoring the server config via the plugin...but totally unsure about this. As said will keep you posted!

embyserver-63791280000.txt

[jameskitt616 9]

Seems like there is an issue with Ubuntu 22.04 and emby. My test with Ubuntu 20.04 seems to be successful, at least so far my first impressions everything seems to be normal again.

[Luke 39652]

Thanks for the update.

[twitchnln 0]

I don't think Ubuntu is the issue, I am seeing similar here running 4.7.5.0 under debian 11. I have a media scan that has been running in excess of 5 days, using 700% of my CPU (AMD FX8320 8-core) and 97% of memory (32GB) according to top. This all began when I recently upgraded to 4.7.5.0 previously this machine had no issues serving Emby to my LAN.  When this happens the time in top process has shown up to 10 min discrepency between screen updates and local system clock.

[jameskitt616 9]

My new Emby install is also starting to slow down again, but only a little bit *yet*. Which is in tolerance. Will keep you posted. 

[jameskitt616 9]

Yes can confirm now. I ran into the same issue again.

After the initial library scan the following took pretty short 30-120 min. Now i am back at 6-10 Hours climbing up per scan basically and the longer Emby is running without reboot the slower it gets.

[Luke 39652]

Can you please provide a new example server log from 4.7.5? Thanks !

[jameskitt616 9]

Sure, here you go.

embyserver.txt

[Luke 39652]

hi @jameskitt616 are you still having an issue with this?

[jameskitt616 9]

Yes, i am back to the original problem, library scan taking almost an day. Everything is not responsive, loading times are very long etc.

So doesnt seem to be a problem with the Ubuntu version.

[Luke 39652]

Hi, that's strange because I don't see anything obvious here to cause that. How many Emby libraries have you created and what is the content type of each one?

[jameskitt616 9]

I have no explanation for this either, i honestly don't know.

As said, my library is fully in cloud storage via RClone mounted.
I've got A Movie Library with one path (+1 Collection Library)
I've got a TV shows library with 6 paths in there.

I also started another test today installing Plex to see if i run into the same issue's there, to verify that the issue is on Emby or on my side.

[Luke 39652]

OK, if you'd like to try the beta server version 4.8.0.6, there are some small improvements to the library scan there.

Then in the next build, which will be 4.8.0.7, there will be some more impactful improvements. Since your media is stored on something that will have slower disk I/O, I'd be interested to know if you see any improvements with it.

[jameskitt616 9]

Will look forward to the update!

I/O never was a issue in the past years, i use this it like that for quite a while.

[Luke 39652]

By the way if you don't use nfo files, then disabling the nfo features on each of your libraries should also help.

[jameskitt616 9]

So the past weeks i investigated a bit more and found out interesting things.

So i use ZFS RaidZ1 on 5 SSDs for my Server. The problem is that i used consumer grade SSDs, and the load i was driving on my Server with all the other stuff hosted maxed out their IOPS which caused them to fail and have incredibly high wearout due the parity configuration needs much much higher calculations than a normal stripe or mirror does.

I changed my entire System Raid setup and for testing i installed Emby on one SSD. Now after a few days it is still making the initial library scan but the rest feels fairly snappy again and as it should be (Well so far at least, but i am positive that it will stay like that).

[jameskitt616 9]

I will report back in a few days to a week if everything still is fast and if the library scans are also back to normal time again.

For now i will say sry for all the inconveniences i caused!

[Luke 39652]

Let us know how you get on. Thanks.

[jameskitt616 9]

Alright, after my last message Emby became massively slow again. Slow library scans, slow UI etc.
At that point i wondered what changed.
Everytime i re-setup Emby and did all the initial library scans and everything seemed to work fine, i finished setting up everything, which included adding an API key for Ombi. Apparently i run into the issue that Ombi is hammering Emby with API calls and slowing it down that massively that it's not usable anymore. I removed the connection between the two of them and my library scan this night went down from +15 hours to 1 hour.

[Luke 39652]

Ah yes. That's actually tougher to spot now in the logs with the reduced http logging, unless debug logging is on. Thanks for the feedback.