memory leak in library scans since 7.8

[Statick 9]

as part of my library I have some home movie/photo libraries that are very large esp with many thousands of photos

when running a library scan the memory usage slowly increases and keeps increasing as emby parses over the various folder contents, and this memory usage continues to increase until the scan is completed and then it is freed again

the problem is so bad that I can not complete a full library scan, even if there are no changes made to the library - I have successfully scanned my entire library by individually selecting each sub-folder and scanning those one by one, by doing this I have imported all items and the library looks complete. if I do this across every sub folder, selecting each one and "scan library files" one by one, so they all get queued up, the memory usage usually goes up to around 1-1.5GB then drops again, rises to a similar level then drops, repeating this until it's all completed. it looks to me like the usage increases during each queued up scan then the memory is returned after each one is completed. with no changes made to the library, I can do this across my entire library and it doesn't take long, the entire process is completed in just a few minutes. so as a workaround, this is working for now

but even in this current state where my library is fully imported and there are no changes that need importing or deleteing, if I start a full library scan then it only reaches around 50% until the Emby process is consuming over 4GB of RAM (I have 6GB in my synology) at which point the system has to start using the swap file, which of course is mirrored across all drives so this is extremely slow - Emby becomes very slow and unresponsive (pages take many seconds to load, if they load at all), all the drives start thrashing and the scan slows to a crawl. eventually after hours of constant drive thrashing and the scan barely moving at all, synology DSM reports that it has terminated the process due to excessive memory usage, at this point I think the scan has still only reached about 60%. again there are no library contents that need updating - no video files are being inspected, no scraping is being done, no thumbnails being generated - Emby is just checking for changes and not finding any changes

I previously (on v7.5) was able to complete full library scans on the same library content without any such problem

I updated to 7.8 and started having a number of unrelated problems, eventually I deleted Emby and did a clean reinstall again to 7.8 which solved most of those however this memory problem persists. I've since updated to 7.9 beta and the problem remains there as well

I've attached a server log showing the full library scan taking place, as you can see there's very little in the log because there's no changes to the library content, but you can see some extremely slow response times which are indicative of the problem. not long after the last entry in this log I had no choice but to force stop the process as the server wouldn't respond to shutdown requests and the constant drive thrashing is not good for drive lifespan

 

 

embyserver.txt

Edited April 30 by Statick

[sa2000 22]

You are running emby server 4.8.3 - could you  switch to the current version 4.8.5 - see https://emby.media/download.html

If the problem remains when on 4.8.5, please enable debug logging and provide a fresh log file captured

A fresh launch of the server but running with debug logging enabled - enabled before the server is re-launched and problem problem reproduced and log captured,  Please indicate what the memory usage was at the time

See settings for Logs to enable debug logging

Not much showing in the log without debug

2024-04-29 00:42:43.599 Info TaskManager: Executing Scan media library

2024-04-29 00:42:43.599 Info Server: http/1.1 Response 204 to host1. Time: 3ms. POST http://host2:8096/emby/ScheduledTasks/Running/6330ee8fb4a957f33981f89aa78b030f?X-Emby-Client=Emby Web&X-Emby-Device-Name=Firefox Windows&X-Emby-Device-Id=9f858a49-a835-4b2f-a0b8-11b93c63228d&X-Emby-Client-Version=4.8.3.0&X-Emby-Token=x_secret1_x&X-Emby-Language=en-gb
2024-04-29 01:53:22.481 Info Server: http/1.1 Response 500 to host3. Time: 255ms. GET http://192.168.1.2:8096/emby/system/info/public?format=json
2024-04-29 01:53:22.481 Info Server: http/1.1 Response 500 to host3. Time: 275ms. GET http://192.168.1.2:8096/emby/system/info/public?format=json

 

 

 

Edited April 30 by sa2000

[Statick 9]

heya the log above was from 4.8.3 a couple of days ago, but I've since updated to 4.8.5 and then updated again to the latest beta and the problem remains on all of them. I just had a full log from a clean start to a crashed system saved from a few days ago so that's what I've uploaded here

I'm not at home currently so will get the debug logs over another time

memory usage is very low when not running library scans, usually around 150-200MB, and total system memory usage is under 1GB, as soon as I initiate a library scan the memory usage just goes up and up and up until either the scan completes or the whole system becomes unusable (or DSM terminates the process)

 

Edited April 30 by Statick

[Happy2Play 8361]

@Lukeis there an issue with the failed sync as that is really the only thing shown in that log?

 

[Luke 37272]

3 hours ago, Happy2Play said:

@Lukeis there an issue with the failed sync as that is really the only thing shown in that log?

 

I don't think that's related, but will investigate.

[Luke 37272]

Hi, have you updated to Emby Server 4.8.5? Has that helped?

[Statick 9]

updated to 4.8.5 and then again to 4.9.0.16

no change

 

[Luke 37272]

1 hour ago, Statick said:

updated to 4.8.5 and then again to 4.9.0.16

no change

 

Can you please provide a new server log example? Thanks.

[Statick 9]

okay here's a debug log attached

I didn't want to let the scan eat up all available RAM and start causing drive thrashing again so I let it run for 20 minutes with the debug log running and then cancelled the scan

I monitored the memory usage during this time in the Synology process monitor, here's how much RAM Emby was using for each minute since the server started, and the % of the library scan progress. as you can see the RAM usage just goes up and up for the entire duration of the library scan, upon cancelling the scan the RAM is freed up again and Emby returns to ~200mb usage

 

server start	200mb
01:00 	10% 	320mb
02:00 	11% 	430mb
03:00 	11% 	640mb
04:00 	12% 	820mb
05:00 	13% 	960mb
06:00	14%	1.06gb
07:00	15%	1.18gb
08:00	17%	1.32gb
09:00	18%	1.45gb
10:00	20%	1.59gb
11:00	21%	1.71gb
12:00	22%	1.82gb
13:00	23%	1.93gb
14:00	24%	2.08gb
15:00 	26%	2.24gb
16:00 	26%	2.36gb
17:00	27%	2.48gb
18:00	27%	2.56gb
19:00 	28%	2.67gb
20:00	29%	2.78gb

 

embyserver debug.txt

[Statick 9]

oh one more point - as I'm not sure that this is visible in the server logs - I'm running DSM7 not DSM7.2 as my Synology is quite an old model (DS1815+) and cannot take 7.2. you do have different packages for these two DSM versions so I thought this may be important!

 

 

[Luke 37272]

Hi, thanks for the update. We're looking into it. Thnaks.