Emby leaking memory, process is taking more than 20G of RAM

[rroske79 1]

Experiencing the same issue. Setting up/Testing  a new Emby server, planning to switch from Plex. Running a Hyper-V VM with 4vcpu and 16GB RAM. All other libraries added seemed to scan fine and work great, until I add a Photos library. After a few minutes of scanning, you can watch the RAM usage climb until completely maxed out. I have a fairly large photos library (~150GB), even tried only adding a portion of it (~50GB) and the problem still occurs. I think it may be due to the tagging portion of the photo scanner.

 

Hyper-V VM

Server 2012 R2

16GB  RAM

4vCPU

Emby Server 4.7.2

Edited June 5, 2022 by rroske79

[Luke 42080]

Hi there @rroske79, please attach the emby server log from when this happened:

Thanks.

[rroske79 1]

I am able to reproduce this issue pretty reliably. I restarted the server and added a new Photos library. Within about 20min's, the memory begins to creep up until completely full. The log seems to indicate that it is having trouble reading files, but I have verified full read access with the embyadmin account on the specific files it mentions.

Thanks for any help you can provide!
 

embyserver.txt

[Luke 42080]

We're looking into it. Thanks.

[EZEd 60]

Similar here. I finally got the server to finish the scan on roughly 98K individual photos after 3 days.  I ended up having to do what the OP did, only piecemeal them in to scan a few hundred at a time.  Prior to ver 4.7.2.0 I has able to simply point the database to the folder with multiple sub-folders of photo files.  It would take a while but the scan always finished.  This time (since 4720) the scan would get to a certain point and stall (would not finish - even after waiting days).  That is when I started breaking up the load scans into batches of pictures rather than all at once.

At this point all photos are recognized by the database as as well as all sub-folders (I have them organized by year and have the date taken as the leading characters in the file name).  The only way to reliably view the photos is at the sub-folder level.  When I first enter into the Photos library from the home screen I am presented with thumbs of all of the photos.  But if I scroll to the bottom of the list (image number 98 thousand and change) and hit play on the last couple of images, the screen goes blank and the image never presents.  I can go into the sub-folder where the image exists and click on the image and it almost immediately presents (the same image), so Emby does recognize and present the image but it seems like it is doing some strenuous processing on the main photos screen of thumbs that it never gets to the point of presenting the image.  I can click on the first image in the list of thumbs and that one comes up rather quickly but the further you go down the list the least likely it is to present an image; as if it is going through some extremely large index to get to the image that is/was requested.  I assume it doesn't have this issue in the sub-folder mode because there are fewer images to index, thereby taken fewer resources.

All that I know is that it didn't behave this way prior to 4720.

[MSOBadger 13]

I'm experiencing the same.  I'm in the process of migrating my Emby Server from a Docker on my Unraid box to an Ubuntu VM on a new Proxmox box.  (Emby Server is still running great on the Unraid docker, but I wanted to move to a system with an iGPU for more efficient transcoding.)   My existing and new instances of Emby Server are running 4.7.5.0 (however it has been a good long while since the existing server install has had to perform any kind of large fresh library scan).  All of my media libraries are remaining in their prior locations on my Unraid NAS, and then mounted within the VM.  Unraid and the new Proxmox server are both connected via 10gb NICs. 

My libraries are rather large (~40tb of movies, ~33tm of TV, etc), so the scans take a long time to complete (which is to be expected), but the RAM usage just continues to climb until the VM is pegged at 100%.   It even did it on my music library scan and that's only about half a tb.

I've increased my VM's RAM allocation from 32gb to 48gb but it always just ends up filling whatever's allotted.  One thing I've noticed is that even after a scan is completed, the RAM usage remains wherever it was at when the scan ended.  So the RAM isn't freeing up even after the scan. 

I took a look at the logs and see quite a few errors.  Logs attached.  Let me know if you need more info.  Happy to provide whatever other info I can.

Edit: I see similar errors in the logs that @rroske79 posted above.

embyserver.txt

Edited July 5, 2022 by MSOBadger

[MSOBadger 13]

Some additional info...

I just pointed the new server to the location of my Movies library and ran a scan.  It hummed along until it got to about 90% complete (and did so pretty quickly) and then that's when progress slows to a crawl and the RAM usage starts to climb.  This is the same sort of pattern I saw with my previous library scans (documentaries, sports, docuseries, etc) on the new server install - things go pretty smoothly until the 90% point or so (and the libraries are all quite different in terms of size) and then progress slows way down, the CPU usage increases, and the RAM usage climbs and climbs.   I'ave attached some screenshots from my Proxmox VM summary.

You can see in the CPU and Memory usage timelines how things were sailing along from the start of the scan at ~20:22:00 until ~20:30:00.  The CPU and RAM usage were both low and relatively consistent.   In that time span, the scan had gone from 0% to 86.5%.  Then things went down hill.   As I post this, I've only progressed another ~1.0% in the past hour since that spike occurred and you can see the CPU usage has been over 10x higher than that initial period (sitting at about 11-12% utilization of the 8 cores of i9-11900K that I have passed to the VM), and the RAM usage has crept up to max out all of the 32GB that the VM has allocated.

[Luke 42080]

41 minutes ago, MSOBadger said:

Some additional info...

I just pointed the new server to the location of my Movies library and ran a scan.  It hummed along until it got to about 90% complete (and did so pretty quickly) and then that's when progress slows to a crawl and the RAM usage starts to climb.  This is the same sort of pattern I saw with my previous library scans (documentaries, sports, docuseries, etc) on the new server install - things go pretty smoothly until the 90% point or so (and the libraries are all quite different in terms of size) and then progress slows way down, the CPU usage increases, and the RAM usage climbs and climbs.   I'ave attached some screenshots from my Proxmox VM summary.

You can see in the CPU and Memory usage timelines how things were sailing along from the start of the scan at ~20:22:00 until ~20:30:00.  The CPU and RAM usage were both low and relatively consistent.   In that time span, the scan had gone from 0% to 86.5%.  Then things went down hill.   As I post this, I've only progressed another ~1.0% in the past hour since that spike occurred and you can see the CPU usage has been over 10x higher than that initial period (sitting at about 11-12% utilization of the 8 cores of i9-11900K that I have passed to the VM), and the RAM usage has crept up to max out all of the 32GB that the VM has allocated.

Hi, please try removing these plugins:

2022-07-05 11:24:13.742 Info App: Loading statistics, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null from /var/lib/emby/plugins/Statistics.dll
2022-07-05 11:24:13.743 Info App: Loading VantagePoint, Version=1.0.0.7, Culture=neutral, PublicKeyToken=null from /var/lib/emby/plugins/VantagePoint.dll

Then restart the server and see how that compares. Additionally it appears you enabled nfo saving but did not ensure that emby server has write access to your media folders, so that's probably not helping:

2022-07-05 23:23:12.879 Error ProviderManager: Error in metadata saver
	*** Error Report ***
	Version: 4.7.5.0
	Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
	Operating system: Linux version 5.15.0-40-generic (buildd@lcy02-amd64-047) (gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #43-Ubuntu SMP
	Framework: .NET 6.0.2
	OS/Process: x64/x64
	Runtime: opt/emby-server/system/System.Private.CoreLib.dll
	Processor count: 8
	Data path: /var/lib/emby
	Application path: /opt/emby-server/system
	System.UnauthorizedAccessException: System.UnauthorizedAccessException: Access to the path '/emby/media/videos/cartoons & anime/Golgo 13 (2008)/Season 1/Golgo 13 (2008) - S01E05 - The Superstar's Joint Appearance  (720p BluRay h264 8bit AC3 EN-JP 2.0).nfo' is denied.

I would choose to either turn that off or resolve the permissions problem.

[MSOBadger 13]

Hi @Luke, thanks for the rapid reply and guidance!   

I had completely missed the permissions issue on the shares.  I had the emby user permissions set for rw on the Unraid NAS side, but I hadn't setup the permissions correctly on the new Ubuntu side when mounting those shares.  Thanks for pointing me in the right direction.  I added a 'noperm' tag to my fstab entry that mounts the various media shares.

I also removed the two plugins you referenced (Statistics and VantagePoint).   

I ran a scan of my TV library (which I hadn't yet repointed to the Ubuntu mount point and scanned as of yet).  I kept an eye on the logs as the scan was running and I realized there was another error related to the cache directory.  I had migrated all of the cache files so they were local on the new Ubuntu server, but when I chose the path in Emby's settings I included the "cache" subdirectory in my path so it was effectively looking for '.../cache/cache/...'.  I fixed that mid-scan and those errors all cleared up (after ~12:16:02 timestamp in the attached logs). 

The scan has been running error-free ever since (aside from finding and pointing out one corrupted TV episode file that I had). 

Clearing up the permissions and errors we've discussed appears to have corrected the CPU usage (it's now sitting in the 2%-4% range, about a third of what it was last night).   But the RAM is still being completely pegged.  See screenshots below.

Edit: Maybe that's just to be expected with scans of very large libraries? 

 

embyserver (1).txt

Edited July 6, 2022 by MSOBadger

[Luke 42080]

4 hours ago, MSOBadger said:

Hi @Luke, thanks for the rapid reply and guidance!   

I had completely missed the permissions issue on the shares.  I had the emby user permissions set for rw on the Unraid NAS side, but I hadn't setup the permissions correctly on the new Ubuntu side when mounting those shares.  Thanks for pointing me in the right direction.  I added a 'noperm' tag to my fstab entry that mounts the various media shares.

I also removed the two plugins you referenced (Statistics and VantagePoint).   

I ran a scan of my TV library (which I hadn't yet repointed to the Ubuntu mount point and scanned as of yet).  I kept an eye on the logs as the scan was running and I realized there was another error related to the cache directory.  I had migrated all of the cache files so they were local on the new Ubuntu server, but when I chose the path in Emby's settings I included the "cache" subdirectory in my path so it was effectively looking for '.../cache/cache/...'.  I fixed that mid-scan and those errors all cleared up (after ~12:16:02 timestamp in the attached logs). 

The scan has been running error-free ever since (aside from finding and pointing out one corrupted TV episode file that I had). 

Clearing up the permissions and errors we've discussed appears to have corrected the CPU usage (it's now sitting in the 2%-4% range, about a third of what it was last night).   But the RAM is still being completely pegged.  See screenshots below.

Edit: Maybe that's just to be expected with scans of very large libraries? 

 

embyserver (1).txt 11.28 MB · 1 download

I would look at the server log because there are still permissions errors, but just on different locations:

2022-07-06 12:13:09.570 Error App: Error in TheTVDB
	*** Error Report ***
	Version: 4.7.5.0
	Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
	Operating system: Linux version 5.15.0-40-generic (buildd@lcy02-amd64-047) (gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #43-Ubuntu SMP
	Framework: .NET 6.0.2
	OS/Process: x64/x64
	Runtime: opt/emby-server/system/System.Private.CoreLib.dll
	Processor count: 8
	Data path: /var/lib/emby
	Application path: /opt/emby-server/system
	System.UnauthorizedAccessException: System.UnauthorizedAccessException: Access to the path '/emby/cache/cache/tvdb/281662' is denied.

it looks like your server is still going through the initial scan but I would expect resource consumption to be much lower when it finishes.

[MSOBadger 13]

Thanks @Luke!

I believe those errors were all timestamped before I fixed the Cache path setting that I mentioned.  I made that fix at about 2022-07-06 12:16:xx and these were all before that (the one you quoted was at 2022-07-06 12:13:09).   After that fix, I didn't see any other errors in the log.

My final library scan finished up and after that the resources did indeed all go back to normal levels.  Astounding how much RAM the scans chew up. 

Thanks for the help Luke!  I think I'm good to go.