Memory Usage

[Thor.2 1]

I am running 4.5.4.0 on Fedora 33

I am running into high of memory usage, and no leveraging of the disk resources

All plays and records are being processed and held in ram till they are released/completed to disk

While this is probably the best for speed and user experience, right up until the ram runs out.

I have 32GB of ram and I can easily consume it all with only a few users and recordings. It would be nice if Emby could be more efficient with its memory usage and leverage the disk for some of the buffering. It should immediately disposing the recording data to disk, not holding it in ram memory cache. Same goes for conversions and transcoding-temp

If the server runs out of memory it slows down to a crawl

I have an SSD drive that is bored, and needs to do some work.

 

[Luke 42078]

Hi there, please attach the emby server log from when this happens. Thanks.

[Thor.2 1]

Sorry for the delay

I rebooted emby when there were no client connected, and no tasks but 29.1GB of RAM used. Now again today there is 13 GB with one client running. I don't see anything in the server log that points towards memory usage, so I am not sure that will be much help. There is definitely some sort of memory leakage

Thanks

embyserver (2).txt

[Q-Droid 989]

Your shot of top is showing 1.5 GB used, emby is using about 1GB resident.  Nearly 30GB is available memory being used as cache because unused memory is wasted memory. Nothing from your screen shot looks out of the ordinary. It just looks like a system with a good amount of file I/O which has been cached.

 

[Thor.2 1]

Yes I am aware of that but it gets locked to Emby, and will not be released for other resources. Then swap is used and the server slows to a crawl. There is no need to hold onto that much cache

[Q-Droid 989]

Not sure what you mean by "locked to Emby". The file buff/cache is a system managed resource, borrowed memory when not needed for other functions. What are you using to monitor the memory consumed by the emby processes?

 

[Thor.2 1]

The problem is magnified when there is a large conversion, I have a daily 3 hour mpeg2.ts recording, that I convert to 8mbps mkv

ffmpeg and emby-server consume all the memory and force the os to use some Swap. With the Fedora OS, swap is setup as ZRAM (memory compression)

The server gets into a bit of trouble when the memory runs out like this. As soon as the conversion completes and finishes the transfer the memory drops back down to normal.

It behaves the same while recording (storing a large amount of data in memory)

Why doesn't the data get flushed from memory as it writes to disk?

[Q-Droid 989]

Are you using a ramdisk for any of the conversion work?

If this new top capture is a snapshot of the system during a conversion there is no indication of high memory usage or swap issue. The swap used is negligible, 9MB out of 4GB. Active mem used (RES) is low and available mem is high. Having some swap usage is normal and not a bad thing.

If the application process is writing to files then it does get written to disk. There is no need to "flush" memory if it's not being requested, that is extra work and it gets reclaimed when needed. 

So what is the actual problem you are trying to resolve? What are the symptoms?

 

[Thor.2 1]

4 minutes ago, Q-Droid said:

Are you using a ramdisk for any of the conversion work?

So what is the actual problem you are trying to resolve? What are the symptoms?

 

No the conversion folder is a hard drive folder

Problems are unresponsiveness if large conversions occur while other users are online. The ZRAM usage will max out as well. I'll try to simulate the situation and update.

I have upgraded memory from 16 to 32 and this seems to be better but it still occurs.

CPU and GPU usage is ok and not a problem

 

[Luke 42078]

Quote

Problems are unresponsiveness if large conversions occur while other users are online.

In that case you may want to configure the schedule of the convert media scheduled task to a time that would make sense for you.

[Thor.2 1]

Hi Luke

I actually just did this last night, as I was in the middle of the SB when it converted the news recording. Brought all my clients down with frozen screens and choppy playback.... I had to stop the convert process. The only resources that were fully consumed were memory.

Is there anyway to demand transcoding on the fly? Can the hdhomerun be defined as to always HW transcode first? The only conversions I do is from the hdhomerun tuners. I would like them to be only available in mpeg4 at 8mbps. The whole record to disk in mpeg2 and then convert to mpeg4 later seems wasteful. I have a P2000 GPU that can easily handle the HW transcode.  

Thanks

 

 

[Luke 42078]

What do you mean demand on the fly? That's what we do for playback, but the convert feature is something that runs in the background.

[Q-Droid 989]

On 2/6/2021 at 3:20 PM, Thor.2 said:

No the conversion folder is a hard drive folder

Problems are unresponsiveness if large conversions occur while other users are online. The ZRAM usage will max out as well. I'll try to simulate the situation and update.

I have upgraded memory from 16 to 32 and this seems to be better but it still occurs.

CPU and GPU usage is ok and not a problem

 

 

I might have an idea of what's going on with your server because it's something I've noticed on mine. It's not a memory issue but one of disk I/O. There's some pretty heavy I/O going on during conversions and the bigger and faster the conversion the more intense the I/O.

For example: When a conversion is doing a remux or audio transcode, being light on processing, the files can be read from the media folder and written to the conversion temp path at or near copy speed. When it's done the job has to copy the converted file back to the media folder, again at copy speed. This is for a single file and depending on file size and storage capability it can take from one to several minutes of heavy I/O for each conversion. The Emby server can become less responsive while the heavy I/O is going on.

When the conversion includes transcoding of video this slows down the I/O from media folder to the temp path so it's less noticeable, usually not a problem. But when the conversion is done the file still does get copied back to the media folder at full speed. So again, depending on size and throughput it can take from 20 seconds to a few minutes per file.

A large batch conversion is going to affect active users. I usually schedule my conversions to run overnight, as Luke suggested above.

A single disk server is going to take a heavy hit with the I/O during conversions. Having media and temp paths on separate devices can help shorten the duration. Large conversions  to/from NAS storage can make for a pretty bad active user experience.

 

[Thor.2 1]

52 minutes ago, Luke said:

What do you mean demand on the fly? That's what we do for playback, but the convert feature is something that runs in the background.

So all Live tv channels on HD home run (ie playback or record) gets transcoded mpeg4 with a max bitrate. 

Then I would only have to use convert on a case by case basis, and not needed for regular recordings.

[Luke 42078]

Yes you can configure the HDHR tuner in Emby to utilize HDHR transcoding, if your tuner supports that.