High CPU without transcoding. Playback hangs every few minutes.

[G4n0nD0rf 6]

Hi and happy new year everyone!

Two days ago I was watching a movie with Emby when all of the sudden playback hangs for a couple of seconds every few minutes. After a lot of troubleshooting I'm still experiencing the same problem with high bitrate (UHD bluray) files. Never had this problem before.

I run Emby Server in an Unraid docker, which uses a lot of cpu (> 2 cores 100%) when playing such a file even though no transcoding takes place. Lower bitrate files also use more cpu than expected but don't seem to hang. I've noticed DLNA updated to 1.3.3 at about the time my problems started. I don't now if it's related. I disabled DLNA anyway because I don't use it but that doesn't solve my problem. It did lower CPU from about 30% to a few % of a core when nothing was playing.

Any way to find out what's causing the high cpu? Any way to reverse the DLNA update?

[Luke 42080]

 

Hi there, let's look at an example. Please attach the information requested in how to report a media playback issue. Thanks!

 

[G4n0nD0rf 6]

embyserver.txthardware_detection-63839783806.txt

Hi

I embedded the requested server logs. I also send the logs from the app on my Shield TV with the debug option (user HVL). Problems were the same as described above. Playback hanged at:

9:36 CET -> 3:36 AM UTC -5
9:37
9:40
9:41

Thanks for looking into my problem!

[Luke 42080]

Hi.  Can you try sideloading our standard android app on the same device and see how that compares?

https://emby.media/emby-for-android.html

Thanks.

 

[G4n0nD0rf 6]

Same problem. Playback hanged at 21:06, 21:10 and 21:11. No logs appeared in this app version even though app logging was enabled.

embyserver.txt

[Luke 42080]

It could be that this is your issue:

[G4n0nD0rf 6]

There is no transcoding happening. The Shield TV is fully compatible with the used codecs and plays fine in both Dolby Vision and TrueHD Atmos with subtitles. I have been using this setup for 3 years now without any problems, including the same movies that now hang every few minutes.

Is there any way to find out what else can cause the high cpu usage on the server during playback? I've noticed cpu usage stays over 200% when playing high bitrate files, while it only spikes to 200% for a second every 30 seconds with lower bitrate files.

[Q-Droid 989]

Direct play/direct stream should produce single digit CPU usage on average. Are any ffmpeg* logs produced? Or is there something else about your setup that you're not sharing - remote filesystem/cloud storage, encrypted storage, traffic monitoring (IDS/IPS), etc?

Edited January 3, 2024 by Q-Droid

[rbjtech 5284]

If you have access to the command line, then using commands such as 'top' as an included os tool will tell you what processes are using what cpu resources.   

Remember that even for direct play (via http), emby has to read the files from storage, possibly pass through firewalls or containers, convert to http packets and then send - this all takes 'some' cpu - but for it to hit 100% on various cores, I'd say something is definately wrong. 

Have you gone back to basics - and put a test media file (and test library) on the same storage as the server - try to get the system as simple as possible - and start eliminating possible issues one by one.. ?

Edited January 3, 2024 by rbjtech

[G4n0nD0rf 6]

No strange setup that I'm aware off. Embyserver is running as a docker on my Unraid server which provides the file share, as it has been for 3 years now. I did change my server hardware over a month ago so I didn't thought it to be related, but since the problem only seems to occur with movies that have a rather high bitrate like 70+ mbps average I may just not have noticed before with this hardware. I went from a 4 core i5-2500k to a 6 core HT Xeon E5-2620, so the cores themselves maybe slower. I don't now what the cpu usage was on my old hardware, but I never experienced any problems with high bitrate movies before. I wouldn't expect them to be so taxing on my server cpu either.

With the 'top' command on my Unraid server I just see 'EmbyServer' as process eating 200+ % cpu. With the 'top' command in the docker console I only see the process '/system/EmbyServer -programdata /config -ffdetect /bin/ffdetect -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartexitcode 3' using any cpu time.

I tried playing around a bit with CPU pinning for the EmbyServer docker in Unraid. With only one CPU core without HT selected the CPU usage off course stays just under 100%. I expected to see much more playback problems but actually the opposite is true. The playback has been almost perfect this way with just a few stutters in an hour. But this is of course not a great solution, especially when multiple users would use my server at the same time. When selecting either both HT threads of a single core, or a single thread of all 6 cores, the problem became worse again and cpu time raised to almost 200%.

[Luke 42080]

Are you using our official docker container?

https://emby.media/docker-server.html

[Q-Droid 989]

Your current CPU has no graphics capabilities and I'm still not convinced that you're not converting something

Are you not generating any ffmpeg* logs at all?

Sustained high CPU load at higher bitrates and spikes at lower bitrates are a telltale sign of conversion/transcoding going on.

I/O should have a negligible effect on CPU unless you happen to be running with a degraded volume (parity). Network throughput would also have no effect at typical video bitrates (an order of magnitude lower than the NIC's capabilities) unless you happen to be running traffic filters and security applications monitoring LAN traffic.

Encrypted storage/files and compression can add overhead but media files are already compressed and ignored by by those storage options.

 

[G4n0nD0rf 6]

20 hours ago, Luke said:

Are you using our official docker container?

https://emby.media/docker-server.html

I use this one from the Unraid Community Applications store. As far as I can tell it's the same one.

https://registry.hub.docker.com/r/emby/embyserver/

[G4n0nD0rf 6]

19 hours ago, Q-Droid said:

Your current CPU has no graphics capabilities and I'm still not convinced that you're not converting something

Are you not generating any ffmpeg* logs at all?

Sustained high CPU load at higher bitrates and spikes at lower bitrates are a telltale sign of conversion/transcoding going on.

I/O should have a negligible effect on CPU unless you happen to be running with a degraded volume (parity). Network throughput would also have no effect at typical video bitrates (an order of magnitude lower than the NIC's capabilities) unless you happen to be running traffic filters and security applications monitoring LAN traffic.

Encrypted storage/files and compression can add overhead but media files are already compressed and ignored by by those storage options.

 

This should prove it I suppose. Last ffmpeg log is from yesterday when running some other file from pc through Emby Theater.

[Q-Droid 989]

Cool, more info. Thx.

I'll start by stating that with a 12-thread CPU numbers in top will look inflated unless you turn off Irix mode (shift-i) but that's okay. In this case Emby is still tying up 2+ threads for direct play and that's too much.

Now, your box seems to have a lot of overhead and that could be affecting things all around.

I see what I think are at least 2(?) VMs (qemu-system-x86) that are somewhat busy. Are these associated with Emby in any way?

Then there is rslsync and smbd doing something, not terribly busy. But the shfs process looks busier than you might expect.  All of these are tied to I/O operations/services. Is this a steady state? My limited understanding of shfs is that it's FUSE based and for user shares and file caching but I have no idea if it can be tuned or how. If Emby is reading/writing to the same areas and having to go through these services I wonder if that also skews the stats. Concurrency and contention can make things worse.

What jumps out the most is seeing that CPU usage overall (total) for system operations is over 28% while user operations are 24%, a bit lopsided. 

I would follow the advice from @rbjtech and try to isolate Emby and go back to basics. See if you can play media from a storage device/volume/location that is not managed or used by smbd, rslsync and shfs. Maybe stop the VMs temporarily. Bring things back in one-by-one and hopefully that will help identify what makes the load shoot up.

On a bare metal box and even in Docker direct playback should use very little resources. But you have a few layers and side things possibly contributing to the load and maybe amplifying it.

[G4n0nD0rf 6]

I was able to improve performance of the qemu-system-x86 and shfs processes, but it doesn't seem to make any difference. I've also tested Emby playing the same movie without any other docker or VM running. Results stay the same though. System is completely idle when no movie is running with processes only using a few % of cpu time max. With the movie playing, EmbyServer goes over 200% and smbd and shfs also come up to about 25%, which seems normal considering the file is actually hosted by the Unraid server. Playback still hangs very often.

Again with allowing EmbyServer to use only one thread it's %cpu stays under 100% and playback is way better, but not perfect. Very strange. Smbd and shfs stays the same.

[Q-Droid 989]

11 minutes ago, G4n0nD0rf said:

EmbyServer goes over 200% and smbd and shfs also come up to about 25%, which seems normal considering the file is actually hosted by the Unraid server. Playback still hangs very often.

Does CPU usage by smbd and shfs go up when Emby is streaming and down when not? Using 25% of a logical CPU by each still seems high.

[G4n0nD0rf 6]

11 hours ago, Q-Droid said:

Does CPU usage by smbd and shfs go up when Emby is streaming and down when not? Using 25% of a logical CPU by each still seems high.

Yes it does.

[Q-Droid 989]

That to me is a sign that both of those are in the processing path, adding overhead and possibly a bottleneck.

You have a Linux based based server/storage platform (Unraid) and you're running a Linux based application (Emby). Why are you using a Windows storage protocol in this setup?

Have you tried the suggestions to copy sample media and stream from other storage paths/volumes?

I know you said this setup had no problems before but things change over time - data volume, workloads, software, etc. You might have been close to a threshold and something tipped it over. There could be some posts in these forums with best practices for Emby on Unraid, many use it. If you have to use SMB/CIFS then do so for media only. The Emby core files like config, cache, databases, metadata, logs, transcoding paths should be on the most directly accessible and fastest storage you can give them, bypassing pooling, caching and FUSE layers if possible. Your setup may be like this already but it's worth throwing it out there.

 

[G4n0nD0rf 6]

I've done a lot more testing and made some more changes which led me to major improvements. There still is some weird stuff going on leading to higher cpu usage than expected, but at least it seems I got rid of my problems.

I'm pretty sure I used Direct File Access when I set up Emby about 3 years ago. That's why I use SMB shares. Also I already used those already for Kodi and PC access to the shares. Never really got those working with NFS in Unraid.

It's very possible that the high cpu usage already started back when Direct File Access got blocked with Android 11. I never really noticed it before. The playback issues only really start with video bitrates of 70+ mbps. My previous server hardware had slightly higher cpu clockspeeds, so it is indeed very possible i stayed just under the problem treshold back then.

I've made all the necessary improvements to my Unraid setup that I can think of the past few days. They definitely make a difference for my VMs, but not for Emby.

Yesterday I tried a movie with 85 mbps video bitrate. This was completely unplayable on Emby, but plays just fine with Kodi through my SMB shares. CPU usage was about 5% for shfs and 2% for smbd, so I'd say that's the baseline we should be heading for.

But your question why I'm using SMB got me thinking if I could get around that, and it seems I can. Since EmbyServer is running on my storage server and the Emby client isn't using the SMB shares anymore, I am able to point my library folders directly to /mnt/user/ instead of the SMB share. This makes a huge difference! EmbyServer cpu usage is now around 75% and shfs around 15%. Smbd isn't used anymore of course. I haven't noticed any playback problems since.

So it seems I have solved my playback problem, but cpu usage is still way higher than it should be. I also couldn't resist ordering a new CPU in the process with 10 cores and similar clockspeeds like my old server 

Anyway if anyone has got any other ideas to improve my performance I'm glad to try it out. Thanks everyone!

[Q-Droid 989]

Nice, sounds like you've made a good deal of progress.

I ran a test on my server for reference but my highest source is 40 mbps. Streaming this to a Shield TV settles into low single digit (<5% of one core) usage on an 8th gen i3. Doubling that bitrate could very well result in a non-linear increase for CPU usage and I may try that sometime as a test. So yes, that usage still feels high but I'm comparing oranges to lemons... The important thing is that your main problem of high bitrate streaming is improved if not eliminated.

Were you using the optional network path for the libraries or SMB as the primary? The feature is there to define both so that Emby uses the direct path while some clients can use the network share.

 

[rbjtech 5284]

Unless you are running Android 9 on the shield and AndroidTV (which I run on the Shield with Direct File Access), then the only native emby client left which will use SMB directly is Emby Theater (Windows version 100%), Linux version of ET I'm not sure as I don't run it - but presume you may be able to mount a volume via SMB ?

I'm almost sure I've seen this behaviour before - using SMB mounts in Linux leads to extremly chatty communication (from emby) leading to pegging the CPU's.  If you use NFS or some other natively mounted storage on the Linux emby server - I'm fairly certain this resolved the issues.     

I'll see if I can find the thread.