Jump to content

High bitrate media impacting network


Scuffed

Recommended Posts

Scuffed

Hi all,

TLDR; I have (what I think) is a networking bandwidth problem when playing high bitrate media (i.e. HD and 4k remux files 25 Mbps and above). Non-remux content and any other content that I set to play at under 25 Mbps plays perfectly. In theory, everything in my network is fast enough to handle local playback of these files, so I'm not sure what I'm missing.

Emby server setup

  • Beelink U59 Pro running Proxmox 8.1.5
  • Emby running as a Debian 12 LXC using tteck script
  • Dell Poweredge T620 running TrueNAS Scale w/ an NFS share to serve media
  • NFS share mounted in Proxmox using fstab and a mount point in the LXC

Networking setup

  • Two Google Wifi 6E access points, which also act as routers
  • Proxmox server and TrueNAS box hardwired into a TP-Link 1GB switch
  • 1GB NICs installed in Proxmox server and TrueNAS

Clients I have tested with (same results on both)

  • Local streaming only
  • Chromecast w/ Google TV 4K
  • Hard wired laptop w/ Emby Theater connected directly to server via switch

Hardware transcoding is set up properly in Proxmox, and I can confirm it's working well as I can transcode 4K down to 1080p without seeing much increase in CPU. This is why I don't think I have a transcoding issue here.

Why do I think it's a networking issue?

1080p and 4K remux content will play fine on all clients for about 30 seconds or so. Then it starts pausing, resuming, pausing, resuming. After a few minutes, playback stops all together and I get a blue spinning circle on top of the media. When I refresh the player page on desktop, I can no longer reach the server (ERR TIMED OUT). I also can no longer reach my Proxmox GUI page (ERR TIMED OUT). After waiting for a few minutes, the server becomes accessible again and I'm able to get to the Proxmox GUI. Emby also comes back up at the same time. This happens while playing media either on WiFi or while hardwired into my switch.

One piece of info that casts doubt on my diagnosis however: Direct playback of media files via an SMB share and VLC on a wired Windows laptop works flawlessly.

What have I tried so far?

  • Fresh Emby installation on bare metal (not Proxmox)
  • New switch (same model)
  • Many reboots of server, switch, router, and clients

Nothing has changed as far as my setup goes - high bitrate content has never worked since I started my media server journey about six months ago. I'm just now getting around to asking for help, as I've exhausted my abilities as a home labber ;)

Logs attached. Hoping someone can suggest other troubleshooting steps to try, or point out an obvious flaw in my set up. Cheers!

 

hardware_detection-63847051312.txt ffmpeg-transcode-ff265005-2731-4bca-bae6-bd3dee4ba152_1.txt embyserver.txt

Link to comment
Share on other sites

jaycedk

In client/user app settings -> playback.

Try settings it to something else than Auto. 

That sometimes messes up.

Link to comment
Share on other sites

Scuffed
32 minutes ago, jaycedk said:

In client/user app settings -> playback.

Try settings it to something else than Auto. 

That sometimes messes up.

No dice on a hard-wired Windows laptop. Media plays for a few seconds, then stops, then starts again. Haven't tried the Chromecast over Wi-Fi yet.

Link to comment
Share on other sites

rbjtech

Getting the 'media' from the network (via a NAS) is notoriously heavy/chatty on the network - even via NFS.

There are a few threads on people having the same issues and tweaks for buffers etc make all the difference.

Link to comment
Share on other sites

Happy2Play

This is why I don't agree with bitrate double for transcodes.

TranscodeReasons=VideoCodecNotSupported,AudioCodecNotSupported

Item bitrate

"Bitrate":65167130

Converted to

-b:v:0 97750695

14:23:27.499     Side data:
14:23:27.499       cpb: bitrate max/min/avg: 97750695/0/97750695 buffer size: 195501390 vbv_delay: N/A

From a transcoding standpoint these are things you have to consider.  Otherwise you need compatible codecs or a client that can posssibly support them.  And a browser is definately not one of them.

So for this example if you set a playback quality to say 60Mbps or lower on the client do you still have the same issue?

So Emby Theater may be a better choice.

53 minutes ago, Scuffed said:

Direct playback of media files via an SMB share and VLC on a wired Windows laptop works flawlessly.

That would a key difference Direct Play vs Transcoding.

Edited by Happy2Play
Link to comment
Share on other sites

rbjtech
1 hour ago, Scuffed said:

One piece of info that casts doubt on my diagnosis however: Direct playback of media files via an SMB share and VLC on a wired Windows laptop works flawlessly.

There are only a few emby clients that will do REAL direct playback - ie they will pull the data directly from the storage device rather than stream it (via http) from the emby server.   

Emby Theater is likely your only available client left (setup properly, with the optional 'Shared Network Folder' fully accessable) - it will use SMB.   The only other option is the Shield Pro on an old version of Android with the AndroidTV client - that also uses SMB - I'm still clinging to this.. ;)

I suspect this is your issue - you are flooding the network/buffers with traffic.

One very quick way to check this - is to copy a trouble file from the NAS to the local device that runs emby - then point the library to that (as a test) - if it then streams without issues - you know the issue is getting data 'to' emby from the NAS. 

Edited by rbjtech
Link to comment
Share on other sites

Hi, if the bitrate is too high for the connection then I would try lowering it even further using the quality setting. Please see if that helps.

Link to comment
Share on other sites

Lessaj

Your networking equipment sounds more than capable of delivering content at this bitrate, even if it has to also be utilized to deliver the content to your emby server first it should still be capable of several hundred Mbit, far beyond the bitrate of the media. What does it look like if you try to download a file that you're having trouble playing? Does it start fast and then slow down heavily? The fact that you can't even reach your proxmox GUI is suspect. It's running on a mini PC which should be fine to just pass media along. What does the performance of the hypervisor look like in terms of CPU/memory utilization? Any swap usage?

Link to comment
Share on other sites

Scuffed
5 hours ago, rbjtech said:

Getting the 'media' from the network (via a NAS) is notoriously heavy/chatty on the network - even via NFS.

There are a few threads on people having the same issues and tweaks for buffers etc make all the difference.

I did a search and couldn't find anything definitive. Do you have any direct links?

5 hours ago, Happy2Play said:

So for this example if you set a playback quality to say 60Mbps or lower on the client do you still have the same issue?

I do not have the issue when I lower the media playback to 20 Mbps or below. Everything plays just fine at that bitrate, so I know my transcoding is working fine.

5 hours ago, rbjtech said:

There are only a few emby clients that will do REAL direct playback - ie they will pull the data directly from the storage device rather than stream it (via http) from the emby server.   

Emby Theater is likely your only available client left (setup properly, with the optional 'Shared Network Folder' fully accessable) - it will use SMB.   The only other option is the Shield Pro on an old version of Android with the AndroidTV client - that also uses SMB - I'm still clinging to this.. ;)

I suspect this is your issue - you are flooding the network/buffers with traffic.

One very quick way to check this - is to copy a trouble file from the NAS to the local device that runs emby - then point the library to that (as a test) - if it then streams without issues - you know the issue is getting data 'to' emby from the NAS. 

This is disappointing to hear, but great info nonetheless. And that's a great testing idea - I'll give that a try soon.

3 hours ago, Luke said:

Hi, if the bitrate is too high for the connection then I would try lowering it even further using the quality setting. Please see if that helps.

Thanks Luke, and yes - I can confirm lowering the quality to 20 Mbps or lower makes the problem go away entirely.

3 hours ago, Lessaj said:

Your networking equipment sounds more than capable of delivering content at this bitrate, even if it has to also be utilized to deliver the content to your emby server first it should still be capable of several hundred Mbit, far beyond the bitrate of the media. What does it look like if you try to download a file that you're having trouble playing? Does it start fast and then slow down heavily? The fact that you can't even reach your proxmox GUI is suspect. It's running on a mini PC which should be fine to just pass media along. What does the performance of the hypervisor look like in terms of CPU/memory utilization? Any swap usage?

Thanks for the reassurance on my network equipment - I was hoping I wouldn't have to spend more money than I already have :) I have not tried to download a media file from Emby yet, but I'll give that a try and report back. The Proxmox performance is fine, and little to no swap usage is happening on my Emby LXC. I have a few other LXCs running on the same box, and I'm pretty consistently running at 75% RAM and CPU utilization. I like to make the most out of my hardware!

Link to comment
Share on other sites

Scuffed
4 hours ago, Lessaj said:

Your networking equipment sounds more than capable of delivering content at this bitrate, even if it has to also be utilized to deliver the content to your emby server first it should still be capable of several hundred Mbit, far beyond the bitrate of the media. What does it look like if you try to download a file that you're having trouble playing? Does it start fast and then slow down heavily? The fact that you can't even reach your proxmox GUI is suspect. It's running on a mini PC which should be fine to just pass media along. What does the performance of the hypervisor look like in terms of CPU/memory utilization? Any swap usage?

Downloading a file from Emby using the Edge browser on my Windows laptop gives me speeds of around 80 MB/s while wired directly into the switch the Proxmox is connected to. I guess that proves it's not really a network bottleneck, but an issue with how Emby communicates to my NAS (like @rbjtech alluded to earlier)?

Next step is to create a local Emby library on the SSD Proxmox is installed on and try playing a high bitrate file from there, taking my NAS out of the picture. I'll update tomorrow after the files copy over.

  • Thanks 1
Link to comment
Share on other sites

rbjtech

Via SMB, assuming no bottlenecks, you should be getting ~110-112 MB/sec over a 1Gig wired Ethernet connection - so 80 MB/sec does seem a little slow.   That obviously equates to many 100 Mbit streams, so would not appear to be an immediate issue, but I'm wondering why it is not maxing the line capacity.. ?

Some other ideas -

  • Put your emby server > nas on a backhaul network (vlan or isolated physical network) - tbh, you should do this anyway to keep the traffic seperated.  This will also give you options to use Jumbo frames and tweak the NAS for storage use over the network.
  • Run some client/server networks tests such as iperf3 - and run with different packet size settings to ensure the buffers and offloading on the hardware are working well.
  • Run network bandwidth tooling on the server and client - seeing how the streams are created is going to be key - is it 'spiky' with lots of buffer fills (emby>nas(smb) and emby>client(http)) or is it a consistent streams based on media bitrate ? 
  • Network capture with Wireshark etc is also going to allow you to see any hidden issues in your network, retranmissions etc will be quickly identified - and a quick source/destination filter will tell you if the network is functioning as it should.
Edited by rbjtech
Link to comment
Share on other sites

Lessaj

Agreed, 80 MB/s is a little on the slower side but would certainly still be fast enough for the bitrate of the media. The way that direct play and transcoded streams are delivered are different. Direct play is a little harder to work with in the sense that you need to be observing your network traffic to see what the traffic looks like since it won't actually report the web request until the client finishes the request, whereas transcoded streams are delivered in 3 second chunks. It will be a single request for original.mkv (for example) with a reported time based on the duration of playback. From observation, which may not apply to every client I'm mostly referring to a web browser, when you start to play something that requires transcoding the client will quickly request for 10 chunks (30 seconds) and every 3 seconds of playback it will request for another chunk. You can see in the emby server logs how long this request takes. If you have a reverse proxy in front of emby you could also observe this with the correct log format along with the size of each request. In my environment I actually have 2 reverse proxies if accessing externally and the time reported by emby, the first reverse proxy, and the second reverse proxy increases - so emby might report 3ms, the first reverse proxy might report 80ms, and the second might report 180ms just as an example. This is sufficient for 3 second chunks.

Link to comment
Share on other sites

Scuffed

Here are some iperf logs. -u is for UDP, and -b 0 is for unlimited bitrate. 

Client to Emby server:

$ iperf3 -c 192.168.86.48 -u -b 0
Connecting to host 192.168.86.48, port 5201
[  5] local 172.26.35.247 port 38415 connected to 192.168.86.48 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   285 MBytes  2.39 Gbits/sec  206260
[  5]   1.00-2.00   sec   329 MBytes  2.76 Gbits/sec  238140
[  5]   2.00-3.00   sec   330 MBytes  2.77 Gbits/sec  238890
[  5]   3.00-4.00   sec   319 MBytes  2.67 Gbits/sec  230860
[  5]   4.00-5.00   sec   329 MBytes  2.76 Gbits/sec  238390
[  5]   5.00-6.00   sec   332 MBytes  2.78 Gbits/sec  240060
[  5]   6.00-7.00   sec   332 MBytes  2.78 Gbits/sec  240390
[  5]   7.00-8.00   sec   366 MBytes  3.07 Gbits/sec  264780
[  5]   8.00-9.00   sec   557 MBytes  4.68 Gbits/sec  403610
[  5]   9.00-10.00  sec   611 MBytes  5.13 Gbits/sec  442720
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  3.70 GBytes  3.18 Gbits/sec  0.000 ms  0/2744100 (0%)  sender
[  5]   0.00-21.90  sec  1.92 GBytes   753 Mbits/sec  0.017 ms  382379/1805889 (21%)  receiver

 

Emby server to TrueNAS:

# iperf3 -c 192.168.86.104 -u -b 0
Connecting to host 192.168.86.104, port 5201
[  5] local 192.168.86.48 port 55861 connected to 192.168.86.104 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   105 MBytes   881 Mbits/sec  76100  
[  5]   1.00-2.00   sec   108 MBytes   905 Mbits/sec  78120  
[  5]   2.00-3.00   sec   106 MBytes   887 Mbits/sec  76600  
[  5]   3.00-4.00   sec   108 MBytes   910 Mbits/sec  78540  
[  5]   4.00-5.00   sec   106 MBytes   886 Mbits/sec  76490  
[  5]   5.00-6.00   sec   107 MBytes   895 Mbits/sec  77300  
[  5]   6.00-7.00   sec   104 MBytes   871 Mbits/sec  75170  
[  5]   7.00-8.00   sec   111 MBytes   932 Mbits/sec  80420  
[  5]   8.00-9.00   sec   105 MBytes   880 Mbits/sec  75950  
[  5]   9.00-10.00  sec   107 MBytes   894 Mbits/sec  77130  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.04 GBytes   894 Mbits/sec  0.000 ms  0/771820 (0%)  sender
[  5]   0.00-10.00  sec  1.04 GBytes   894 Mbits/sec  0.015 ms  170/771809 (0.022%)  receiver  

 

I think all of this looks...fine? I haven't delved into Wireshark or any other tools yet.

Link to comment
Share on other sites

Lessaj

I'm not sure the client to server seems right, didn't you say your equipment is all 1gig? You should probably be using TCP for the test because all of Emby's traffic is TCP.

Link to comment
Share on other sites

  • 2 weeks later...
Scuffed
On 3/27/2024 at 8:00 PM, Lessaj said:

I'm not sure the client to server seems right, didn't you say your equipment is all 1gig? You should probably be using TCP for the test because all of Emby's traffic is TCP.

Yes, everything I have is 1GB (switches, client and server NICs, cabling). Here's a new iperf using TCP instead:

Client to Emby:

josh@RATMAN:~$ iperf3 -c 192.168.86.48
Connecting to host 192.168.86.48, port 5201
[  5] local 172.24.242.133 port 33102 connected to 192.168.86.48 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  99.6 MBytes   835 Mbits/sec    0   3.15 MBytes
[  5]   1.00-2.00   sec   102 MBytes   860 Mbits/sec    0   3.15 MBytes
[  5]   2.00-3.00   sec  97.5 MBytes   818 Mbits/sec    0   3.15 MBytes
[  5]   3.00-4.00   sec   100 MBytes   839 Mbits/sec    0   3.15 MBytes
[  5]   4.00-5.00   sec   101 MBytes   849 Mbits/sec   91    805 KBytes
[  5]   5.00-6.00   sec  91.2 MBytes   765 Mbits/sec    0    877 KBytes
[  5]   6.00-7.00   sec   106 MBytes   891 Mbits/sec    0    953 KBytes
[  5]   7.00-8.00   sec  90.0 MBytes   755 Mbits/sec    0   1013 KBytes
[  5]   8.00-9.00   sec   109 MBytes   912 Mbits/sec    0   1.06 MBytes
[  5]   9.00-10.00  sec  93.8 MBytes   786 Mbits/sec   85    229 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   991 MBytes   831 Mbits/sec  176             sender
[  5]   0.00-10.00  sec   987 MBytes   828 Mbits/sec                  receiver

Client to TrueNAS

~$ iperf3 -c 192.168.86.104
Connecting to host 192.168.86.104, port 5201
[  5] local 172.24.242.133 port 54944 connected to 192.168.86.104 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   953 Mbits/sec    0   3.16 MBytes
[  5]   1.00-2.00   sec   111 MBytes   933 Mbits/sec    0   3.16 MBytes
[  5]   2.00-3.00   sec   111 MBytes   933 Mbits/sec    0   3.16 MBytes
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec    0   3.16 MBytes
[  5]   4.00-5.00   sec   105 MBytes   881 Mbits/sec    0   3.16 MBytes
[  5]   5.00-6.00   sec  55.0 MBytes   461 Mbits/sec    0   3.16 MBytes
[  5]   6.00-7.00   sec   101 MBytes   849 Mbits/sec    0   3.16 MBytes
[  5]   7.00-8.00   sec   111 MBytes   933 Mbits/sec    0   3.16 MBytes
[  5]   8.00-9.00   sec   111 MBytes   933 Mbits/sec    0   3.16 MBytes
[  5]   9.00-10.00  sec   110 MBytes   923 Mbits/sec    0   3.16 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.02 GBytes   873 Mbits/sec    0             sender
[  5]   0.00-10.03  sec  1.02 GBytes   871 Mbits/sec                  receiver

Emby server to TrueNAS

Quote
root@prox01:~# iperf3 -c 192.168.86.104
Connecting to host 192.168.86.104, port 5201
[  5] local 192.168.86.101 port 58846 connected to 192.168.86.104 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   105 MBytes   878 Mbits/sec    0    355 KBytes       
[  5]   1.00-2.00   sec   108 MBytes   909 Mbits/sec    0    355 KBytes       
[  5]   2.00-3.00   sec   105 MBytes   878 Mbits/sec    0    379 KBytes       
[  5]   3.00-4.00   sec  56.8 MBytes   477 Mbits/sec    0    379 KBytes       
[  5]   4.00-5.00   sec  94.9 MBytes   797 Mbits/sec    0    379 KBytes       
[  5]   5.00-6.00   sec  96.4 MBytes   809 Mbits/sec    0    431 KBytes       
[  5]   6.00-7.00   sec   100 MBytes   839 Mbits/sec    0    431 KBytes       
[  5]   7.00-8.00   sec  98.4 MBytes   825 Mbits/sec    0    431 KBytes       
[  5]   8.00-9.00   sec  84.0 MBytes   704 Mbits/sec    0    431 KBytes       
[  5]   9.00-10.00  sec   101 MBytes   845 Mbits/sec    0    628 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   949 MBytes   796 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   946 MBytes   793 Mbits/sec                  receiver

 

 

Link to comment
Share on other sites

Lessaj

Okay those results are certainly perfectly respectable, maybe just a smidge on the low side because even though it's 1gig you won't actually get 1gig you'll usually top around 940 to 950 at most so seeing at least 900 would be a bit better. The client to the server had some retries which can happen with wireless clients but I'm a little surprised to see that from a LAN connection. Maybe you need to run the tests for a bit longer to see if there are periods of retries, maybe when there's other traffic occuring through the routing/switching device(s), can do -t 60 or higher to make it run longer. But again these are still speeds several times over the bitrate of the media so I'm not sure that the networking equipment is the problem here.

I'm still concerned that you are not able to reach the proxmox GUI when this occurs. This has me leaning towards the mini PC. Are you able to run a continuous ping against the device while introducing the problem? This may also provide a clue while streaming the high bitrate content if ping times jump rapidly. Additionally if possible, have an SSH connection going with some kind of process monitor (top, htop, glances, whatever you like) to monitor for anything like sudden spikes in memory/cpu or even if the session just starts being laggy or downright unresponsive. It may also be worth trying a telnet to the proxmui web GUI as well - if you can reach the web GUI port via telnet that suggests that a web GUI component is not responding thus causing the page to not load, but is otherwise accepting the port connection request.

Edited by Lessaj
  • Like 1
Link to comment
Share on other sites

rbjtech
On 27/03/2024 at 02:38, Scuffed said:

Next step is to create a local Emby library on the SSD Proxmox is installed on and try playing a high bitrate file from there, taking my NAS out of the picture. I'll update tomorrow after the files copy over.

Did you try this ?

Raw Network performance should not be an issue here - but we need to eliminate the stack for the moment and ensure emby is capable of using the files from a local source. (which it is, it can stream ~300Mbit files without issue from previous experiments in this area)

 

Edited by rbjtech
Link to comment
Share on other sites

Scuffed
On 4/9/2024 at 2:19 AM, rbjtech said:

Did you try this ?

Raw Network performance should not be an issue here - but we need to eliminate the stack for the moment and ensure emby is capable of using the files from a local source. (which it is, it can stream ~300Mbit files without issue from previous experiments in this area)

 

Funny thing with that - I've tried many times to upload the file over SFTP (WinSCP) with no luck. It gets about 30% uploaded and then the speed drops to X KB/s. I've left it overnight a few times, but the file never uploads. This is uploading a local 4K video file from my laptop (Windows NVMe) to the Proxmox host (NVMe SSD) over ethernet. I can probably transfer the file via USB, but I haven't got around to that yet. Not sure if that gives us any clues into the larger issue, but I figured I'd mention it.

Link to comment
Share on other sites

rbjtech
11 hours ago, Scuffed said:

Funny thing with that - I've tried many times to upload the file over SFTP (WinSCP) with no luck. It gets about 30% uploaded and then the speed drops to X KB/s. I've left it overnight a few times, but the file never uploads. This is uploading a local 4K video file from my laptop (Windows NVMe) to the Proxmox host (NVMe SSD) over ethernet. I can probably transfer the file via USB, but I haven't got around to that yet. Not sure if that gives us any clues into the larger issue, but I figured I'd mention it.

 If SFTP cannot transfer the file and it essentially hangs - then something is badly wrong and I suspect it is hitting exactly the same issues as your original problem.

I would go back to the real basics - put two basic windows machines on the same network switch path - and try again using smb/sftp etc - watch the transfer in perfmon - if it's consistent, then you know the issue is something in Proxmox etc.    If you recreate the issue again using Windows - then it's 100% the switches.    If it is the switches, the run in isolation - ie own vlan etc as something like a broadcast storm could be happening and taking out your entire network. 

Just a process of elimination I'm afraid. 

Edited by rbjtech
  • Agree 1
Link to comment
Share on other sites

  • 2 weeks later...

Thanks for the ideas all. I've been trying all sorts of things, but keep coming back to Kodi because it just works (especially now that v21 supports Dolby Vision in MKV files). I'm moving house soon, so my networking set up will be entirely different, so I might give Emby another try then. I'll report back if I'm able to figure out what my issue is.

Edited by Scuffed
  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...