Excessive Bandwidth Usage

[ebr 16176]

13 hours ago, SherlockLin said:

I tried to cut the video to 5 minutes, but the cut did not have the problem I described

Have you tried remuxing the original video?

[SherlockLin 2]

56 minutes ago, ebr said:

Have you tried remuxing the original video?

Remuxed video looks fine.

I have a 100+ megabyte video that replicates this problem, how do I send it to you?

[Luke 42078]

12 minutes ago, SherlockLin said:

Remuxed video looks fine.

I have a 100+ megabyte video that replicates this problem, how do I send it to you?

HI, dropbox or google drive would be great.

[SherlockLin 2]

22 minutes ago, Luke said:

HI, dropbox or google drive would be great.

Sent you a private message with the link.

[Luke 42078]

Hi, are you still having an issue with this?

[SherlockLin 2]

2 hours ago, Luke said:

Hi, are you still having an issue with this?

This problem still exists. Were you able to reproduce the problem with the video I gave you?

[Luke 42078]

On 11/1/2024 at 12:16 AM, SherlockLin said:

This problem still exists. Were you able to reproduce the problem with the video I gave you?

@SherlockLin

Hi, I was not, no. Can you also PM it to @Carlo? I'd like to see if he can reproduce. Thanks.

[Carlo 4561]

@SherlockLinPlease PM me a link for the file.

if you want to you can setup a test account and PM credentials along with a file or two having this issue. Send me the info you would see for the media file when looking at the bottom of the detail screen.  It will have all the basic media information.  If you don't want to do this, it's not a problem, but I it might help me spot something unusual.

Do you know if this also happens if you play it locally (on LAN) or does it only happen with remote clients?

Carlo

[SherlockLin 2]

2 hours ago, Carlo said:

@SherlockLinPlease PM me a link for the file.

if you want to you can setup a test account and PM credentials along with a file or two having this issue. Send me the info you would see for the media file when looking at the bottom of the detail screen.  It will have all the basic media information.  If you don't want to do this, it's not a problem, but I it might help me spot something unusual.

Do you know if this also happens if you play it locally (on LAN) or does it only happen with remote clients?

Carlo

Sent you a private message, the iOS client can reproduce the problem in LAN, the web client cannot.

[Carlo 4561]

I'm able to produce it playing back from your system with Firefox and MS Edge. It doesn't happen every time and not always to the same browser.
I opened Resource Monitor up, had both browsers logged in ready to play. I'd click play on both then watch the bandwidth use of both (direct playing). 

• Sometimes no issue
• Sometimes Firefox uses 2X or 3X bandwidth of Edge
• Sometimes Edge 2X or 3X bandwidth of Firefox

I haven't been able to reproduce locally (yet), but I can on your system.  I see that it does sometimes use more bandwidth, but it happens at different times; It could start 15 seconds one run, not the next 2 times, then 2:40 the next time while direct playing. At least that's what stats for nerds shows in the player.

It would be really helpful to be able to see the server log(s) real-time as I could stop playback when I see it happen, then look at the log to see what it shows. I have the feeling the server logs won't show anything out of the ordinary because I don't think it's actually doing anything to cause this. I hope I'm wrong and the logs show something as that means we can at least identify it or add some additional debug tracking to a test build to gather additional information.  But like I mentioned, I have a feeling it won't show up in the logs as it "feels" like it's environmental, sort of like a TCP packet has a wrong header or has a pattern that triggers a NIC to do something when it shouldn't like applying CRC hashing, applying base encoding, thinking it's getting jumbo packets or fragmented packets.

It's actually kind of hard to look at the actual data in the browser without checking 4 or 5 different things for each get/post/request looking at header, payload for every everything.  Even then some info is still missing such as how we received the packet (perhaps out of order).  I'm going to check a couple things out in the morning on a different computer. I've got a web stress testing tool I can script that emulates a browser but logs everything linearly, which is how I'd like to look at the data.  If it doesn't do what I want, I'm thinking of setting up a virtual machine with a packet capture on the NIC. I could set up a filter to start recording traffic when it identifies a manifest or HLS m3u8 "master" file.  With a few runs like this recorded, I can hopefully file compare or filter each run to see why and why they are different.

It may be a little tricky to figure it out nut as long as we can reproduce it, we can break it down and analyze it. With packet captures, theoretically filters can be applied to remove noise as well as look at the data at the data from a network packet standpoint to see if packets are good, doubled, encoded, etc...

Anyway, I just wanted to let you know I'm seeing what you described and working on a way to identify the increase in bitrate and what's causing it.  The strange thing is that from a playback standpoint, it says it's directly playing and does play smooth, which would seem to indicate the "base" data is there with something else, perhaps adding info to the packets increasing their size or double/triple sending packets.

Carlo

[SherlockLin 2]

1 hour ago, Carlo said:

I'm able to produce it playing back from your system with Firefox and MS Edge. It doesn't happen every time and not always to the same browser.
I opened Resource Monitor up, had both browsers logged in ready to play. I'd click play on both then watch the bandwidth use of both (direct playing). 

• Sometimes no issue
• Sometimes Firefox uses 2X or 3X bandwidth of Edge
• Sometimes Edge 2X or 3X bandwidth of Firefox

I haven't been able to reproduce locally (yet), but I can on your system.  I see that it does sometimes use more bandwidth, but it happens at different times; It could start 15 seconds one run, not the next 2 times, then 2:40 the next time while direct playing. At least that's what stats for nerds shows in the player.
我还不能在本地重现（目前还不能），但我可以在你的系统上重现。 我发现它有时确实会占用更多带宽，但发生的时间不同；可能一次运行 15 秒，下两次就不会了，然后下一次直接播放时又是 2:40。至少在播放器中，stats for nerds 是这么显示的。

It would be really helpful to be able to see the server log(s) real-time as I could stop playback when I see it happen, then look at the log to see what it shows. I have the feeling the server logs won't show anything out of the ordinary because I don't think it's actually doing anything to cause this. I hope I'm wrong and the logs show something as that means we can at least identify it or add some additional debug tracking to a test build to gather additional information.  But like I mentioned, I have a feeling it won't show up in the logs as it "feels" like it's environmental, sort of like a TCP packet has a wrong header or has a pattern that triggers a NIC to do something when it shouldn't like applying CRC hashing, applying base encoding, thinking it's getting jumbo packets or fragmented packets.
如果能实时查看服务器日志，将非常有帮助，因为我可以在看到这种情况时停止播放，然后查看日志，看看它显示了什么。我感觉服务器日志不会显示任何异常情况，因为我认为它实际上没有做任何事情来导致这种情况。我希望我是错的，日志会显示出一些东西，因为这意味着我们至少可以识别出它，或在测试构建中添加一些额外的调试跟踪，以收集更多信息。 但就像我提到的，我感觉日志中不会显示这个问题，因为它 "感觉 "是环境问题，有点像 TCP 数据包有一个错误的报头，或者有一种模式会触发网卡做一些不该做的事情，比如应用 CRC 哈希算法、应用基本编码、认为自己收到了巨型数据包或碎片数据包。

It's actually kind of hard to look at the actual data in the browser without checking 4 or 5 different things for each get/post/request looking at header, payload for every everything.  Even then some info is still missing such as how we received the packet (perhaps out of order).  I'm going to check a couple things out in the morning on a different computer. I've got a web stress testing tool I can script that emulates a browser but logs everything linearly, which is how I'd like to look at the data.  If it doesn't do what I want, I'm thinking of setting up a virtual machine with a packet capture on the NIC. I could set up a filter to start recording traffic when it identifies a manifest or HLS m3u8 "master" file.  With a few runs like this recorded, I can hopefully file compare or filter each run to see why and why they are different.
实际上，如果不对每个 get/post/request 检查 4 或 5 项不同的内容，就很难在浏览器中查看实际数据，因为要查看每项内容的头和有效载荷。 即便如此，还是会丢失一些信息，比如我们是如何收到数据包的（可能顺序错了）。 我打算明天早上在另一台电脑上检查一下。我有一个网络压力测试工具，我可以编写一个脚本来模拟浏览器，但会线性记录所有内容，这也是我想查看数据的方式。 如果它不能满足我的要求，我想在虚拟机上设置网卡数据包捕获功能。我可以设置一个过滤器，在识别到清单或 HLS m3u8 "主 "文件时开始记录流量。 记录了几次这样的运行后，我就可以对每次运行进行文件比较或过滤，看看它们为什么不同。

It may be a little tricky to figure it out nut as long as we can reproduce it, we can break it down and analyze it. With packet captures, theoretically filters can be applied to remove noise as well as look at the data at the data from a network packet standpoint to see if packets are good, doubled, encoded, etc...
只要我们能重现数据，就能对其进行分解和分析。通过捕获数据包，理论上可以使用滤波器去除噪音，并从网络数据包的角度观察数据，查看数据包是否良好、加倍、编码等......

Anyway, I just wanted to let you know I'm seeing what you described and working on a way to identify the increase in bitrate and what's causing it.  The strange thing is that from a playback standpoint, it says it's directly playing and does play smooth, which would seem to indicate the "base" data is there with something else, perhaps adding info to the packets increasing their size or double/triple sending packets.
总之，我只是想让你知道，我看到了你所描述的情况，并正在想办法识别比特率的增加和导致比特率增加的原因。奇怪的是，从播放的角度来看，它说它在直接播放，而且播放得很流畅，这似乎表明 "基本 "数据与其他东西一起存在，也许是在数据包中添加了信息，增加了数据包的大小，或者是双倍/三倍发送数据包。

Carlo

I'll send you a private message with the access to the Log for you to see it in real time, plus I opened the Debug logs.

[rbjtech 5284]

It should be very evident in a simple wireshark data capture on the number of TCP Retransmission packets.     

I don't believe you are going to see anything in the logs - this is all happening at a packet level under the guise of tcp/ip - which is 'handling' the error, but at the expense of bandwidth (retransimisions).

My initial thoughts are a latency timeout issue or buffer mismatch on the client - but without a packet capture - it's impossible to say.

If you have a pcap to share, then I'm happy to take a quick look.

Edited November 18, 2024 by rbjtech

[SherlockLin 2]

1 hour ago, rbjtech said:

It should be very evident in a simple wireshark data capture on the number of TCP Retransmission packets.     

I don't believe you are going to see anything in the logs - this is all happening at a packet level under the guise of tcp/ip - which is 'handling' the error, but at the expense of bandwidth (retransimisions).

My initial thoughts are a latency timeout issue or buffer mismatch on the client - but without a packet capture - it's impossible to say.

If you have a pcap to share, then I'm happy to take a quick look.

Here’s the pcap from the Emby Mac client on a local network. Playback stops after a few seconds because if it runs too long, the capture file will become too large.

emby.pcap.zip

Edited November 18, 2024 by SherlockLin

[rbjtech 5284]

ok tks.

This has one 1 retransmission - so that is not the issue ... but ...

... I'm confused why there are so many ACK's for each HTTP packet ... 

Perhaps the Dev's can wade in here - now they have the pcap, they should be able to do the same.

But it would be extra useful if you also provided a pcap that didn't have the issue - so a 1:1 comparison could be made.

 

 

Edited November 18, 2024 by rbjtech

[SherlockLin 2]

36 minutes ago, rbjtech said:

ok tks.

This has one 1 retransmission - so that is not the issue ... but ...

... I'm confused why there are so many ACK's for each HTTP packet ... 

Perhaps the Dev's can wade in here - now they have the pcap, they should be able to do the same.

But it would be extra useful if you also provided a pcap that didn't have the issue - so a 1:1 comparison could be made.

 

 

Attached are two samples of 30 seconds of playback, the pcap of a normal video using the Mac client and the pcap of the problem video using the Chrome client (Traffic is normal when playing with Chrome)

emby_normal_with_macos_client_30s.pcap.zip emby_normal_with_chrome_30s.pcap.zip

[rbjtech 5284]

14 minutes ago, SherlockLin said:

Attached are two samples of 30 seconds of playback, the pcap of a normal video using the Mac client and the pcap of the problem video using the Chrome client (Traffic is normal when playing with Chrome)

emby_normal_with_macos_client_30s.pcap.zip 34.9 MB · 0 downloads emby_normal_with_chrome_30s.pcap.zip 17.17 MB · 0 downloads

Tks - will take a look later on today.  This should provide a good idea on what is going on - but the simple fact the pcap file is twice as big suggests twice as much traffic/packets are being passed !

[SherlockLin 2]

48 minutes ago, rbjtech said:

Tks - will take a look later on today.  This should provide a good idea on what is going on - but the simple fact the pcap file is twice as big suggests twice as much traffic/packets are being passed !

Both pcap files were grabbed with normal buffered traffic, the macos one is bigger than the chrome one because the video played in the macos client is 1080p, while the video played in chrome is 720p. The abnormal pcap is the 96.56MB one, and it plays the same 720p video, which is played in the MacOS client for less than 5 seconds. 

Edited November 18, 2024 by SherlockLin

[rbjtech 5284]

so assuming I'm comparing the correct files - the 30s mac os capture has zero reset packets, but the 5s original capture has a mass of them...

Other than that - I don't see a huge difference in actual data transfer .. but I'm not really all that familiar on how the HTTP packets for streaming are supposed to be formed.

Maybe the Dev's can chime in ..

[craigdabbs 0]

On 10/10/2024 at 04:37, Luke said:

No, this is just the video player downloading pieces of the file at a time. Very normal.

One thing that can cause excessive bandwidth usage is users running improperly configured reverse proxies where the whole file gets sent back on every single partial request.

Hi luke,

 

Sorry to hijack but im also receiving excessive bandwidth being used from clients from any video file.  From this post you mentioned incorrect reverse proxy causing the issue.  Ive recently setup tailscale on a vps with nginx reverseproxy on it.  This is causing the issue for me, as switching back to a cloudflare tunnel has normal bandwidth again. 

Could i ask what could be misconfigured to be causing this behavior.

 

Many thanks

 

Craig

[Luke 42078]

32 minutes ago, craigdabbs said:

Hi luke,

 

Sorry to hijack but im also receiving excessive bandwidth being used from clients from any video file.  From this post you mentioned incorrect reverse proxy causing the issue.  Ive recently setup tailscale on a vps with nginx reverseproxy on it.  This is causing the issue for me, as switching back to a cloudflare tunnel has normal bandwidth again. 

Could i ask what could be misconfigured to be causing this behavior.

 

Many thanks

 

Craig

Hi, one example could be if range requests aren’t being honored thereby sending back the entire file on every request.

[craigdabbs 0]

how would running through tailscale be causing emby to do that.  Think i might try anothe vpn solution and test.

[Luke 42078]

27 minutes ago, craigdabbs said:

how would running through tailscale be causing emby to do that.  Think i might try anothe vpn solution and test.

An incorrectly configured reverse proxy could cause that.