Playback - Transfer and Latency Issues

[schamane 20]

On 11/5/2022 at 5:07 AM, archecon said:

At the moment I have identified 2 areas.

1. Internet neutrality is deteriorating. There are countries where distribution of paid content takes up most of the bandwidth. A server on 1 GBit optics - works perfectly.. but on the client side the provider with optics frees up bandwidth to stream 4Mbit.. No it's not a joke... Social networks, commercially distributed video, TV etc. take priority... Especially tragic then is the connection via Single TCP.

This is something what I stumbled over and over again.

Single TCP Connection are a mess in foreign countries, I have very limited Bandwidth because of that.

But when I use a downloadmanager and add more connections it raises a lot.

 

So is there a possibillity, to force the client to establish more then one connection or a bigger buffer at least?

[ebr 16178]

17 hours ago, cptlores said:

But instead of trying your hardest to find an acceptable solution that makes your product better and give your product an competitive edge, you have instead spent time and energy trying to explain why you don't want to support it.

Hi.  I disagree.  We are trying to explain why doing exactly what is being asked here doesn't accomplish the goal you indicated.   So, we're instead, spending time and effort of achieving that goal as mentioned here many times.

 

[softworkz 5066]

7 hours ago, schamane said:

Single TCP Connection are a mess in foreign countries, I have very limited Bandwidth because of that.

But when I use a downloadmanager and add more connections it raises a lot.

This is a misunderstanding of how the internet is working. What's routed over the internet are IP packets, no matter whether it's TCP or UDP and no matter whether the packets are from a single or multiple TCP connections. Routers don't even look at that (except some provider gateways doing DPI, but that's rare, and even then, they're not tracking TCP connections).

The reason why you are getting an advantage when doing multiple parallel downloads is something that is happening at the remote server and their upstream link only:

Assume there's a web server which is serving 100 downloads and you are starting to download from that server, then the server will try to serve all (TCP) connections equally, so you would get 1% of the server's upstream capacity. When you open additional (TCP) connections to the server to perform parallel downloading, you will get a multiple of the capacity - again, because the server tries to serve all TCP connections equally.

But as soon as those IP packets arrive at the first hop on the internet (and from then on, all the way to your own machine), it doesn't play a role anymore whether the packets are from a single or multiple TCP connections. The routers will treat all the packets in the same way, just looking at the IP destination.

[softworkz 5066]

To expand on the above a little further:

There are some Roku firmware versions which try to perform parallel downloading of HLS segments form Emby server by doing range requests. The problem with that is that it doesn't accelerate anything - as long as there's just a single client connected to your Emby server. No matter whether a segment gets downloaded by 3 connections in parallel or with a single connection only - the total download bandwidth will always remain the same. No acceleration!

Let's assume you would have two clients connected to your Emby server, and your internet capacity would be just sufficient to serve both clients for media playback, so each client would get 50% or the upstream bandwidth. 

And now, let's assume that one client would be such Roku client and the other client would be a regular client. The Roku tries to do 3 parallel downloads for a segment, so the result of this will be that the Roku client will get 75% of the available bandwidth and the other client only 25%. This means that the other client might no longer be able to playback properly.

Conclusion: Parallel downloads do not provide any benefits when using Emby as a personal media server and even worse: it can even have an adverse effect on other clients.

[Carlo 4561]

12 hours ago, schamane said:

This is something what I stumbled over and over again.

Single TCP Connection are a mess in foreign countries, I have very limited Bandwidth because of that.

But when I use a downloadmanager and add more connections it raises a lot.

 

So is there a possibillity, to force the client to establish more then one connection or a bigger buffer at least?

Historical support in browsers and communication over TCP.

HTTP/1.1 without pipelining: Each HTTP request over the TCP connection must be responded to before the next request can be made.

HTTP/1.1 with pipelining: Each HTTP request over the TCP connection may be made immediately without waiting for the previous request's response to return. The responses will come back in the same order.

HTTP/2 multiplexing: Each HTTP request over the TCP connection may be made immediately without waiting for the previous response to come back. The responses may come back in any order.

Under HTTP/1.1 you would only download one file at a time on your HTTP/1.1 connection. So your browser downloads the HTML, then it asks for the CSS file(s). When that's returned it asks for the JavaScript file(s). When that's returned it asks for the first image file... etc. HTTP/1.1 is basically synchronous or serial - once you send a request you're stuck until you get a response. This means most of the time the browser is not doing very much, as it has fired off a request, is waiting for a response, then fires off another request, then is waiting for a response... etc.

Pipelining was added to 1.1 but due to buggy proxies and other software made it pretty hard to implement and no mainstream modern browser supports it. HTTP/2 multiplexing achieves what 1.1 pipelining was never able to do as well as additional features like header compression.

So yes, single connections to the same server done synchronously or in serial (one after the other) isn't a good use of bandwidth. Multiplexed (multiple requests to the same server over the same connection) or using multiple requests in parallel can and do make much better use of bandwidth and greatly reduce the latency. This is why modern browsers use 6 to 8 concurrent requests to reduce latency and make better use of the bandwidth available.  Most browsers can manage this setting on their own for the environment it's running it.

The same holds true of downloading multiple files from the same or different servers as it's a better use of resources you're paying for.

What doesn't always work well is using multiple connections to the same file.  Some devices and browsers will try this using range requests with the idea of grabbing upcoming packets before they are needed, but this can have issues.  You can't jump ahead on live broadcasts. A rewind might happen abandoning the packets you tried to grab in the near future. the m3u manifest files might not have the information available yet. If something is transcoding then jumping around requesting 2 or 3 different ranges can be self-defeating.  Last but not least the industry have better ways to handle streaming so the most immediate data is delivered first and is fast enough, so the stream never stalls. Normally this is done by the client who keeps requesting ranges until the buffer is as full as it can be.  On platforms that have pre-generated streaming files already prepared in different bitrates these multiple streams can be present in the m3u manifest and can be chosen by the client to lower bitrate if the buffer can't be kept full or can be increased to get higher resultuion assuming the client an keep the bufer full. So in general there is no "parallel" requests done on single file media files.

[schamane 20]

The strange thing is.

I carry a Stick with me, which is connecting to my home via vpn, so all traffice goes the same way, no peerings of netflix etc. have an effect.

Streaming netflix or prime works like a charm and I get a reliable stream with a good HD quality. Roughly transfering 4 mbit in every direction, my upload is also much bigger, so no bottleneck on that side

Streaming live tv or even a recorded movie via the installed emby client is not working stable even with dropping the quality to 2 Mbit.

I get that live tv maybe an issue cause of peering and the need of a stable connection etc. (even if prime has no issue with that) but streaming a file should work, cause the client should be able to buffer like netflix or prime does, but emby just seems unable to buffer enough, even if I pause for a moment it's not getting better. It's like emby isn't buffering at all.

I would really like to understand that.

And Cayars is right, there is also just a single connection to netflix, looked into it with tcpdump

Edited March 7, 2023 by schamane

[ebr 16178]

3 hours ago, schamane said:

even if I pause for a moment it's not getting better. It's like emby isn't buffering at all.

Hi.  Similar discussion here:

Netflix and the other big streamers are using adaptive streaming techniques as well as highly distributed CDNs to be sure they can get you the content as interruption-free as possible.

[schamane 20]

got that, but I mentioned already, I am using a vpn, which is routing all the traffic over my home connection.

So CDNs are not effecting anything in here.

Adaptive streaming may of course make the stream more robust.

But as one asks in the other Thread, I think a small buffer could fix some of those fluctations of bandwitdth.

I don't get why it is not possible to give us a configurable buffer

[ebr 16178]

15 hours ago, schamane said:

So CDNs are not effecting anything in here

Neither here nor there for our purposes but CDNs are 100% affecting your use of the big streaming platforms...

[schamane 20]

sorry, but I have to disagree.

 

The exit point of my client is my home, cause I am using a 0.0.0.0/0 route on my stick for the vpn and I see the connection on my home router, so it's using the same cdn's as I would watch netflix from my home.

and my emby server is located at home, so it's even having a more "direct connection" then netflix

[Carlo 4561]

7 hours ago, ebr said:

Netflix and the other big streamers are using adaptive streaming techniques as well as highly distributed CDNs to be sure they can get you the content as interruption-free as possible.

Don't forget what is probably their #1 feature to reduce issues with streaming. 
Edge Servers

Companies like Netflix will install as many edge servers as possible directly in large ISP networks as well as peering points. Many people's Netflix streaming traffic never touches the internet at all and basically runs off their ISPs network or peering point.  That lowers the latency as well as saves bandwidth which easily pays for the edge servers.

A caching CDN will be the next best thing as the streaming data will be regionally based and likely collocated at a local peering point. It may not have the full catalog available but will have the popular and most streamed media available. The CDN will normally fetch the complete media file once you've started playing it and due to it's bandwidth it can fetch and have it ready to use from cache very quickly. By the time you've downloaded the first couple segments the CDN will have enough that you'll only be pulling from the CDN's local cache.

@schamane, edge servers and caching CDN servers make a huge difference as it physically puts the storage as close as it gets to the end user often avoiding the internet completely.  This greatly lowers the latency.

Contrary to what you might think you're VPN use is likely making things worse by requiring additional bandwidth as well as adding latency. It takes time to decrypt and encrypt packets as well as take additional time to transmit as the encryption adds on average 10 to 15% more data. The VPN tunnel itself will reduce the amount of data that can be carried as well which means you require more packets sent/received (adds latency). A VPN IPSec Tunnel with an Encrypted IP GRE Tunnel can increase the size by 40%!

https://www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/25885-pmtud-ipfrag.html

A VPN will always reduce available bandwidth, take longer to transmit and have higher latency.

Thos are the 3 things we are trying to reduce!

 

[softworkz 5066]

@cayars - Just from his last message I think I understood what kind of setup he's talking about:

• He's offroad and connecting via VPN to his home
• The Emby server is located at his home
• When he compares Emby with Netflix, in both cases, the traffic is going through VPN to his home..
  • where the Emby server responds directly
  • and Netflix traffic goes to his home as well and from there to the next Netflix edge server

[Carlo 4561]

That could be but Netflix will still have lower latency if Emby has to transcode.

[softworkz 5066]

@schamane @ebr @cayars

I've split this into a separate topic. This has nothing to do with a feature request, not even with H.265.

[softworkz 5066]

23 minutes ago, cayars said:

That could be but Netflix will still have lower latency if Emby has to transcode.

True - the case needs to be looked at to find out..

[schamane 20]

I am aware of reducing my bandwidth slightly via vpn 

 

But Softworkz is right, its exactly my setup.

Transcoding isn't an issue as well in my opinion.

1. It's hardware transcoding

2. I see, the red bar for transcoding is in front of my green bar, so enough buffer on the emby server should be available

 

As far as I can see, netflix is also dropping the used bandwidth at sometime, but it seems to have an bigger buffer which leads to an constant reliable stream.

Thats why I think a configurable buffer would be nice, even for live tv, I would prefer a small time delay instead of a stuttering stream.

And because of having more bandwidth with more connections to the server (download a file with 3 connections instead of one nearly triples my download rate) it would have been nice to have a http2 stream.

I mean I get, that this may harm other users, that's why it would be nice to make it configurable per user or so, but in my example, I am mostly using emby just for myself, not sharing with x users. So me having prio no.1 is absolutly fine

[softworkz 5066]

8 minutes ago, schamane said:

more bandwidth with more connections to the server (download a file with 3 connections instead of one nearly triples my download rate)

This can only be when

1. either there's something very wrong with your VPN
2. or there are other users having transfers to or from your Emby server
3. or you do have other traffic between yours (remote VPN endpoint) and the Emby Server machine

In case 2 or 3, you need to eliminate that traffic in the first place - rather than trying to fight against it via multiple connections. This is not a way for controlling and limiting traffic and surely not something that Emby will implement.

[schamane 20]

1. It' s a usual wireguard vpn, I use for ages without any issues.

Within my country there is no problem at all.

 

2. Even if I am the only one.

 

3. Nope

 

I can also see, that my Upload ist not nearly fully used ever.

 

I guess it's about peering over half of the Globus.

Latency ist jumping around as Well, so in my opinion it's just a bad peering, which affects the connection and establishing more connections Just pushes my bandwidth a bit.

But Netflix seem to be able to handle that much better within it's client.

 

But I guess I have to live with it

 

[softworkz 5066]

5 minutes ago, schamane said:

But Netflix seem to be able to handle that much better within it's client

Which client are you using actually?

[schamane 20]

Hm, it's an firetv 4k Stick, I guess it's updating itself to the latest Version.

But ATM I can't check 

[softworkz 5066]

Is it the same when using a browser as client?

[schamane 20]

That's how I recognized that bandwidth increase with a Download Manager.

 

Cause I can't Download with a fire TV Stick.

Mostly it feels worse with the webclient

[softworkz 5066]

Where does the VPN terminate - in a router in your home network or in a software on the Emby server?

Linux or Windows?

Have you done a network capture (in the home network)?

Do you see packet losses or strongly varying roundtrip times when doing a continuous ping?
(would try both - through the VPN and at the same time outside)

What happens when you connect to the Emby server without VPN (by publishing the port)?

[schamane 20]

Opnsense is doing the wireguard.

So BSD is used as Firewall.  the Desktop Client Linux is used and embyserver ist running in Linux as Well.

and yes, at some times of the day I recognized Packet loss, so I even tried to Route the Traffic through an Server in a Data Center, cause it has very less Hops and no Packet loss, and from this Server to my Home I have no loss as Well, but sadly Not nearly reliable as Netflix, Netflix even is able to handle the Packet loss

[softworkz 5066]

1 minute ago, schamane said:

Netflix even is able to handle the Packet loss

Are they using a UDP based protocol for streaming? I don't know, don't use it..

 

Anyway, packet loss still doesn't explain why multiple TCP connections would provide more throughput.

Have you tried a TCP based VPN protocol for comparison?

2 minutes ago, schamane said:

So BSD is used as Firewall.  the Desktop Client Linux is used and embyserver ist running in Linux as Well.

Is BSD the same physical machine as the emby server?

 

What happens when you connect to the Emby server without VPN (by publishing the port)?