something limiting my upload to 50Mb/s

[caffeineshock 18]

heyho

from the beginning i would like to mention that i have already searched for hours in this and other forums for solutions and answers, but unfortunately found none.
my setup:
win10 
i3 10100
rtx 3060 (which should not matter)
16gb ram
newest version emby (also tried beta)
emby cache and decode path on a nvme ssd
emby port forwarding / nat on https, ssl certificate. problem occurs with https and also without.
internet bandwidth: 700/200 mbit down/up tested in different ways. (provider: german mnet)

following problem:
in lan, no matter if on localhost or a device in the same network, i reach bandwidths i expect: 200mbit upload from server, which allows me to stream everything in original quality (talking about 1:1 bluray quality 4k hdr).
however, as soon as i leave my lan and stream outside the network over the internet, the maximum upload drops to 50mbit.
i am aware that traffic shaping exists. but can it really happen here?
after all, the traffic is 1: encrypted (i strongly assume that emby encrypts when https is enabled) and 2. i don't use a standard port.

AND if the problem is actually caused by the provider, how can i prove it? sure, i can show it, but... can i log it somehow?
or even better: is it possible to bypass it without using vpn?

thanks a lot

Edited November 3, 2021 by caffeineshock

[Abobader 3464]

Hello caffeineshock,

** This is an auto reply **

Please wait for someone from staff support or our members to reply to you.

It's recommended to provide more info, as it explain in this thread:

Thank you.

Emby Team

[Luke 42078]

Hi there, could be port throttling occurring on either side of the connection (client or server). Can you look into that?

[caffeineshock 18]

already did
I have tried with https and without
different ports for http and https (8096 443 as default, but also 8443 9443 and 1 or 2 more). ofc its still possible... but im pretty sure thats not the case here
different clients (mobile emby, mobile kodi with emby and internet browser on windows and both kodi and emby on android tv)
tried it with lte with telekom and o2, also with dsl with telekom and vodafone
it is definitely the internet of the server
and its still a mystery to me why and how the provider is trafficshaping despite it beeing encrypted (didnt analyze it yet with wireshark. but i might give it a try)
 

Edited November 3, 2021 by caffeineshock

[Carlo 4561]

Find you highest bandwidth movie and stream it remote.  If you can use Emby Theater or a client that will direct play the media.
Does Emby direct play it or transcode it?

If Emby is direct playing it then Emby is not doing anything to limit bitrates. So that's an easy test.  If it does transcode remotely then you need to see why and if there is any mention of bitrate limit.

I guess it would be helpful to understand how you arrived at the 50Mb rate when external.  Is this based on a single stream you tried yourself when remote? If so you need to know how the other network is setup as that could be limiting things.

If you have 4 of these big streams going remotely is your server still only sending 50Mb in total or closer to 200Mb?

What country are you located in?

[Luke 42078]

An even better test is to use the web app to download a file and see what download speeds you get there.

[caffeineshock 18]

9 hours ago, cayars said:

Find you highest bandwidth movie and stream it remote.  If you can use Emby Theater or a client that will direct play the media.
Does Emby direct play it or transcode it?

If Emby is direct playing it then Emby is not doing anything to limit bitrates. So that's an easy test.  If it does transcode remotely then you need to see why and if there is any mention of bitrate limit.

I guess it would be helpful to understand how you arrived at the 50Mb rate when external.  Is this based on a single stream you tried yourself when remote? If so you need to know how the other network is setup as that could be limiting things.

If you have 4 of these big streams going remotely is your server still only sending 50Mb in total or closer to 200Mb?

What country are you located in?

it will direct play it. i almost never use transcode (except when i play something in browser cause most of the browsers do not support the formats).

i know that emby does not limit anything. thats why i mentioned thebehavior inside the lan. i know its some sort of isp-magic happening. the question is how to prevent it (without vpn)

other networks do not limit. 1 playback results in 50mbit. 2 -> 50 mbit. 3 streams -> still 50 mbit 

germany (server isp provider german mnet)

 

i did. downloading file results in, you wont believe me, in 50 mbit

[caffeineshock 18]

im pretty sure its isp limiting the bandwith for this use case
how can we veil what use case this is (without vpn) since https does  not seem to be enough

[Carlo 4561]

Well, Emby can limit global or individual transfers/streams but you the admin need to set that up.
 

2 hours ago, caffeineshock said:

i did. downloading file results in, you wont believe me, in 50 mbit

By any chance did you try two downloads at one time from the same location remotely?

What port are you currently using?  Did you by any chance try using a different port for Emby Server?

I take it you're in Germany?

I'm in the US which makes this tougher but I'd be willing to try a few tests with you depending on a initial test to see what kind of bandwidth we would have between us. But there are often a few things that you can try to see if you can improve speed or work around ISP QOS which may/may not work.

Sometimes ISP only throttle certain ports like 80/443/21 which are common as well as popular ports which could include 8096 & 8920. Picking an obscure port number to use is an easy test for this one. You mentioned a VPN connection which is another good test. You could try a tunneled connection which would not be restrictive to users of your system as they would never know it exists.  One way to do this is using Cloudflare but setup over a tunnel.  Clients all go through Cloudflare to get to your Emby Server which also provides caching as well as additional protection as well but over a tunnel which may help.

If possible, see if you have any friends or relatives using the same ISP as you. Have them try to stream/download from you. This is an interesting test because the packets likely never touch the internet but reside on the ISP network.  Depending on where they have any type of QOS/management in place might bypass that. By this I mean they could be using QOS but not on the interior of the network and only at Internet connection points which would be bypassed on this test.

So there are a few things you can try.

[caffeineshock 18]

yes, like i already mentioned: if i download a file it will download at 50mbit outside the lan. inside the lan it is not limited. no limit inside lan at all

like already mentioned: right now it is 8443 and https is forced. but i also tried different ports, also default port for https and http as well. same situation

ofc i could try 100 different ports but im afraid that this is not the issue -.-

yes, im in germany

im aware of isp beeing able to determine what kind of traffic that is. thats the primary reason why i switched several different ports and why i mention its https cause if it is encrypted AND the port is just a random port (which 8443 is) isp should not be able to tell what kind of data i am sending. ofc if i use vpn the isue will be gone. but i really dont want to use an other service to route my traffic through (which will add costs, ofc)

thats a great idea. i just tried. its the same speed (well right now its not 50, its about 70-80mbit, which is still too slow )
mnet -> telekom: 70 mbit | mnet -> mnet 70 mbit. no difference there

going to talk to the provider now... this just cant be possible...

[ebr 16177]

So the title of this topic is a typo, correct?  Something is limiting you to 50Mb not 50k...?

[caffeineshock 18]

absolutly right

it should be 50mbit/s in upload

Edited November 4, 2021 by caffeineshock

[rbjtech 5284]

Just because the protocol is encrypted, it does not mean the packet formations cannot be analysed and thus 'shaped'.    The fact you are hitting a hard 'stop' suggests your ISP is indeed limiting the stream based on the traffic analysis.

If an 'upload' internet 'speed test' gives you the expected 200Mbit/sec - It would be interesting if you also tried an upload speed test at the same time as the emby video stream is happening - does the speed test now only give you 150Mbit/sec ?

[Q-Droid 989]

Have you tried a test using your public URL from within your LAN (NAT loopback/hairpin)?

 

[caffeineshock 18]

@rbjtech i tried it. when emby uses 50mbit to upload (which seems to be the limit) i get additional 160mbit upload in a speedtest
oh i didnt know it is still possible to inspect the package when its encrypted.

@Q-Droid i cannot reach the server within the lan that way. but that behavior is expected, i guess

 

today i talked to my isp and they told me they dont do such thing as traffic-shaping. i replied that i have a proof. they told me to write an mail with an explanation and screenshots as a proof so the second level support could take a look at it. so far, no reply

Edited November 5, 2021 by caffeineshock

[rbjtech 5284]

46 minutes ago, caffeineshock said:

@rbjtech i tried it. when emby uses 50mbit to upload (which seems to be the limit) i get additional 160mbit upload in a speedtest
oh i didnt know it is still possible to inspect the package when its encrypted.

To be clear here - nobody can inspect the contents of the packet (it's encrypted!) but they can inspect anything in the packet stream which is not such as the source/destination/port/protocol/frequency/nature of the packet etc.  They can, as an example, easily tell a netflix stream vs a torrent (but they cannot see inside the stream).

So the speedtest is reduced by the amount emby is using (approximately) - that's interesting.  I think @cayars touched on this earlier, but if you try a 2nd, parallel session (to another destination, same source(you) - the upload limit is STILL 50Mbit/sec - is that correct ?  does each upload(stream) then take 25Mbit/sec ?

Edited November 5, 2021 by rbjtech

[caffeineshock 18]

@rbjtechexactly. if i stream, the sum of all uploads / bitrate will always be the same: 50mb in this example

 

[GrimReaper 4740]

Just because it wasn't mentioned anywhere (or I might've skipped it): maybe router QoS is playing funky there, if you do have it enabled. Some rule might be interferring. 

Edited November 5, 2021 by GrimReaper
Typo

[Carlo 4561]

Yes, that's a good point.  Might want to turn off anything security or QOS related for a few minutes to test.
Keep track of what you turned off so you can re-enable it again it not the source of the problem.

[caffeineshock 18]

thats a great point. i dont think the router has any sort of qos that needs to be disabled (its pfsense vm using a virtual intel lan on an amd epyc as vm handling all the network stuff) but i will have a look at it. might be some sort of hardware-offloading or simirlar, idk
gonna try it and report 

or do you think the router can contribute to it because of the nat (since there is no need for a nat for an outgoing connection like a speedtest but for an incomming like an emby stream. the data does not cross the router (pfsense) if i do it inside the lan since its the same lan and i dont have any vlans setup -> no nat inside the lan. so.... nat -> router somehow slowing all the traffic down because of it. thats at least an idea. to test it i would need to setup a vlan in my lan, make it untagged before entering the firewall/router/pfsense, enabling firewall between those 2, enabling a nat between them, connect a client to vlan1 and the server to vlan2 and make them conenct through the firewall using a nat. this would be the way to make sure the firewall and the nat is not the issue....)

and that kids is how you create an issue out of nothing  

Edited November 5, 2021 by caffeineshock

[Carlo 4561]

Hard to say without knowing how it's actually all setup.

[rbjtech 5284]

11 hours ago, caffeineshock said:

thats a great point. i dont think the router has any sort of qos that needs to be disabled (its pfsense vm using a virtual intel lan on an amd epyc as vm handling all the network stuff) but i will have a look at it. might be some sort of hardware-offloading or simirlar, idk
gonna try it and report 

or do you think the router can contribute to it because of the nat (since there is no need for a nat for an outgoing connection like a speedtest but for an incomming like an emby stream. the data does not cross the router (pfsense) if i do it inside the lan since its the same lan and i dont have any vlans setup -> no nat inside the lan. so.... nat -> router somehow slowing all the traffic down because of it. thats at least an idea. to test it i would need to setup a vlan in my lan, make it untagged before entering the firewall/router/pfsense, enabling firewall between those 2, enabling a nat between them, connect a client to vlan1 and the server to vlan2 and make them conenct through the firewall using a nat. this would be the way to make sure the firewall and the nat is not the issue....)

and that kids is how you create an issue out of nothing  

Very unlikely, yes DNAT and cross VLAN Routing is gonna play a part in the traffic flow - but even given a single thread on the epyc, this is gonna have more grunt than the average ISP router, so it should not be a problem but you might wanna check things like any IPS (snort etc, not sure what the pf runs) - as this does have the potential to slow things down. 

I run DNAT and cross VLAN Routing and IPS - I've never had a bandwidth issue and that's on a low powered cpu, admittedly with 4 cores, but it never hits more than 30% cpu. (Sophos XG)

 

[caffeineshock 18]

2 hours ago, rbjtech said:

Very unlikely, yes DNAT and cross VLAN Routing is gonna play a part in the traffic flow - but even given a single thread on the epyc, this is gonna have more grunt than the average ISP router, so it should not be a problem but you might wanna check things like any IPS (snort etc, not sure what the pf runs) - as this does have the potential to slow things down. 

I run DNAT and cross VLAN Routing and IPS - I've never had a bandwidth issue and that's on a low powered cpu, admittedly with 4 cores, but it never hits more than 30% cpu. (Sophos XG)

 

yeah i dont think it will be the cause of the issue but i will try to make sure
maybe i will just rip the emby-server out of the dmz and put it right before the pfsense just to make sure it does not cause the issue. 

and then check what GrimReaper said

[Carlo 4561]

Do you have the ability to setup  a simple website on a laptop or something like that?
Put a couple of the test files from here at different bitrates:
https://jell.yfish.us/

Depending on the type of connection you have you may be able to bypass your router and plug that directly into the notebook and let it get the WAN address.
Now you know it's not any of your equipment on a remote test. If not in your DMZ would work.

Fire up performance monitor or some other LAN measurement tool and watch it.

Now while that's streaming from the notebook hit both of these sites:
https://speed.cloudflare.com/
https://testmy.net/latency?addr=youtube.com

This will help you know if part of the problem is latency or jitter as well under load.

[caffeineshock 18]

12 hours ago, cayars said:

Do you have the ability to setup  a simple website on a laptop or something like that?
Put a couple of the test files from here at different bitrates:
https://jell.yfish.us/

Depending on the type of connection you have you may be able to bypass your router and plug that directly into the notebook and let it get the WAN address.
Now you know it's not any of your equipment on a remote test. If not in your DMZ would work.

Fire up performance monitor or some other LAN measurement tool and watch it.

Now while that's streaming from the notebook hit both of these sites:
https://speed.cloudflare.com/
https://testmy.net/latency?addr=youtube.com

This will help you know if part of the problem is latency or jitter as well under load.

can you tell me the point of it?
cause i already did this kind of test as i mentioned that while emby does transfer at 50mbit, i still have 150mbit upload while performing a speedtest

or i just didnt get it