Question about streaming api

[enqbcvqw 2]

This looks like bad behavior. I'm trying to set emby up in front of a caching CDN to improve performance for some of my faraway users. When I introduce the CDN, it slices each request into 8 MB chunks and requests each one individually ahead of the client requesting that portion of the data.

 

 

The request comes from the CDN into an Nginx which then forwards to emby on its internal port. Here are the request/responses. I've obscured some personal stuff:

 

 

Request on external interface to nginx:

T x.x.x.x.:51915 -> x.x.x.x:80 [AP]
  GET /emby/videos/1761651/stream.mp4?api_key=a7e06082a2e7446dafde7af9a83daef6 HTTP/1.1
  ..CDN-ServerId: 551..CDN-ServerZone: MI..Accept-Encoding: identity..Via: BunnyCDN..Ho
  st: x.x.x.x..Range: bytes=0-8388607..CDN-Host: x.x.x.x..CDN-Mob
  ileDevice: false..CDN-RequestCountryCode: US..CDN-RequestUrl: https://x.x.x.x
/emby/videos/1761651/stream.mp4..CDN-GzipEnabled: true..CDN-BrotliEnabled: fals
  e..X-Real-IP: 108.162.210.135..X-Forwarded-For: 2001:4878:8151:800:d1fd:67cf:4740:d22
  2, 108.162.210.135..BunnyCDN-LBKey: j0YiR5gUfYiZmQrWSLV6opVewPUZFwJD21K1W9dkKWg4EnYSj
  bS5c9..CF-RAY: 5638793da575d4f9-MIA..X-Forwarded-Proto: https..CF-Visitor: {"scheme":
  "https"}..CF-EW-Via: 15..CDN-Loop: 1..Accept-Language: en-US,en;q=0.9,es;q=0.8..Accep
  t: */*..Cache-Control: no-cache..Referer: http://app.emby.media/..User-Agent: Mozilla
  /5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrom
  e/74.0.3729.169 Safari/537.36..CF-Connecting-IP: 2001:4878:8151:800:d1fd:67cf:4740:d2
  22..dnt: 1..origin: http://app.emby.media..pragma: no-cache..sec-fetch-dest: video..s
  ec-fetch-mode: cors..sec-fetch-site: cross-site..cf-worker: x.x.x.x....

Request on internal interface from nginx to emby:

  GET /emby/videos/1761651/stream.mp4?api_key=a7e06082a2e7446dafde7af9a83daef6 HTTP/1.
  1..Range: bytes=0-8388607..Host: emby_backend_cache..Connection: close..CDN-ServerId
  : 551..CDN-ServerZone: MI..Accept-Encoding: identity..Via: BunnyCDN..CDN-Host: x.x.x.x.

..CDN-MobileDevice: false..CDN-RequestCountryCode: US..CDN-RequestUr
  l: https://x.x.x.x./emby/videos/1761651/stream.mp4..CDN-GzipEnabled: tr
  ue..CDN-BrotliEnabled: false..X-Real-IP: 108.162.210.135..X-Forwarded-For: 2001:4878
  :8151:800:d1fd:67cf:4740:d222, 108.162.210.135..BunnyCDN-LBKey: j0YiR5gUfYiZmQrWSLV6
  opVewPUZFwJD21K1W9dkKWg4EnYSjbS5c9..CF-RAY: 5638793da575d4f9-MIA..X-Forwarded-Proto:
   https..CF-Visitor: {"scheme":"https"}..CF-EW-Via: 15..CDN-Loop: 1..Accept-Language:
   en-US,en;q=0.9,es;q=0.8..Accept: */*..Cache-Control: no-cache..Referer: http://app.
  emby.media/..User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit
  /537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36..CF-Connecting-IP: 20
  01:4878:8151:800:d1fd:67cf:4740:d222..dnt: 1..origin: http://app.emby.media..pragma:
   no-cache..sec-fetch-dest: video..sec-fetch-mode: cors..sec-fetch-site: cross-site..
  cf-worker: x.x.x.x.....

Response from emby (see source port):

  HTTP/1.1 200 OK..Access-Control-Allow-Headers: Accept, Accept-Language, Authorizatio
  n, Cache-Control, Content-Disposition, Content-Encoding, Content-Language, Content-L
  ength, Content-MD5, Content-Range, Content-Type, Date, Host, If-Match, If-Modified-S
  ince, If-None-Match, If-Unmodified-Since, Origin, OriginToken, Pragma, Range, Slug,
  Transfer-Encoding, Want-Digest, X-MediaBrowser-Token, X-Emby-Token, X-Emby-Authoriza
  tion..Access-Control-Allow-Methods: GET, POST, PUT, DELETE, PATCH, OPTIONS..Access-C
  ontrol-Allow-Origin: *..Server: UPnP/1.0 DLNADOC/1.50..Content-Type: video/mp4..Acce
  pt-Ranges: none..Date: Tue, 11 Feb 2020 18:44:22 GMT..Connection: Close..Transfer-En
  coding: Chunked

I was able to repro this manually using curl:

curl -IXGET 'http://localhost:8091/emby/videos/1761651/stream.mp4?api_key=a7e06082a2e7446dafde7af9a83daef6' -H "Range: bytes=0-8388607"
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Accept, Accept-Language, Authorization, Cache-Control, Content-Disposition, Content-Encoding, Content-Language, Content-Length, Content-MD5, Content-Range, Content-Type, Date, Host, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since, Origin, OriginToken, Pragma, Range, Slug, Transfer-Encoding, Want-Digest, X-MediaBrowser-Token, X-Emby-Token, X-Emby-Authorization
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, PATCH, OPTIONS
Access-Control-Allow-Origin: *
Server: UPnP/1.0 DLNADOC/1.50
Content-Type: video/mp4
Accept-Ranges: none
Date: Tue, 11 Feb 2020 18:57:52 GMT
Transfer-Encoding: Chunked

It looks like this has to do with the stripping of query parameters but this behavior is not RFC compliant.

 

https://tools.ietf.org/html/rfc7233#page-10

 

 

 

 

[Luke 37248]

Hi, no, definitely send 206 responses. Please attach a server log example. It's probably your reverse proxy configuration causing this.

[Luke 37248]

See:

[Luke 37248]

 

 

I was able to repro this manually using curl:

That's not correct api usage. To learn more please check out our API wiki:

https://github.com/MediaBrowser/Emby/wiki

[enqbcvqw 2]

I did see the difference in behavior when other parameters were present after I posted this.

 

If a range is satisfiable it should be satisfied. I'm not sure what design choice leads to choosing to satisfy a range based on the presence or absence of query parameters. A more ideal behavior would be to satisfy whenever possible as this is what's encouraged. Even if the origin MAY decide to ignore the Range header, it's not supposed to unless it's an IMS that should result in 304, the range is in a unit it doesn't understand or if the client is broken (requesting overlapping ranges or different, non-ascending ranges). 

 

IMHO, if you want to fail to satisfy based on the absence of a proper request with the proper parameters, a better way would be to fail altogether with a 400 Bad Request. Even if this shouldn't happen under normal conditions it's a much better signal to the client (or the engineer in this case ) that something is wrong with the request.

[Luke 37248]

Thanks for the feedback.

[enqbcvqw 2]

Something still looks odd. I continue to get 200's in the CDN logs. Based on the URLs the clients are requesting, they look like this:

 

/emby/Videos/1778502/stream.mp4?static=true&mediaSourceId=31ae48dad5863b4d427cf715928d1eb8&api_key=4b79fcb30d0140c1ac

02f66a12ccde54

 

I'm not stripping or modifying any of the query parameters, headers or cookies. Is it expected for some clients to get 200's back? The logs show this is a Roku device:

 

HIT|200|1581493425108|18629618|111885|162.158.246.0|-|https://xxxx/emby/Videos/1778502/stream.mp4?static=true&mediaSourceId=31ae48dad5863b4d427cf715928d1eb8&api_key=4b79fcb30d0140c1ac02f66a12ccde54 |LA|Roku/DVP-9.20 (119.20E04805A)|f6b3153283b38258b66587bfe63c9ec8|US

 

Edited February 12, 2020 by enqbcvqw

[Luke 37248]

I can't really answer with that little information. I would start by comparing your nginx configuration to that of @@pir8radio

[pir8radio 1293]

What CDN?    also i guess i'm not quite following your issue...  what problem are you seeing?  other than the chunked send..

Edited February 12, 2020 by pir8radio

[enqbcvqw 2]

BunnyCDN.

 

I've noticed two problems with the CDN getting a 200 back from Emby to a range request:

 

1/ Certain clients and/or content tend to stop attempting to direct play and will choose to direct stream instead and I can't cache transcoded content since it has too many arguments and some of them are specific to that transcode session.

2/ On clients and/or content that does direct play regardless of the 200, the user cannot skip forward or backwards since the response to them skipping is the entire file (200 OK), so whatever they're watching starts over and they can't scrub back to the point where they were.

 

This is the nginx config I'm toying with:

    slice 8m;
    proxy_set_header Range $slice_range;
    expires max;
    proxy_buffering on;
    proxy_cache emby_cache;
    add_header X-FTV-Cache $upstream_cache_status;
    add_header Cache-Control "public";
    proxy_pass http://emby_backend_cache;
    proxy_cache_valid 200 206 30d;
    proxy_cache_lock off;
    proxy_cache_use_stale error timeout updating;
    proxy_force_ranges on;
    proxy_cache_revalidate on;
    proxy_cache_key $scheme$proxy_host$uri$slice_range;

8MB is equal to the chunk range the CDN requests. The problem appears to go beyond the CDN though. If I remove it and just do local nginx caching I get a bunch of these in nginx's error.log:

2020/02/14 10:21:40 [error] 11201#11201: *250715 unexpected range in slice response: 2088763392-2097152000 while reading response header from upstream, client: 2001:4878:8151:800:8162:d59:c1b3:f629, server: redacted.domain.com, request: "GET /emby/videos/1797741/stream.mp4?DeviceId=TW96aWxsYS81LjAgKE1hY2ludG9zaDsgSW50ZWwgTWFjIE9TIFggMTBfMTRfMykgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzc0LjAuMzcyOS4xNjkgU2FmYXJpLzUzNy4zNnwxNTU4NzE1MDU0ODUw&MediaSourceId=c386d85493124d45014e9d3a2d25e38a&Static=true&PlaySessionId=e9633efa377f460fad6a40366a14bce6&api_key=a7e06082a2e7446dafde7af9a83daef6 HTTP/1.1", upstream: "http://127.0.0.1:8091/emby/videos/1797741/stream.mp4?DeviceId=TW96aWxsYS81LjAgKE1hY2ludG9zaDsgSW50ZWwgTWFjIE9TIFggMTBfMTRfMykgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzc0LjAuMzcyOS4xNjkgU2FmYXJpLzUzNy4zNnwxNTU4NzE1MDU0ODUw&MediaSourceId=c386d85493124d45014e9d3a2d25e38a&Static=true&PlaySessionId=e9633efa377f460fad6a40366a14bce6&api_key=a7e06082a2e7446dafde7af9a83daef6", host: "redacted.domain.com", referrer: "http://app.emby.media/"

The corresponding Emby log shows a 206 response but these logs just look messed up:

2020-02-14 10:21:40.822 Info HttpServer: HTTP GET http://emby_backend_cache:8091/emby/videos/1797741/stream.mp4?DeviceId=TW96aWxsYS81LjAgKE1hY2ludG9zaDsgSW50ZWwgTWFjIE9TIFggMTBfMTRfMykgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzc0LjAuMzcyOS4xNjkgU2FmYXJpLzUzNy4zNnwxNTU4NzE1MDU0ODUw&MediaSourceId=c386d85493124d45014e9d3a2d25e38a&Static=true&PlaySessionId=e9633efa377f460fad6a40366a14bce6. Range=bytes=2088763392-2097151999, Host=emby_backend_cache, Connection=close, Accept-Encoding=gzip, X-Forwarded-For=2001:4878:8151:800:8162:d59:c1b3:f629, CF-RAY=5650605aa1cac89f-MIA, X-Forwarded-Proto=https, CF-Visitor={"scheme":"https"}, CF-EW-Via=15, CDN-Loop=cloudflare; subreqs=1, Accept-Language=en-US,en;q=0.9,es;q=0.8, Accept=*/*, Cache-Control=no-cache, Referer=http://app.emby.media/, User-Agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.106 Safari/537.36, CF-Connecting-IP=2001:4878:8151:800:8162:d59:c1b3:f629, dnt=1, origin=http://app.emby.media, pragma=no-cache, sec-fetch-dest=video, sec-fetch-mode=cors, sec-fetch-site=cross-site
2020-02-14 10:21:40.831 Info HttpServer: SocketException: http://emby_backend_cache:8091/emby/videos/1797741/stream.mp4?DeviceId=TW96aWxsYS81LjAgKE1hY2ludG9zaDsgSW50ZWwgTWFjIE9TIFggMTBfMTRfMykgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzc0LjAuMzcyOS4xNjkgU2FmYXJpLzUzNy4zNnwxNTU4NzE1MDU0ODUw&MediaSourceId=c386d85493124d45014e9d3a2d25e38a&Static=true&PlaySessionId=e9633efa377f460fad6a40366a14bce6&api_key=a7e06082a2e7446dafde7af9a83daef6
2020-02-14 10:21:40.831 Info HttpServer: HTTP Response 206 to 2001:4878:8151:800:8162:d59:c1b3:f629. Time: 10ms. http://emby_backend_cache:8091/emby/videos/1797741/stream.mp4?DeviceId=TW96aWxsYS81LjAgKE1hY2ludG9zaDsgSW50ZWwgTWFjIE9TIFggMTBfMTRfMykgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzc0LjAuMzcyOS4xNjkgU2FmYXJpLzUzNy4zNnwxNTU4NzE1MDU0ODUw&MediaSourceId=c386d85493124d45014e9d3a2d25e38a&Static=true&PlaySessionId=e9633efa377f460fad6a40366a14bce6

This was in the browser and the video never actually started. It looks more like Emby doesn't seem to like nginx's slice module at all.

 

EDIT: looks like the source file was broken. When I tried the same workflow with a different file it's working properly so I'll get rid of that file and will continue to test.

Edited February 14, 2020 by enqbcvqw

[pir8radio 1293]

Yea.....  Good luck lol...   I messed with caching the video early on (years ago) and didnt have too much luck, similar issues, loosing functions, etc.   

 

 

So are you really trying to improve performance for "far away users" or are you trying to improve performance for a mass customer base?    Caching is almost needed for the latter....    But if you you are really just trying to improve for distant users,  why is your CDN not good enough today?      You should have few hops through the CDN and their network should be much quicker than the open internet..   I have had good success with Cloudflare across the globe..    With the exception of Asia they like to sniff and modify all of the traffic real time (government censorship) which makes a connection into some countries from others rough.

Edited February 15, 2020 by pir8radio

[enqbcvqw 2]

It's the latter, users are spread out anywhere and the connectivity to the origin isn't great from everywhere. Plus, I'm a performance freak

 

I do have CloudFlare but they do not allow caching of video so I bifurcate those requests via BunnyCDN.

 

Anything that misses Bunny goes through several mid tier layers before it gets to origin. Based on my current testing things appear to be working, or did when I tested several different files that weren't that one I opened the thread with.

[pir8radio 1293]

It's the latter, users are spread out anywhere and the connectivity to the origin isn't great from everywhere. Plus, I'm a performance freak

 

I do have CloudFlare but they do not allow caching of video so I bifurcate those requests via BunnyCDN.

 

Anything that misses Bunny goes through several mid tier layers before it gets to origin. Based on my current testing things appear to be working, or did when I tested several different files that weren't that one I opened the thread with.

 

So its not the latter (lol i think i worded it wrong)   you are not trying to support LOTS of users or are you?       CF works pretty well for just spread out users without caching, thats what i was hinting at..    Yea they dont do caching of video, i was just trying to figure out the goal of caching for you..   Well have to share your final setup when you figure it out... I always like learning a thing or two..    

Edited February 15, 2020 by pir8radio

[enqbcvqw 2]

I can't really answer with that little information. I would start by comparing your nginx configuration to that of @@pir8radio

 

I now understand my problem better. Nginx's slice module breaks down the request into smaller slices of 1 MB by default. In the case of the CDN using this it's 8 MB chunks.

 

Based on my observations the problem appears to be a bug in Emby's partial response logic.

 

Take a look at the following response from Emby:

# curl 'http://localhost:8091/emby/videos/1859715/stream.mp4?DeviceId=TW96aWxsYS81LjAgKFgxMTsgQ3JPUyB4ODZfNjQgMTI2MDcuODEuMCkgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzc5LjAuMzk0NS4xMTkgU2FmYXJpLzUzNy4zNnwxNTgxODI4NTM1MjI2&MediaSourceId=4050641264f42a5700b0a2545d04319d&Static=true&PlaySessionId=b94e49f249a84759b6d735fd8f3e0717&api_key=79fd2fabba0b4aadb1fd680553b24d39' -H 'range: bytes=0-' --compressed -IXGETHTTP/1.1 206 PartialContent
Access-Control-Allow-Headers: Accept, Accept-Language, Authorization, Cache-Control, Content-Disposition, Content-Encoding, Content-Language, Content-Length, Content-MD5, Content-Range, Content-Type, Date, Host, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since, Origin, OriginToken, Pragma, Range, Slug, Transfer-Encoding, Want-Digest, X-MediaBrowser-Token, X-Emby-Token, X-Emby-Authorization
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, PATCH, OPTIONS
Access-Control-Allow-Origin: *
Server: UPnP/1.0 DLNADOC/1.50
Content-Type: video/mp4
Accept-Ranges: bytes
Content-Range: bytes 0-1261146521/1261146522
ETag: "8b7d25736e30679c0a148485d5bb0e8f"
Cache-Control: public
Date: Sun, 16 Feb 2020 05:16:07 GMT
Content-Length: 1261146522

Now, look what happens if I request a range that spills over the file's actual Content-Length:

# curl 'http://localhost:8091/emby/videos/1859715/stream.mp4?DeviceId=TW96aWxsYS81LjAgKFgxMTsgQ3JPUyB4ODZfNjQgMTI2MDcuODEuMCkgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzc5LjAuMzk0NS4xMTkgU2FmYXJpLzUzNy4zNnwxNTgxODI4NTM1MjI2&MediaSourceId=4050641264f42a5700b0a2545d04319d&Static=true&PlaySessionId=b94e49f249a84759b6d735fd8f3e0717&api_key=79fd2fabba0b4aadb1fd680553b24d39' -H 'range: bytes=1261146500-1261146550' --compressed -IXGET
HTTP/1.1 206 PartialContent
Access-Control-Allow-Headers: Accept, Accept-Language, Authorization, Cache-Control, Content-Disposition, Content-Encoding, Content-Language, Content-Length, Content-MD5, Content-Range, Content-Type, Date, Host, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since, Origin, OriginToken, Pragma, Range, Slug, Transfer-Encoding, Want-Digest, X-MediaBrowser-Token, X-Emby-Token, X-Emby-Authorization
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, PATCH, OPTIONS
Access-Control-Allow-Origin: *
Server: UPnP/1.0 DLNADOC/1.50
Content-Type: video/mp4
Accept-Ranges: bytes
Content-Range: bytes 1261146500-1261146550/1261146522
ETag: "8b7d25736e30679c0a148485d5bb0e8f"
Cache-Control: public
Date: Sun, 16 Feb 2020 05:18:00 GMT
Content-Length: 51

This is incorrect behavior. This file is 1261146522 bytes yet Emby happily served me 51 bytes which is 28 bytes over the size. If you look at the Content-Range header, the last-byte-pos is greater than complete-length and this is not compliant. From the RFC:

 

A Content-Range field value is invalid if it contains a

byte-range-resp that has a last-byte-pos value less than its

first-byte-pos value, or a complete-length value less than or equal

to its last-byte-pos value.

 

What Emby _should_ do is cut last-byte-pos at the last byte and serve a Content-Range/Length headers that look like this:

 

Content-Range: bytes 1261146500-1261146521/1261146522

Content-Length: 22

 

Even if Nginx is slicing the content and requesting a Range larger than the file size, it's still a compliant proxy and it expects last-byte-pos to fall within the range of the file's length and since it doesn't it aborts and serves and empty response, which the CDN can't understand and translates into a 502.

Edited February 16, 2020 by enqbcvqw

[Luke 37248]

Thanks, I'll add this for the next release.

[enqbcvqw 2]

Is this still on track for the next release? Would really like to give video caching with emby a shot. Thanks!

[Luke 37248]

Yes it is.

[enqbcvqw 2]

Awesome! What version would that be?

[Luke 37248]

4.4+.