Page doesn´t reload: 304 status code

[dcrdev 254]

I believe it's the ones that are:

 

Going back to my point around lack of content-length headers - that in combination with the chunked data Apache doesn't know when to close the connection; so it stays open for the duration of whatever the default value of ProxyTimeout actually is. I think this is magnified by whatever is cached by serviceworker.js because it is continually receiving empty 304 responses from over a very short period of time, in effect causing a race condition.

 

I think this is reaffirmed by the fact that it working with an IIS reverse proxy - looking at the wording here https://technet.microsoft.com/en-us/library/dd443533(v=ws.10).aspx seems to suggest that keep-alive isn't implemented by default.

[Luke 40144]

@@dcrdev what if you revert the send chunked setting back to default?

[dcrdev 254]

@@dcrdev what if you revert the send chunked setting back to default?

 

Yeah it doesn't really make a difference - the option doesn't affect chunked data received from the server only data sent through the proxy to the server; this was added as a just-in-case. It's essentially only the proxy-nokeepalive option that's allowing it to work. But again I really do believe it's the proxy not knowing when to close a connection due to the encoding and lack of content-length headers (on some resources).

 

If you look at the (Apache) server logs you see each and every request made to a resource that is is failing to load in Chrome's debugger, is hitting the ProxyTimeout wall; Emby has closed the connection but Apache isn't detecting it due to the encoding.

Edited December 21, 2016 by dcrdev

[Luke 40144]

@@Luke

General
-----------
Request URL:https://emby.mydomain.net/emby/system/info/public
Request Method:GET
Status Code:200 OK
Remote Address:192.168.1.2:443

Response Headers
------------------
Connection:Keep-Alive
Date:Tue, 20 Dec 2016 12:16:43 GMT
Keep-Alive:timeout=5, max=98
Server:Mono-HTTPAPI/1.1, UPnP/1.0 DLNADOC/1.50
Strict-Transport-Security:max-age=15768000
Transfer-Encoding:chunked
Vary:Host

Request Headers
----------------
accept:application/json
Accept-Encoding:gzip, deflate, sdch, br
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
DNT:1
Host:emby.mydomain.net
Referer:https://emby.mydomain.net/web/index.html
User-Agent:Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36

I'm just speculating here - but doesn't the client need to explicitly receive a content-length header when dealing with gzip, otherwise it can't differentiate between delay and the end of the chunk? Emby seems to be sending out that header for some content, but not all...

 

For the above request, this is what comes out of Emby Server.

Access-Control-Allow-Headers:Content-Type, Authorization, Range, X-MediaBrowser-Token, X-Emby-Authorization
Access-Control-Allow-Methods:GET, POST, PUT, DELETE, PATCH, OPTIONS
Access-Control-Allow-Origin:*
Content-Encoding:deflate
Content-Length:162
Content-Type:application/json; charset=utf-8
Date:Wed, 21 Dec 2016 14:52:38 GMT
Expires:-1
Server:Mono-HTTPAPI/1.1, UPnP/1.0 DLNADOC/1.50
Vary:Accept-Encoding
X-UA-Compatible:IE=Edge

[Guest plexman]

 

Ok I now believe there is some kind of race occurring between each request and serviceworker.js which is causing a timeout in some browsers.

 

I don't have an ideal solution but - for now you can close the connection forcefully after each request is made.

 

For Apache this is done via the environment variables:

SetEnv proxy-nokeepalive 1

SetEnv proxy-sendchunked 1

 

I've not done much testing - but the web interface refreshed about 50 times in a row. 

 

Also playback won't work initially - you'll have to disable the Christmas theme; I believe that is because mono is sending out a CORS header and the christmas theme music is hosted at raw.githubusercontent.com which has no CORS header.

 

Edit: To elaborate ^ it's requesting a javascript resource, not getting it and that's resulting in an undefined '$' error.

 

Thanks for this!

[dcrdev 254]

 

For the above request, this is what comes out of Emby Server.

Access-Control-Allow-Headers:Content-Type, Authorization, Range, X-MediaBrowser-Token, X-Emby-Authorization
Access-Control-Allow-Methods:GET, POST, PUT, DELETE, PATCH, OPTIONS
Access-Control-Allow-Origin:*
Content-Encoding:deflate
Content-Length:162
Content-Type:application/json; charset=utf-8
Date:Wed, 21 Dec 2016 14:52:38 GMT
Expires:-1
Server:Mono-HTTPAPI/1.1, UPnP/1.0 DLNADOC/1.50
Vary:Accept-Encoding
X-UA-Compatible:IE=Edge

 

 

For the above request, this is what comes out of Emby Server.

Access-Control-Allow-Headers:Content-Type, Authorization, Range, X-MediaBrowser-Token, X-Emby-Authorization
Access-Control-Allow-Methods:GET, POST, PUT, DELETE, PATCH, OPTIONS
Access-Control-Allow-Origin:*
Content-Encoding:deflate
Content-Length:162
Content-Type:application/json; charset=utf-8
Date:Wed, 21 Dec 2016 14:52:38 GMT
Expires:-1
Server:Mono-HTTPAPI/1.1, UPnP/1.0 DLNADOC/1.50
Vary:Accept-Encoding
X-UA-Compatible:IE=Edge

 

So I've done some investigation this evening - the above request was erroneous because it was taken mid timeout.

 

If you look at a complete proxied response:

Access-Control-Allow-Headers:Content-Type, Authorization, Range, X-MediaBrowser-Token, X-Emby-Authorization
Access-Control-Allow-Methods:GET, POST, PUT, DELETE, PATCH, OPTIONS
Access-Control-Allow-Origin:*
Cache-Control:public
Connection:Keep-Alive
Content-Encoding:deflate
Content-Length:1560
Content-Type:text/html; charset=UTF-8
Date:Wed, 21 Dec 2016 19:40:37 GMT
ETag:"f78c63cc3676190f75b5db292707ad1d"
Expires:-1
Keep-Alive:timeout=5, max=98
Server:Mono-HTTPAPI/1.1, UPnP/1.0 DLNADOC/1.50
Strict-Transport-Security:max-age=15768000
Vary:Accept-Encoding
X-UA-Compatible:IE=Edge

You'll see that it is *almost identical to that of a response from Emby directly.

 

The ONLY difference is the addition of the Keep-Alive and Connection: Keep-Alive headers - these are being added by mod_proxy for some reason and can't be unset using mod_headers. Now I believe with the http 1.1 spec these headers are redundant - it is assumed that a connection should be kept alive until such a time as the server issues a Connection: close header. I don't know whether setting Keep-Alive overrides the assumed behaviour of http 1.1 i.e. is it overriding the timeout value.

 

The big question here is what has changed within Emby in that regard to cause this issue because those arbitrary headers inserted by mod_proxy have always been there. It is unclear whether or not disabling KeepAlive for a particular virtualhost impedes the default behaviour of http 1.1 - if it does then that may potentially be causing issues with Emby that are yet to be discovered.

[Luke 40144]

So you're saying having the proxy add Connection:Keep-Alive is resolving it?

 

I am still investigating but I don't think there are any changes in headers between this release and the last. My current theory is that the connection might be closed but the server is not adding the Connection:Close header - not so much an issue with the headers but the fact that it's being closed when it shouldn't. I am not positive yet though.

[dcrdev 254]

So you're saying having the proxy add Connection:Keep-Alive is resolving it?

 

I am still investigating but I don't think there are any changes in headers between this release and the last. My current theory is that the connection might be closed but the server is not adding the Connection:Close header - not so much an issue with the headers but the fact that it's being closed when it shouldn't. I am not positive yet though.

 

No the opposite -

 

I'm saying that no matter what you do mod_proxy adds those headers, they cannot be unset, filtered out or rewritten; it's intentional behaviour, although there is a proposed patch somewhere on the Apache bugtracker. What I am saying is that since Emby is providing HTTP/1.1 responses then KeepAlive is on by default regardless of headers - Connection:Keep-Alive and Keep:Alive are only relevant for HTTP/1.0 responses.

 

For whatever reason it looks like the redundant headers might be being respected (the timeout value) - possibly because Emby is breaking the HTTP/1.1 spec in some fashion. If your above theory is correct and according to the HTTP/1.1 (see "Persistent Connections" here: http://www8.org/w8-papers/5c-protocols/key/key.html)spec you should close a connection with a Connection: close header; I guess if that's not happening it could fallback to the extremely short timeout value being added by mod_proxy.

[Luke 40144]

@@dcrdev, would you mind performing a test. Please unzip this file, locate the equivalent file in your emby server installation and replace it with this one. (make sure to backup). Thanks !

SocketHttpListener.Portable.zip

[dcrdev 254]

I just tested it and it doesn't appear to have made a difference.

 

Just to be clear these are the steps I took:

• Backed up /usr/lib/emby-server/bin/SocketHttpListener.Portable.dll
• Moved in the library.
• Copied ownership and permissions from the old library.
• Removed the workaround from the proxy configuration.
• Restarted both Apache and Emby
• Cleared browser cache entirely
• Emby loads over proxy intially - but fails to load after second attempt.

I'm getting the ERR_INCOMPLETE_CHUNK message in the console and nothing else:

 

[Luke 40144]

if you click on that home request it will show all the request headers and response headers. can you post a screenshot of those? thanks.

[dcrdev 254]

Sure thing:

 

[Luke 40144]

Thanks.

[Luke 40144]

the server should be setting a content length on that. in other words, that html file should not be a chunked response.

[dcrdev 254]

It does set a content-length but not on partial page loads such as that, if you manage to get more of the page to load by refreshing 4-5 times you get:

 

[Luke 40144]

We have a fix that will be in 3.1.2. thank you all for your patience.