All playback failing in Roku app

[BenLand100 1]

After updating emby server to 4.5.4 and running a library scan (unsure if relevant), all media playback fails in the roku app (4.0.4). Libraries can be explored, all metadata appears, but when attempting to play any media, the "loading ring" will appear for about a minute, and then the app goes back to the previous page. No error message present on the roku app, but nothing will play.

Server details:
Linux (docker latest) 4.5.4.0
Running behind apache proxy (previously working, no changes here)
 

2021-01-01 00:00:00.004 Info App: Application version: 4.5.4.0
2021-01-01 00:00:00.004 Info App: Emby
    Command line: /system/EmbyServer.dll -programdata /config -ffdetect /bin/ffdetect -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartexitcode 3
    Operating system: Linux version 5.4.85-gentoo (root@bland) (gcc version 9.3.0 (Gentoo 9.3.0-r2 p4)) #1 SMP Sun Dec 27 11:15:43 EST 2020
    Framework: .NET Core 3.1.8
    OS/Process: x64/x64
    Runtime: system/System.Private.CoreLib.dll
    Processor count: 8
    Data path: /config
    Application path: /system
2021-01-01 00:00:00.004 Info App: Plugins:
    Bluray Folder Support 1.0.0.0
    Cinema Intros 1.0.35.0
    Dlna 1.0.39.0
    Dvd Folder Support 1.0.0.0
    Fanart.tv 1.0.10.0
    IMVDb 1.0.8.0
    MovieDb 1.3.8.0
    MusicBrainz 1.0.13.0
    Nfo Metadata 1.0.34.0
    OMDb 1.0.12.0
    Open Subtitles 1.0.27.0
    Port Mapper 1.0.4.0
    Studio Images 1.0.3.0
    TheAudioDb 1.0.11.0
    TheTVDB 1.0.36.0
    Webhooks 1.0.14.0

Debugging attempted:
- Checked playback via web app on PC, all playback works fine, no issues
- Downgraded server to 4.5.2 which worked the previous day, same behavior
- Uncorrelated to media type (encoding or library settings) - all media affected
- Reset roku app fully, setup server again, same behavior
- Ran a full library scan on server, same behavior
- Restarted emby server (and roku), same behavior
- Installed Emby channel on a different Roku (TCL roku TV instead of discrete roku device) and setup server, same behavior

Log showing roku app starting, attempted playback, and failure attached. [SERVER_DOMAIN] and [ROKU_IP] replaces the actual values in this log.

Any help would be appreciated!

emby-roku.log

[speechles 1929]

Is your server expecting HTTPS?

 

Spoiler

2021-01-01 16:31:04.332 Info Server: http/1.1 Response 500 to [ROKU_IP]. Time: 300103ms. http://[SERVER_DOMAIN]/emby/Sessions/Capabilities/Full
2021-01-01 16:31:31.060 Error Server: Error processing request
    *** Error Report ***
    Version: 4.5.4.0
    Command line: /system/EmbyServer.dll -programdata /config -ffdetect /bin/ffdetect -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartexitcode 3
    Operating system: Linux version 5.4.85-gentoo (root@bland) (gcc version 9.3.0 (Gentoo 9.3.0-r2 p4)) #1 SMP Sun Dec 27 11:15:43 EST 2020
    Framework: .NET Core 3.1.8
    OS/Process: x64/x64
    Runtime: system/System.Private.CoreLib.dll
    Processor count: 8
    Data path: /config
    Application path: /system
    Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content.
       at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
       at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
       at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.CopyToAsyncInternal(Stream destination, CancellationToken cancellationToken)
       at ServiceStack.StreamExtensions.CopyToNewMemoryStreamAsync(Stream stream) in C:\BuildAgent\work\912418dcce86a188\src\ServiceStack.Text\StreamExtensions.cs:line 517
       at ServiceStack.Memory.NetCoreMemory.DeserializeAsync(Stream stream, Type type, DeserializeStringSpanDelegate deserializer) in C:\BuildAgent\work\912418dcce86a188\src\ServiceStack.Text\NetCoreMemory.cs:line 169
       at Emby.Server.Implementations.Services.ServiceHandler.CreateRequest(HttpListenerHost host, IRequest httpReq, RestPath restPath)
       at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: Microsoft.AspNetCore.Server.Kestrel.Core
    TargetSite: Void Throw(Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.RequestRejectionReason)

2021-01-01 16:31:31.060 Info Server: http/1.1 Response 500 to [ROKU_IP]. Time: 300103ms. http://[SERVER_DOMAIN]/emby/Items/46614/PlaybackInfo?UserId=80d3a7d6af1c4bf1a7739f629a38ee1b&isplayback=false&maxstreamingbitrate=4000000&starttimeticks=00000000

 

[BenLand100 1]

Yes and no. There is a permanent redirect in apache config from http://* to https://* and ProxyPass is only setup for the HTTPS server listening on port 443. ProxyPass is configured to redirect traffic to Emby's http port (set to the default 8096). Relevant lines in apache config are:
 

    ProxyPass "/embywebsocket" "ws://127.0.0.1:8096/embywebsocket"
    ProxyPassReverse "/embywebsocket" "ws://127.0.0.1:8096/embywebsocket"

    ProxyPass "/" "http://127.0.0.1:8096/"
    ProxyPassReverse "/" "http://127.0.0.1:8096/"

 

which (at least in the past) has been sufficient for emby to work on all platforms I use. No remote services connect directly to emby via port 8096 (or https on 8920). External traffic to these ports are dropped. So from the embyserver's POV, all inbound traffic is http on port 8096.

Settings on Emby's network page are all default, except "Secure connection mode" is set to "Handled by reverse proxy"

[Luke 37233]

Hi, I know you said you're using apache, but I would suggest comparing your reverse proxy configuration to that of @pir8radio nginx configuration as that is kind of the ideal model.

[pir8radio 1293]

how about the full apache config...   Have you tried files other than mp4 format do they play?  do you use subdomain or subdirectory on your reverse proxy?     emby.yourdomain.com  or yourdomain.com/emby/        Just curious...  

[BenLand100 1]

Thanks for taking a look. The reverse proxy is using a subdomain of the form `emby.yourdomain.com` replaced with `[SERVER_DOMAIN]` in the following apache config:

<VirtualHost _default_:80>
    ServerName [SERVER_DOMAIN]

    Redirect permanent / https://[SERVER_DOMAIN]/

    Include /etc/apache2/headers.conf

    ErrorLog /var/log/apache2/[SERVER_DOMAIN]_error_log
</VirtualHost>


<IfDefine SSL>
<IfDefine SSL_DEFAULT_VHOST>
<IfModule ssl_module>

<VirtualHost _default_:443>
    ServerName [SERVER_DOMAIN]

    <proxy *>
        AddDefaultCharset off
        require all granted
    </proxy>

    ProxyRequests     Off
    ProxyPreserveHost On

    ProxyPass "/embywebsocket" "ws://127.0.0.1:8096/embywebsocket"
    ProxyPassReverse "/embywebsocket" "ws://127.0.0.1:8096/embywebsocket"

    ProxyPass "/" "http://127.0.0.1:8096/"
    ProxyPassReverse "/" "http://127.0.0.1:8096/"

    ErrorLog /var/log/apache2/[SERVER_DOMAIN]_ssl_error_log

    Include /etc/apache2/ssl.conf
    Include /etc/apache2/headers.conf

    SSLCertificateFile /etc/ssl/apache2/[SERVER_DOMAIN].crt
    SSLCertificateKeyFile /etc/ssl/apache2/[SERVER_DOMAIN].key

</VirtualHost>

</IfModule>
</IfDefine>
</IfDefine>

If you could point me to a known-working configuration (even for nginx) I will sanity check this configuration. I don't recall where I got the inspiration for this several years ago.

That said, this reverse proxy configuration does work for the browser version for emby, and does display libraries and episode/series metadata correctly on the roku app. I suppose if there was a new version of the roku player recently rolled out, that may have changed the requirements for the reverse proxy, but it did previously work for the roku app as well.

 

I have tried .mp4, .mkv, and .avi media: all play on the browser player, but none on the roku app.

[BenLand100 1]

Scouring my apache logs I see the failed request from the Roku app:
 

[CLIENT_IP] - - [04/Jan/2021:11:39:42 -0500] "POST /emby/Sessions/Capabilities/Full HTTP/1.1" 503 349 "-" "Roku/DVP-9.40 (939.40E04200A)"

Playing media from the web browser issues a similar request, but critically for a _different_ path omitting the /emby/ prefix:
 

[CLIENT_IP] - - [04/Jan/2021:11:28:04 -0500] "POST /Sessions/Capabilities/Full?X-Emby-Client=Emby%20Web&X-Emby-Device-Name=Firefox&X-Emby-Device-Id=[id ommitted]&X-Emby-Client-Version=4.5.4.0&X-Emby-Token=[token omitted] HTTP/1.1" 204 - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"

I've tested a POST to both URLs with `curl`  from my laptop (using the parameters from the latter request) and both seem to return status 204, so the path difference is likely not the issue (though curiously redundant).

Based purely on the apache logs, this appears to happen before video streaming is even attempted, so it seems like the Roku app is not sending this request properly, and simply timing out. This seems to be consistent with the emby log from the first post ("Unexpected end of request content").

[pir8radio 1293]

8 hours ago, BenLand100 said:

Thanks for taking a look. The reverse proxy is using a subdomain of the form `emby.yourdomain.com` replaced with `[SERVER_DOMAIN]` in the following apache config:

If you could point me to a known-working configuration (even for nginx) I will sanity check this configuration. I don't recall where I got the inspiration for this several years ago.

That said, this reverse proxy configuration does work for the browser version for emby, and does display libraries and episode/series metadata correctly on the roku app. I suppose if there was a new version of the roku player recently rolled out, that may have changed the requirements for the reverse proxy, but it did previously work for the roku app as well.

 

I have tried .mp4, .mkv, and .avi media: all play on the browser player, but none on the roku app.

click my avitar, then the "About Me" tab, there is a working nginx config there.  

 

I did find this in the forum,  Maybe it will help you with apache...  I'm an nginx fanboy lol,  guessing it has to do with the "proxy-sendchunked" setting.

 

Edited January 4, 2021 by pir8radio

[BenLand100 1]

Unfortunately I don't see any significant differences between @pir8radio's nginx config and mine. Comparing to the (much older, from 2016) apache config in that thread only reveals one significant difference: the URI /socket is referenced for websocket proxying whereas my (previously working) configuration used the /embywebsocket URI for the same. Changing this in my config doesn't have any impact on the Roku app not working, however.

For good measure I've switched to identifying any websocket protocol by headers instead of by URI so that this will be handled correctly regardless of what URI is used. I also tried the various modifications to http headers and environment settings performed in that apache configuration, but none fixed the roku app failing to play.  

 

I'm fairly confident this is not an issue with my reverse proxy setup, since:
* The roku app is able to communicate with the emby server behind the reverse proxy enough to display libraries and metadata (etc)
* The web app in browser works fully with all the various methods of dealing with proxying websockets
* Its evidently a plain http POST from the Roku app that is failing before streaming starts. As I understand it, this precludes an issue with websockets, as they don't support POST requests, and the rest of the reverse proxy setup is a standard recipe.

Any other ideas what might be going on here, or what I could try?

[BenLand100 1]

Inexplicably, I went to test more possible fixes after work today and found that (both) roku devices were playing media again with no changes to any of my setup. The good news is that means my reverse proxy configuration was not the issue. The bad news is I have no idea and no way to test why these Roku wouldn't play media for a span of five days... Fingers crossed it doesn't happen again, I suppose...

[pir8radio 1293]

1 hour ago, BenLand100 said:

Inexplicably, I went to test more possible fixes after work today and found that (both) roku devices were playing media again with no changes to any of my setup. The good news is that means my reverse proxy configuration was not the issue. The bad news is I have no idea and no way to test why these Roku wouldn't play media for a span of five days... Fingers crossed it doesn't happen again, I suppose...

did the server update, or it just fix itself?

[BenLand100 1]

I was using version 4.5.4.0 from the emby/embyserver docker repo since the first post. Prior to that I briefly reverted back to 4.5.2.0 to check if updating to 4.5.4.0 was the issue.

I don't have any auto-updating enabled, and neither did docker container restart between nonworking and working conditions.

The roku and apache server process were restarted several times testing various configurations,  but ultimately what I had initially seems to work fine.

[speechles 1929]

14 hours ago, BenLand100 said:

The roku and apache server process were restarted several times testing various configurations,  but ultimately what I had initially seems to work fine.

I had this happen when my ISP was having issues granting IP addresses. You can tell if your Roku has a flashing white light on it. The Roku TV and Roku devices will flash a white light when they cannot reach roku.com and see if updates are available. If during this time your Roku is blinking it does not see internet and you might just need to wait it out. I had internet fail for 5 hours and turns out it was my ADSL2+ modem needed replacing.

[BenLand100 1]

2 minutes ago, speechles said:

I had this happen when my ISP was having issues granting IP addresses. You can tell if your Roku has a flashing white light on it. The Roku TV and Roku devices will flash a white light when they cannot reach roku.com and see if updates are available. If during this time your Roku is blinking it does not see internet and you might just need to wait it out. I had internet fail for 5 hours and turns out it was my ADSL2+ modem needed replacing.

In my case the Roku seemed fine (Netflix and Amazon apps worked) and internet was stable both for the server and client.

[speechles 1929]

My modem had fubar DNS which is what was causing my issue to obtain an IP address via IPV6. No matter what I would use for name servers it had serious lag and would time out. At that point I knew my modem was a POS and exchanged it.

Edited January 6, 2021 by speechles