Playback pausing issues / websocket and hls.min.js.map 404s

[vhodh 0]

Hi there. I'm experiencing random but regular pausing during playback.

 

I'm running emby in a container on unraid. It is behind a letsencrypt container with nginx reverse-proxy settings.

 

I can confirm playback issues occur on multiple client devices, both during transcoding and direct play/stream.

 

Attached below are my log files from the most recent session that ended with this error. While looking through them I noticed a few errors that might be important:

1) Info HttpServer: HTTP Response 404 to 192.168.1.1. Time: 6ms. http://emby.mydomain.com:8096/web/bower_components/hlsjs/dist/hls.min.js.map

2) Info HttpServer: HTTP Response 404 to 192.168.1.1. Time: 0ms. http://emby.mydomain.com:8096/emby/embywebsocket?deviceId=somenumbers

3) HLS Error: Type: mediaError Details: bufferFullError Fatal: false

Note that errors 2) and 3) are not in the current logs, but I can try to recreate another error to produce fresh "/emby/embywebsocket" 404s if desired.

Error 3) is recorded from the console log in firefox during playback.

 

I included a screenshot from goaccess, a viewer for my nginx access.log.

 

I obscured data I thought was private, but let me know if device id/api ids help.

 

Here is my proxy.conf from the letsencrypt/nginx container:

client_body_buffer_size 128k;

proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

send_timeout 5m;
proxy_read_timeout 1h;
proxy_send_timeout 1h;
proxy_connect_timeout 1h;

proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Ssl on;
proxy_redirect  http://  $scheme://;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_cache_bypass $cookie_session;
proxy_no_cache $cookie_session;
proxy_buffers 32 4k;
proxy_headers_hash_bucket_size 128;
proxy_headers_hash_max_size 1024;

These lines were modified most recently (previously 260 seconds) to see if it was a timeout issue (no luck):

proxy_read_timeout 1h;
proxy_send_timeout 1h;
proxy_connect_timeout 1h;

Important parts from my default.conf:

server {
	listen 80;
	listen [::]:80;
	server_name _;
	return 301 https://$host$request_uri;
}

server {
	listen 443 ssl http2 default_server;
	listen [::]:443 ssl http2 default_server;

	root /config/www;
	index index.html index.htm index.php;

	server_name _;

        include /config/nginx/proxy-confs/*.subdomain.conf;

	include /config/nginx/ssl.conf;

	client_max_body_size 0;

	location / {
		try_files $uri $uri/ /index.html /index.php?$args =404;
	}

	location ~ \.php$ {
		fastcgi_split_path_info ^(.+\.php)(/.+)$;
		fastcgi_pass 127.0.0.1:9000;
		fastcgi_index index.php;
		include /etc/nginx/fastcgi_params;
	}
}

proxy_cache_path cache/ keys_zone=auth_cache:10m;

Here is the emby.conf (subdomain):

server {
    listen 80;
    listen [::]:80;
    server_name emby.*;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name emby.*;

    include /config/nginx/ssl-sub.conf;

    client_max_body_size 0;

    location / {
        include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        set $upstream_emby embyserver;
        proxy_pass http://192.168.1.4:8096;
        proxy_set_header Range $http_range;
        proxy_set_header If-Range $http_if_range;
    }
	
    location ~ (/emby)?/socket {
        include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        set $upstream_emby embyserver;
        proxy_pass http://192.168.1.4:8096;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $http_connection;
   }
}

Finally, this is the ssl-sub.conf I use:

# session settings
ssl_session_timeout 10m;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;

# Diffie-Hellman parameter for DHE cipher suites at 4096 bit
ssl_dhparam /config/nginx/dhparams.pem;

# ssl certs
ssl_certificate /config/keys/letsencrypt/fullchain.pem;
ssl_certificate_key /config/keys/letsencrypt/privkey.pem;

# protocols
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384';
ssl_ecdh_curve secp384r1;


# HSTS, remove # from the line below to enable HSTS
add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload" always;

# OCSP Stapling
ssl_stapling on;
ssl_stapling_verify on;

# Optional additional headers
add_header Content-Security-Policy "frame-ancestors https://*.mydomain.com https://mydomain.com https://$server_name";
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header X-Robots-Tag "noindex, nofollow, nosnippet, noarchive" always;
add_header Feature-Policy "geolocation none;midi none;notifications none;push none;sync-xhr none;microphone none;camera none;magnetometer none;gyroscope none;speaker self;vibrate none;fullscreen self;payment none;" always;

During my research over the past few days on this issue, I ran across the known pausing bug on nvidia shield devices. I do use a shield for my primary tv watching, but my pausing issue is confirmed on other devices as well.

 

I have also read that the emby logs will incorrectly (?) interpret the internal IPs sometimes - otherwise I would be more confused by why the 404s in those logs tend to be from 192.168.1.1 (my gateway) and not the 192.168.1.4 (server) or an internal address (local or container). I am not knowledgeable enough to know if that's actually a problem.

 

One final note, although it is extremely anecdotal: the pausing issue occurs dramatically more often with 4k content. This happens on both HEVC and H264 encoded 4k, and during both direct play/stream and transcoding. I say anecdotal as the pausing still occurs during 1080p content playback and altogether the issue seems irregular, so it's easy to hit a window of no issues and watch a full episode or movie.

 

Thank you in advance to anyone that lends a hand/pair of eyes. I'm happy to provide additional information or configuration settings.

 

Cheers.

 

 

edit - see bottom post for updated log files

Edited May 1, 2019 by vhodh

[Luke 37065]

Just FYI, you'll only see the .map errors when opening the browser debug console, so i wouldn't worry about that.

[vhodh 0]

I am somehow just noticing that the 404 there is for "/web/bower_components/" - is that supposed to be "browser_components" ?

 

But if you say that's from debug stuff, then I'll ignore it.

 

edit - I'm seeing it's from the package manager, my bad.

Edited April 19, 2019 by vhodh

[vhodh 0]

After disabling the letsencrypt container entirely and switching the emby container to a "host" network in unraid, I am not experiencing any pauses or playback issues on any of the media that was causing them to occur.

 

This leads me to believe that the reverse-proxy is mishandling something important. I've spent the most time tweaking these settings, so any help would be greatly appreciated as I'm running out of ideas.

[KMBanana 84]

Here's my emby serverblock, I don't keep it in a separate .conf file but put it right in the "site-confs/default" file below the main server block.  It's mostly based on work done by Swynol and pir8radio so credit to them, but see if this works any better.  I also am using the linuxserver/letsencrypt container so it should already have similar ssl.conf and nginx.conf files and includes.

#Emby
server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2;
        server_name emby.domain.tld;
        include /config/nginx/ssl.conf;
        client_max_body_size 0;
        location / {
                proxy_pass http://192.168.2.99:8096; # Local emby ip and non SSL port
                #satisfy all;
                #auth_basic "Restricted";
                #auth_basic_user_file /config/nginx/.htpasswd;
                proxy_hide_header X-Powered-By;
                proxy_set_header Range $http_range;
                proxy_set_header If-Range $http_if_range;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                #Next three lines allow websockets
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_set_header        Authorization $http_authorization;
                proxy_pass_header       Authorization;
        }
}

[vhodh 0]

 

Here's my emby serverblock...

 

Thanks! I'm going to try simply swapping your config for mine directly and then testing. I'm seeing that most people don't use the extra location settings for a specific websockets address, and I haven't tried that yet.

 

-edit: still no luck with that config. I tried it directly, then changed my ssl config back to default as well so not use any other header variables - both have the same issue.

Edited April 20, 2019 by vhodh

[vhodh 0]

Welp. The proxy was not the issue.

 

I experienced a pausing problem with letsencrypt container disabled and Emby running in host mode.

 

I have now tried Emby, Emby Beta, Jellyfin, and Plex. All have a similar pausing problem. - they all use ExoPlayer on the Shield - so this might not be that informative?

 

I tried changing network topology by swapping out my entire Unifi network back to the ISP-provided modem.... similar pausing issue.

 

ABSOLUTELY HUGE CAVEAT: ultimately I used the Nvidia Shield TV when testing the majority of times. I did test on a desktop and laptop but not as often.... so I may have missed something here.

 

... The reason that's such a huge caveat is due to the nearly-identical pausing bug apparently present on the Shield TV. My control has been a laptop streaming through Firefox, but I've done so many tests I lost track of which ones were silo'd away from the Shield.

 

 

I am currently running a 48-hour memtest to see if there is a hardware failure, as I am running out of other options.

 

Due to the way that unraid (and I assume other linux derivatives) handles RAM, I'm guessing that if there was a hardware failure it would be more present in 4k/larger movies due to literally more RAM being taken up as a buffer, thus increasing the odds it hits a bad sector/flash/whatever. Is that an accurate assumption?  During all of my tests, all media players had appdata/config folders on a dedicated SSD. I've learned that I can specify a transcoding folder, and am thinking of changing it from it's default to "/tmp" to see if that alters any behavior. Again... I'm imagining this would increase the odds of bad RAM being hit due to more being used. I am absolutely not certain on this - just a guess.

 

Lastly, I should mention that I have not experienced any other issues that could be attributed to bad RAM or other hardware failures. I ran multiple benchmarking and stress tests when I built my server and had no red flags when doing so. I do not abuse my server beyond serving up multiple media streams/transcodes, so I might not have hit the required amount of real-world use to trigger an error until now. Anecdotal, of course, but I thought it worth mentioning.

 

Am I going in the right direction here?

Edited April 25, 2019 by vhodh

[vhodh 0]

Here is a snapshot of Netdata at the exact moment a 4k stream pauses (and ultimately never continues):

 

 

Not sure if this is useful at all... but there it is.

[vhodh 0]

Here is a snapshot of the same event but including AFTER the pause. Note that it continues to read from disk but never sends any data over the network. I believe this is due to the client still sending playback position information, or maybe the reverse? I can provide logs/gif of a web console during a pause if that helps.

 

 

 

 

After browsing the other subforums (duh), I came across this similar post in the Windows section:

 

https://emby.media/community/index.php?/topic/72236-emby-stops-during-playback/

 

Looks like there's an extremely similar bug happening here also with no current resolution. I wonder if they are related.

Edited April 25, 2019 by vhodh

[nl9800 0]

I am some way or another simply seeing that the 404 there is for "/web/bower_components/" - is that expected to be "browser_components" ?

 

However, on the off chance that you state that is from investigate stuff, at that point I'll disregard it.

 

alter - I'm seeing it's from the bundle supervisor, my awful.

Edited April 25, 2019 by CBers

[KMBanana 84]

Two things I would check.  Firstly does this same stuttering happen if you force the video to transcode by setting the video to low quality?

Secondly what sort of results do get with this test for bufferbloat, for both your server and client network setup?  http://www.dslreports.com/speedtest

I wouldn't use the shield at all for testing because of the known issue.  

[vhodh 0]

Thanks for your reply, KMBanana.

 

1) I do see better results with transcoding to a lower quality. By better, I mean I can stream problematic media longer than normal, but I still run into pausing issues. Initially, I assumed that my issue was 4k ONLY as that is where I first experienced the problem and 1080p content did not seem to result in pauses.

 

After extended testing and real-world use, I found that 1080p content DID pause occasionally. This led me to believe again that larger files indeed have more regular and consistent pausing due to dumping a larger amount of RAM. This was mostly tested on a Shield, which is obviously not ideal.

 

Further testing away from the Shield on a laptop and desktop have shown me that pausing will occur SOONER on those devices than a shield. This is tested in Firefox/web-client.

 

Anecdotally, I have been told that dramatically lowering bitrate (1Mbit @ 720p) on an iOS device on an EXTERNAL network was the only thing that stopped pausing/connectivity issues.

 

2) Buffer bloat is fine. I get "A" scores when I test on an ethernet device. My server and clients during this testing have ALL been on the same INTERNAL network - including the testing where I disabled the nginx/reverse-proxy where absolutely everything should have been handled on a local network.

 

Internal network: gigabit fiber WAN -> unifi-usg -> unifi switch -> server/clients (complete path is 1Gbps all the way)

 

The only testing I did on an external network was done by a friend and is the anecdote I mention in the previous point.

 

 

Finally: all of my testing from now on will be explicitly laptop/desktop clients and NOT my Shield. I will loop back around to re-testing my network hardware at the next convenient time.

 

A related question: what devices are KNOWN to have zero pausing issues that I can confidently test on? I see that in my previously referenced post - https://emby.media/community/index.php?/topic/72236-emby-stops-during-playback/ - the pausing issues look to be occurring on Firefox, Chrome, and Emby Theater apps.

Edited April 25, 2019 by vhodh

[Luke 37065]

There are no apps with known pausing issues. I realize you're running into that, but I'm sure we can get to the bottom of it.

[vhodh 0]

Thanks, Luke.

 

I hope I'm not coming across as negative, just trying to narrow it down.

 

It does seem that the Shield TV currently has a known firmware issue - thus making it a bad testing device.

 

Regardless, I am doing my testing on Firefox in Windows 10 for the time being until otherwise advised.

[Luke 37065]

Can you try lowering the in-app quality setting to force the above example to transcode?

[vhodh 0]

Hi Luke,

 

I replied in post #12 about that, yes. I have tried it, and it does seem to help - the problem being that it eventually still pauses.

 

It is rather difficult to test quickly since this issue can occur within 10-30 seconds on some platforms (desktop/laptop moreso), or take as long as 45+ minutes. I will try a longer testing of my primary file that is consistent in giving me issues and let you know. It will have to wait on my memtest, unfortunately, so I'll reply tomorrow with results if I can.

[vhodh 0]

Memtest did not reveal anything out of the ordinary or all that informative.

 

I am currently testing forced transcoding and here are my results thus far:

 

media info: 4K H264 AVC 8-bit @ 37Mbit

player: web-client in Firefox on Windows 10

bitrate cap = NO (120Mbit)

 

default playback: direct-streaming - pauses at ~30 seconds

4k @ 120Mbit: direct-streaming - at ~30 seconds

4K @ 40Mbit: direct-streaming - pauses at ~30 seconds

1080p @ 60Mbit: direct-streaming - pauses at ~30 seconds

1080p @ 40Mbit: direct-streaming - pauses at ~30 seconds

1080p @ 30Mbit: transcoding - NO APPARENT PAUSING

1080p/720p < 40Mbit: transcoding - NO APPARENT PAUSING

 

Watching the "pausing" occur in the Emby Server Dashboard, emby continues to show playback occurring, but it resets position once ever 5-10 seconds or so - reverting back to the "paused" position and then "continuing" again until it resets.

 

The direct-streaming CONTINUES to be processed according to the dashboard. I waited until the entire direct-streaming process had completed (100%) and did not touch the media stream while doing so... waiting for it to pick back up after it finished.... but it did not. It stays paused after the process is complete.

 

I'd be happy to package up a set of log files (I assume embyserver.txt + ffmpeg results) if that helps beyond my initial set I posted in the OP.

Edited April 25, 2019 by vhodh

[vhodh 0]

Update:

 

Transcoding by forcing a low bitrate continues to work and so far has consistently not prompted pausing issues.

 

I've noticed that after an initial transcoding occurs (file/whatever is created), subsequent streams at the same bitrate will similarly not trigger pausing. I imagine this is due to the same file simply being streamed again - no new transcoding is needed and the streaming of data on the network doesn't seem to be an issue at all.

 

One thing that bothers me here is the lack of ability to direct-play files for testing. This is the main reason I own an Shield TV - direct playback of 4K UHD/HDR files over the network. It seems that Chrome and Firefox both do not support this. Luckily (!!) I am an Emby Premiere subscriber and apparently I can try the Emby Theater app on Windows 10. I will try to see if this supports direct-play and if my issues continue in that scenario.

 

Has this moved my issue from something unique into a known problem? It seems that forcing transcoding by lowering bitrate is a common response to a lot of problems on this forum. Puzzling, but I'm concerned that if there isn't a solution beyond forcing transcoding I won't be able to use Emby for direct-play at all. It's essentially the only reason I'm paying for premiere, lol.

 

Thanks again for the eyes and suggestions.

 

 

 

Edit: Direct-play is a thing in the Theater app!!! and it works!!! no pausing so far on the most problematic content....

Edited April 26, 2019 by vhodh

[vhodh 0]

Ok, so if the Shield is the only one pausing with direct-play I'm happy to rule that out for now.

 

Direct-streaming is causing the same symptoms as the known Shield bug, but this occurs in Firefox and Chrome on multiple devices.

 

Totally unrelated? Similar network-sending-an-update-and-it-gets-lost type of issue? This updated post on the official geforce forums (https://forums.geforce.com/default/topic/1091469/shield-tv/shield-plex-playback-issues-on-7-2/11/) shows that a Plex update in their beta channel solved the issue (forum moderator confirms):

 

 

 

Does this mean that a software solution is viable and can be expected from Emby? If one is possible, I'd love to hear any updates or progress on it. From the previously mentioned posts on this issue here on the community forums the response seems to be "wait for nvidia".

 

This seems like an incredibly useful bugfix/update if the issues with the Shield TV can be solved along with these similar issues on other platforms.

 

If those two bugs are related, then a simple Shield firmware update is not going to fix my direct-streaming issues on Chrome and Firefox.

Edited April 26, 2019 by vhodh

[vhodh 0]

Here we go again!

 

More log files since I'm not getting any more feedback, and maybe that's why? Since my last log files were not downloaded, I'm doubting that's the reason, but regardless, here they are.

 

I performed 4 tests in a row on Firefox and Chrome. Two sets of auto-plays, then starting the stream and dropping bitrate first to 1080p50mbit (purposely above transcode), then finally at 1080p20mbit (to force transcode). All tests were performed over an internal LAN.

 

Overview:

 

Server: EmbyServer in running in host in a container on unraid

Client: Chrome 74, Firefox 67 on a Windows 10 desktop - nvidia 1070/i7 3770k

Network: UniFi gigabit LAN - all devices on LAN @ 1Gbps

Issue: Pausing occurs shortly after playback in all instances except when transcoding is forced.

Solution: Force transcoding via client options during playback / use a client that supports true direct-play, NOT direct-stream (Emby Theater)

 

Logs are attached in zip due to number of files.

 

Thanks again for any help offered.

logs.zip

Edited May 1, 2019 by vhodh

[Luke 37065]

Can you provide a sample video for testing?

[vhodh 0]

Can you provide a sample video for testing?

 

Hi Luke,

 

Thanks for your reply.

 

Unfortunately, none of the media that I current archive is public-domain, therefore I am not permitted to share even portions of files due to federal legislation in my country.

 

I will attempt to find a public-domain video that is high-bitrate to test and can ... link you that file somewhere?

 

The issue seems to occur on both H264 and H265 videos, so my best guess is the bitrate being the major culprit. I'll want to test both, I guess.

 

Lastly, my logs have yet to be downloaded again, so I'm puzzled as to what other information is actually needed from me in order to troubleshoot this problem efficiently.

 

Cheers.

[Luke 37065]

If you think the bitrate is the culprit then i would suggest lowering the in-app quality setting. This will use transcoding to reduce the bitrate. You can also try direct playing lower bitrate videos. Please let us know if this helps. Thanks.