Reverse proxy issues after updating to 4.4.2.0

[Ponyo 22]

Hi,

 

I'm running Emby behind a NGINX reverse proxy and everything was working fine until the 4.4.2.0 update a couple of days ago.

 

I can access Emby remotely just like before but playing any media does not work anymore.

 

Browser:

Trying to download a picture by long pressing a thumbnail: Seems to transfer a little bit and then goes into "waiting for network" in the notifications bar on Android. Similar thing happens on IOS.

Video playback on Android results in PLayback errors no compatible streams are currently available or HeaderPlaybackError PlaybackErrorNoCompatibleStream. Only tested on Android.

 

Android app:

I don't have premier so can't test picture downloads but video playback has the endless loading circle. The ffmpeg process is started on the server and hangs after closing the app.

 

When connecting locally to ip:8096 everyhing works fine.

 

On the dashboard remote access shows mydomain:8920. Tried changing it to 443 but no luck.

 

I attached the logs from testing on the browser. My NGINX config is very similar to what is in the NGINX topic on the forum though I believe I have it set up to block old tls versions.

 

Edit: I've checked on my desktop with the browser dev tools enabled and when trying to stream a video all the m3u8 links fail.

When trying to save a picture, Firefox trows an error on the save menu Opening mypicture.jpg C:\path\local\temp\picture.jpg.part could not be saved, because the source file could not be read.

logs.txt

Edited April 10, 2020 by Ponyo

[Ponyo 22]

My NGINX config in case it matters

server {

#        root /var/www/emby/html;
#        index index.html index.htm index.nginx-debian.html;


#        listen [::]:443 ssl http2 ipv6only=on; # managed by Certbot
        listen 443 ssl http2; # managed by Certbot
        ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem; # managed by Certbot
        include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
        gzip off;

        server_name mydomain.com;


        location / {
        proxy_pass http://192.168.0.15:8096;
        proxy_set_header Range $http_range;
        proxy_set_header If-Range $http_if_range;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # Next three lines allow websockets
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $http_connection; #fixed connection header
        # try_files $uri $uri/ =404;

        # Custom error page
        proxy_intercept_errors on;
        proxy_redirect off;
        error_page 401 403 404 /404.html;
        }


        location /404.html {
        root /var/www/emby/html/;
        }

        location /404.jpg {
        root /var/www/emby/html/;
        }

}




server {
    if ($host = mydomain.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        listen 80;
#        listen [::]:80;

        server_name mydomain.com;
    return 404; # managed by Certbot


}

 

[Luke 42086]

Have you compared your reverse proxy config to that of @@pir8radio?

[Ponyo 22]

My config is very similar as I mentioned. I changed it to be pretty much the same except for the http block, which I don't have. No change.

 

The config was working fine until the 4.4.2.0 update a couple of days ago. I also don't have any problems accessing Emby, just starting a stream or downloading pictures.

Edited April 10, 2020 by Ponyo

[Ponyo 22]

Some further information:

 

Video playback only seems to fail on larger files. 3GB+ files fail while smaller ones, 1.5GB or so, work. I don't have that many test files but the smaller ones I have all work while bigger files don't.

 

In the browser dev console it is full of m3u8 errors.

Request URL: https://mydomain.com/emby/videos/35859/main.m3u8?DeviceId=0831f0f0-5069-4936-b710-a7b9471a848e&MediaSourceId=7fc6857f938f0ba5f8f3e0ba42f9060c&PlaySessionId=a3fa15e5555f4b3cbabec9aaacfce41e&api_key=e0929ee004754f7fbe5eb9c68072d6fd&VideoCodec=h264&AudioCodec=mp3,aac&VideoBitrate=3991000&AudioBitrate=384000&AudioStreamIndex=1&SubtitleStreamIndex=2&SubtitleMethod=Hls&TranscodingMaxAudioChannels=2&SegmentContainer=ts&MinSegments=1&BreakOnNonKeyFrames=True&ManifestSubtitles=vtt&h264-profile=high,main,baseline,constrainedbaseline,high10&h264-level=52&TranscodeReasons=ContainerBitrateExceedsLimit,DirectPlayError&allowVideoStreamCopy=false&allowAudioStreamCopy=false

Not sure why it wants to transcode because of bitrate because I set max bitrate to 100mbps. The dashboard also shows direct play though it never gets to the point where anything actually plays. There is also this edit to my first post:
 

Edit: I've checked on my desktop with the browser dev tools enabled and when trying to stream a video all the m3u8 links fail.

When trying to save a picture, Firefox trows an error on the save menu Opening mypicture.jpg C:\path\local\temp\picture.jpg.part could not be saved, because the source file could not be read.

 

 

Maybe this is an NGINX issue, but as the problems only happened after the 4.4.2.0 update, in that case something must have changed on the Emby side that isn't playing nice with NGINX.

 

edit: WRT to image downloads, it puts a partial file in my save location. If I open it it only has like 10 ~ 20% of the top part of the image. So something is causing downloads to stop after a while.  Again the problem is purely with streaming video and downloading images. The webinterface works fine, I can even view pictures through it etc.

Edited April 10, 2020 by Ponyo

[pir8radio 1312]

Some further information:

 

Video playback only seems to fail on larger files. 3GB+ files fail while smaller ones, 1.5GB or so, work. I don't have that many test files but the smaller ones I have all work while bigger files don't.

 

In the browser dev console it is full of m3u8 errors.

Request URL: https://mydomain.com/emby/videos/35859/main.m3u8?DeviceId=0831f0f0-5069-4936-b710-a7b9471a848e&MediaSourceId=7fc6857f938f0ba5f8f3e0ba42f9060c&PlaySessionId=a3fa15e5555f4b3cbabec9aaacfce41e&api_key=e0929ee004754f7fbe5eb9c68072d6fd&VideoCodec=h264&AudioCodec=mp3,aac&VideoBitrate=3991000&AudioBitrate=384000&AudioStreamIndex=1&SubtitleStreamIndex=2&SubtitleMethod=Hls&TranscodingMaxAudioChannels=2&SegmentContainer=ts&MinSegments=1&BreakOnNonKeyFrames=True&ManifestSubtitles=vtt&h264-profile=high,main,baseline,constrainedbaseline,high10&h264-level=52&TranscodeReasons=ContainerBitrateExceedsLimit,DirectPlayError&allowVideoStreamCopy=false&allowAudioStreamCopy=false

Not sure why it wants to transcode because of bitrate because I set max bitrate to 100mbps. The dashboard also shows direct play though it never gets to the point where anything actually plays. There is also this edit to my first post:

 

 

Maybe this is an NGINX issue, but as the problems only happened after the 4.4.2.0 update, in that case something must have changed on the Emby side that isn't playing nice with NGINX.

 

edit: WRT to image downloads, it puts a partial file in my save location. If I open it it only has like 10 ~ 20% of the top part of the image. So something is causing downloads to stop after a while.  Again the problem is purely with streaming video and downloading images. The webinterface works fine, I can even view pictures through it etc.

 

Where did your original config come from?   

 

look in your http section...    some things to look into:

may want to see what your gzip settings are as well.

    sendfile        off;  ## Sendfile not used in a proxy environment.

    proxy_connect_timeout 1h;
    proxy_send_timeout 1h;
    proxy_read_timeout 1h;
    tcp_nodelay on;  ## Sends data as fast as it can not buffering large chunks, saves about 200ms per request.

Edited April 10, 2020 by pir8radio

[Ponyo 22]

Made it myself for another project. I probably did pick some bits from you though I don't remember, its been a while since I made the config.

 

I don't believe those settings are related to the issue but I changed them anyway and the result is the same. The thing is, everything has been working fine for a year until the latest update. Now I'm not saying this is purely a Emby issue, but given the fact that the only thing that changed when things stopped working is Emby, at the very least there must have been a change to Emby that requires a change to NGINX.

 

What changed with the 4.4.2.0 update?

[Ponyo 22]

Okay found the problem. If you look at developer tools you'll see Emby is trying to load multiple hls.min.js files and in my case this was failing. At first I got a http2 error so I disabled http2. Then the error on downloading the hls.min.js files changed to content length mismatch.

 

I added:

proxy_buffering off;

to nginx.conf and as far as I can tell everything is now working again.

 

What changed is that initially the browser could only retrieve several hundred KB of the hls.min.js files while they can be up to several MB.

 

I noticed Night's config has that same setting for Plex but at a very quick glance I couldn't find anybody using it for Emby.

 

Which makes me wonder if there was a change on Emby's end that now might require this setting while previously not needed it.

 

[Luke 42086]

Glad you figured it out.

[pir8radio 1312]

Okay found the problem. If you look at developer tools you'll see Emby is trying to load multiple hls.min.js files and in my case this was failing. At first I got a http2 error so I disabled http2. Then the error on downloading the hls.min.js files changed to content length mismatch.

 

I added:

proxy_buffering off;

to nginx.conf and as far as I can tell everything is now working again.

 

What changed is that initially the browser could only retrieve several hundred KB of the hls.min.js files while they can be up to several MB.

 

I noticed Night's config has that same setting for Plex but at a very quick glance I couldn't find anybody using it for Emby.

 

Which makes me wonder if there was a change on Emby's end that now might require this setting while previously not needed it

 

 

 

 

Good job, I thought i had proxy buffering on but i just looked i do have it disabled within the location block,  not in the http block, sorry, but glad you found it!

 

Going to edit my config posted on here now, to move it from the "location" section to "server" section    on my example config.    There could be some issues with it in location if you have multiple locations and forget to add it to the other blocks..   

Edited April 11, 2020 by pir8radio