TristenP 0 Posted November 5, 2021 Posted November 5, 2021 (edited) Hello how i can optimize the preload from videos (cache) with nginx and cloudflare ? and how i can cache the covers becouse for me is little slow loading ? best regards Edited November 5, 2021 by TristenP
Luke 42079 Posted November 5, 2021 Posted November 5, 2021 HI, I would suggest comparing your nginx configuration to that of @pir8radio
pir8radio 1312 Posted November 5, 2021 Posted November 5, 2021 (edited) 9 hours ago, TristenP said: Hello how i can optimize the preload from videos (cache) with nginx and cloudflare ? and how i can cache the covers becouse for me is little slow loading ? best regards i forget where the post is, but I created a cloudflare settings example somewhere... maybe ill find and add that post to my profile when you click my avatar.. If you transcode video you cant really cache the video.. due to you could have 5 users with 5 devices each, all transcoding the same movie into a different format for whatever reason, size, mbps, audio, etc.. you could be caching like 25 versions of the same movie.. cloudflare doesnt like caching video, and if you do it will be very slow to load the first play.. you should bypass all video transcoding. Ill find the link and post it here too... Edited November 5, 2021 by pir8radio
TristenP 0 Posted November 6, 2021 Author Posted November 6, 2021 Hello thanks for the help. Im using nginx config from dapper. I testet the config from pir8radio too but on his config is the loading from videos very good but the caching from the cover is much lower ( i dont can say why) I switched now to yours cloudflare settings and im testing it now.I have one issue with speedtest ( screenshot uploaded ) Thank you for the hard work. nginx confg : Spoiler worker_processes auto; error_log logs/error.log; events { worker_connections 4096; } http { include mime.types; default_type application/octet-stream; server_names_hash_bucket_size 64; server_tokens off; access_log off; #access logs disabled include cloudflare-allow.conf; #only accept cloudflare IPs deny all; #block all IPs except cloudflare ssl_session_cache shared:SSL:20m; ssl_session_timeout 180m; ssl_session_tickets on; ssl_early_data on; gzip on; gzip_disable "msie6"; gzip_comp_level 6; gzip_min_length 1100; gzip_buffers 16 8k; gzip_proxied any; gzip_types text/plain text/css text/js text/xml text/javascript application/javascript application/x-javascript application/json application/xml application/rss+xml image/svg+xml; tcp_nodelay on; server { listen 443 ssl http2; #only listen for https server_name xxxxxxxx; #domain name here include cloudflare-allow.conf; deny all; auth_basic "Auth"; auth_basic_user_file .htpasswdd; #additional httpauth password protection ssl_certificate ssl/cert.pem; ssl_certificate_key ssl/private.key; ssl_protocols TLSv1.3 TLSv1.2; #prefers v1.3 with minimum v1.2 ssl_prefer_server_ciphers on; ssl_early_data on; ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305:ECDHE+AES128:RSA+AES128:ECDHE+AES256:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!DSS'; #only use ciphers with perfect-forward-secrecy and prefer low-overhead ciphers ssl_conf_command Options PrioritizeChaCha; #prefer chacha for mobile devices that dont have hardware AES ssl_conf_command Ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256; keepalive_timeout 70; resolver 1.1.1.1 1.0.0.1 valid=300s; #cloudflare DNS resolver_timeout 10s; ssl_stapling on; ssl_stapling_verify on; ssl_stapling_responder http://ocsp.cloudflare.com/; ssl_trusted_certificate ssl/certPlusOrigin.pem; #my cert + cloudflare certs combined in 1 file ssl_client_certificate ssl/origin-pull-ca.pem; #from cloudflare ssl_verify_client on; ssl_ocsp on; ssl_ocsp_responder http://ocsp.cloudflare.com/; ssl_ocsp_cache shared:OCSPCache:20m; ssl_dhparam ssl/dhparam.pem; ssl_ecdh_curve secp384r1; add_header Referrer-Policy "same-origin"; add_header X-Frame-Options SAMEORIGIN; add_header X-Content-Type-Options nosniff; add_header X-XSS-Protection "1; mode=block"; client_max_body_size 100M; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Early-Data $ssl_early_data; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_redirect off; proxy_buffering off; location / { auth_basic off; #disable additional httpauth password for emby only proxy_pass http://xxxxxx:8080; include cloudflare-allow.conf; deny all; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header Early-Data $ssl_early_data; proxy_hide_header X-Powered-By; proxy_set_header Range $http_range; proxy_set_header If-Range $http_if_range; proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Ssl on; } } }
pir8radio 1312 Posted November 6, 2021 Posted November 6, 2021 13 hours ago, TristenP said: Hello thanks for the help. Im using nginx config from dapper. I testet the config from pir8radio too but on his config is the loading from videos very good but the caching from the cover is much lower ( i dont can say why) I switched now to yours cloudflare settings and im testing it now.I have one issue with speedtest ( screenshot uploaded ) Thank you for the hard work. nginx confg : Reveal hidden contents worker_processes auto; error_log logs/error.log; events { worker_connections 4096; } http { include mime.types; default_type application/octet-stream; server_names_hash_bucket_size 64; server_tokens off; access_log off; #access logs disabled include cloudflare-allow.conf; #only accept cloudflare IPs deny all; #block all IPs except cloudflare ssl_session_cache shared:SSL:20m; ssl_session_timeout 180m; ssl_session_tickets on; ssl_early_data on; gzip on; gzip_disable "msie6"; gzip_comp_level 6; gzip_min_length 1100; gzip_buffers 16 8k; gzip_proxied any; gzip_types text/plain text/css text/js text/xml text/javascript application/javascript application/x-javascript application/json application/xml application/rss+xml image/svg+xml; tcp_nodelay on; server { listen 443 ssl http2; #only listen for https server_name xxxxxxxx; #domain name here include cloudflare-allow.conf; deny all; auth_basic "Auth"; auth_basic_user_file .htpasswdd; #additional httpauth password protection ssl_certificate ssl/cert.pem; ssl_certificate_key ssl/private.key; ssl_protocols TLSv1.3 TLSv1.2; #prefers v1.3 with minimum v1.2 ssl_prefer_server_ciphers on; ssl_early_data on; ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305:ECDHE+AES128:RSA+AES128:ECDHE+AES256:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!DSS'; #only use ciphers with perfect-forward-secrecy and prefer low-overhead ciphers ssl_conf_command Options PrioritizeChaCha; #prefer chacha for mobile devices that dont have hardware AES ssl_conf_command Ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256; keepalive_timeout 70; resolver 1.1.1.1 1.0.0.1 valid=300s; #cloudflare DNS resolver_timeout 10s; ssl_stapling on; ssl_stapling_verify on; ssl_stapling_responder http://ocsp.cloudflare.com/; ssl_trusted_certificate ssl/certPlusOrigin.pem; #my cert + cloudflare certs combined in 1 file ssl_client_certificate ssl/origin-pull-ca.pem; #from cloudflare ssl_verify_client on; ssl_ocsp on; ssl_ocsp_responder http://ocsp.cloudflare.com/; ssl_ocsp_cache shared:OCSPCache:20m; ssl_dhparam ssl/dhparam.pem; ssl_ecdh_curve secp384r1; add_header Referrer-Policy "same-origin"; add_header X-Frame-Options SAMEORIGIN; add_header X-Content-Type-Options nosniff; add_header X-XSS-Protection "1; mode=block"; client_max_body_size 100M; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Early-Data $ssl_early_data; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_redirect off; proxy_buffering off; location / { auth_basic off; #disable additional httpauth password for emby only proxy_pass http://xxxxxx:8080; include cloudflare-allow.conf; deny all; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header Early-Data $ssl_early_data; proxy_hide_header X-Powered-By; proxy_set_header Range $http_range; proxy_set_header If-Range $http_if_range; proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Ssl on; } } } where is this error from? what tool are you trying to use? "speed test"
TristenP 0 Posted November 7, 2021 Author Posted November 7, 2021 7 hours ago, pir8radio said: where is this error from? what tool are you trying to use? "speed test" cloudflare > Speed
redrobot2121 0 Posted November 7, 2021 Posted November 7, 2021 Any help on how to preload/keep in buffer a big chunk of direct play media? with caddy or nginx... using Cloudflare cdn
Carlo 4561 Posted November 8, 2021 Posted November 8, 2021 No that is directly against Cloudflare's terms of service. Do not try to cache video with them if on the free plan. They have been giving warnings in the forums and talking to a rep who said they may have to start imposing bans on personal video servers from their system on the free plan. Play by the rules properly and read their Terms of Service so you know what you are allowed to do and what you are not. I worked with a guy yesterday evening into early morning remotely from Germany for hours to find out his site was banned on Cloudflare. Technically right now anyone here is in violation as is of section 2.8 I believe. If you need to lower your bandwidth by caching some video files consider organizing your library with new stuff popular stuff in it's own library that is easy to identify. Then setup a remote proxy yourself caching any direct play (mp4, mkv, avi) video in that featured section. That will keep your popular movies in the cache without flushing them as fast and allow you to setup up a proxy with minimal/modest storage needs vs huge amounts.
pir8radio 1312 Posted November 8, 2021 Posted November 8, 2021 8 hours ago, cayars said: No that is directly against Cloudflare's terms of service. Do not try to cache video with them if on the free plan. They have been giving warnings in the forums and talking to a rep who said they may have to start imposing bans on personal video servers from their system on the free plan. Play by the rules properly and read their Terms of Service so you know what you are allowed to do and what you are not. I worked with a guy yesterday evening into early morning remotely from Germany for hours to find out his site was banned on Cloudflare. Technically right now anyone here is in violation as is of section 2.8 I believe. If you need to lower your bandwidth by caching some video files consider organizing your library with new stuff popular stuff in it's own library that is easy to identify. Then setup a remote proxy yourself caching any direct play (mp4, mkv, avi) video in that featured section. That will keep your popular movies in the cache without flushing them as fast and allow you to setup up a proxy with minimal/modest storage needs vs huge amounts. thats not good... i would have to rework my server to bypass the proxy for video.. and it would expose our server ip address... wonder if they would let it slide for the 20/month service lol. this person you were helping, were they an emby server or something else?
Carlo 4561 Posted November 8, 2021 Posted November 8, 2021 Ubuntu server running in at Hetzner with Emby 4.6.4. etzner itself is at the top of the list of most filtered at Cloudflare as lots of bot/intrusions and things come from them. Something I want to try is rewriting the URL of any video coming from EMBY so basically everything goes through Cloudflare except for actual video which would point directly back to the Emby Server. I'm not sure how the clients will react to that or even a browser client as it likely depends on the way the cookies are written. Of course that exposes your IP but better than loosing Cloudflare completely. Be interesting to see how that will work and what obstacles there will be.
redrobot2121 0 Posted November 10, 2021 Posted November 10, 2021 On 11/8/2021 at 7:22 PM, pir8radio said: thats not good... i would have to rework my server to bypass the proxy for video.. and it would expose our server ip address... wonder if they would let it slide for the 20/month service lol. this person you were helping, were they an emby server or something else? isn't *domain/*videos/*/* sufficient ?
Carlo 4561 Posted November 10, 2021 Posted November 10, 2021 Not sure what you mean? If you mean for the bypass rule that will only help you for videos and not audio streams which require their own rule. Keep in mind this is only for Emby. If using other software through Cloudflare you will need to check what rules are needed for those programs. You do not need the first * If you just use your domain name and do not specify a sub domain it will match all subs per their documentation.
pir8radio 1312 Posted November 10, 2021 Posted November 10, 2021 (edited) 6 hours ago, redrobot2121 said: isn't *domain/*videos/*/* sufficient ? No technically cloudflare doesn't want you even passing video through their service.. You would have to create another DNS entry (that exposes your real server IP) that totally bypasses cloudflare. They don't want to deal with the bandwidth passing through their network.. so the bypass rule just makes it not get cached, but the video still passes through CF.. You would need to take that second DNS entry that goes directly to your emby server, and remove the orange cloud from it, then have a rule or something that all video comes from that other server url even though its technically the same emby server.. or do away with cloudflare all together. or pay for cloudflare. They do look like they are cracking down on this.. i just came across a web site with a video that had this: Edited November 10, 2021 by pir8radio
C.S. 93 Posted November 11, 2021 Posted November 11, 2021 So how much is the cheapest CF plan we can push tons of video through?
Carlo 4561 Posted November 11, 2021 Posted November 11, 2021 You're going to pay for the bandwidth so as much as you can afford to pay. Unless you have people playing the same videos the same way (ie all direct play) there is nothing to cache so you would be better off not paying for something that does little to help you. I've been thinking about a possible solution to this to stay on the good side of Cloudflare, but it would require a true cert and not the Cloudflare cert that's only good between them and you. With the worker process they have we could rewrite any URLS on the fly with audio or video to point back directly to your server. This way that type of traffic never goes through Cloudflare. I'm not sure how Emby Server will react to this. You could probably do the same thing with nginx or other reverse proxy as well running locally. This way if Emby doesn't like getting responses from two different IPs NGINX could rewrite things both coming and going so Emby wouldn't know the difference.
C.S. 93 Posted November 11, 2021 Posted November 11, 2021 What if we all agree to keep our max bitrate at a reasonable level?
pir8radio 1312 Posted November 11, 2021 Posted November 11, 2021 9 hours ago, cayars said: You're going to pay for the bandwidth so as much as you can afford to pay. Unless you have people playing the same videos the same way (ie all direct play) there is nothing to cache so you would be better off not paying for something that does little to help you. I've been thinking about a possible solution to this to stay on the good side of Cloudflare, but it would require a true cert and not the Cloudflare cert that's only good between them and you. With the worker process they have we could rewrite any URLS on the fly with audio or video to point back directly to your server. This way that type of traffic never goes through Cloudflare. I'm not sure how Emby Server will react to this. You could probably do the same thing with nginx or other reverse proxy as well running locally. This way if Emby doesn't like getting responses from two different IPs NGINX could rewrite things both coming and going so Emby wouldn't know the difference. any bypass solution would still expose your real server IP address.. if that is not a concern then its pretty easy to bypass the video.. We will have to see if they start blocking us... I'm not sure how they are detecting it right now.. maybe they are targeting cache users, it would be more work for them to detect it when its un-encrypted at the nginx side since it would just look like a pipe full of bandwidth encrypted... right now i think they are just seeing what big files are sitting on their servers and blocking...
pir8radio 1312 Posted November 11, 2021 Posted November 11, 2021 (edited) 11 hours ago, C.S. said: So how much is the cheapest CF plan we can push tons of video through? don't quote me on this. but i think the lowest level is fine.. just being a paying customer will help i assume, vs free. $20/month its another bill i don't need, but if forced I will probably pay.. you get some other bells and whistles but not sure if it would end up being worth $240 a year. Edited November 11, 2021 by pir8radio
C.S. 93 Posted November 11, 2021 Posted November 11, 2021 4 hours ago, pir8radio said: We will have to see if they start blocking us... I'm not sure how they are detecting it right now.. maybe they are targeting cache users, it would be more work for them to detect it when its un-encrypted at the nginx side since it would just look like a pipe full of bandwidth encrypted... right now i think they are just seeing what big files are sitting on their servers and blocking... I'd guess they would start with the domains pushing the most data. I'm always in the hundreds of GB per month according to their monthly email. I think I went over a TB once or twice, but nobody has ever said it's too much. But I'm not caching anything so maybe that's a factor. As far as detecting it, couldn't they do that just by the traffic pattern?
Carlo 4561 Posted November 11, 2021 Posted November 11, 2021 6 hours ago, pir8radio said: don't quote me on this. but i think the lowest level is fine.. just being a paying customer will help i assume, vs free. $20/month its another bill i don't need, but if forced I will probably pay.. you get some other bells and whistles but not sure if it would end up being worth $240 a year. Same for me. $240 a year is better spent on many other things. I'd sooner setup my own remote proxy and have control for 1/4 to 1/2 the price which would work fine for me. I could easily setup a wire guard tunnel as well and not have any ports open. I like having all the images cached for speedy page loads when remote but not for that price for my home server. 2 hours ago, C.S. said: I'd guess they would start with the domains pushing the most data. I'm always in the hundreds of GB per month according to their monthly email. I think I went over a TB once or twice, but nobody has ever said it's too much. But I'm not caching anything so maybe that's a factor. As far as detecting it, couldn't they do that just by the traffic pattern? 6 hours ago, pir8radio said: any bypass solution would still expose your real server IP address.. if that is not a concern then its pretty easy to bypass the video.. We will have to see if they start blocking us... I'm not sure how they are detecting it right now.. maybe they are targeting cache users, it would be more work for them to detect it when its un-encrypted at the nginx side since it would just look like a pipe full of bandwidth encrypted... right now i think they are just seeing what big files are sitting on their servers and blocking... They should be able to easily do both, looking at traffic as well as storage. They would easily be able to tell by running reports if you're running a media server and which one based on the URL patterns. So the question I have is if they start cracking down harder on this if they just "blacklist" the heaviest violators on a case by case basis or if they get tired of that and blacklist certain software like media servers in general.
TristenP 0 Posted November 19, 2021 Author Posted November 19, 2021 (edited) i mixed the configs works thanks for help. I have one more question can you caching the video with nginx ? or is the performance then bad? Edited November 19, 2021 by TristenP
pir8radio 1312 Posted November 22, 2021 Posted November 22, 2021 On 11/18/2021 at 6:47 PM, TristenP said: i mixed the configs works thanks for help. I have one more question can you caching the video with nginx ? or is the performance then bad? ehh you can... but most times nginx is running on the same pc as emby.. so you really don't get much advantage.. is there a particular issue you are trying to overcome? maybe we can give you some better solutions if so?
Carlo 4561 Posted November 23, 2021 Posted November 23, 2021 That would likely only work for direct streamed video as well and I can't see this helping really at all. Not if the proxy is in the same location. 1
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now