Jump to content

HOW TO: Recommended Cloudflare Settings


pir8radio

Recommended Posts

pir8radio
10 minutes ago, vaise said:

In my spare time, I have been looking for backup solutions in case this happens to me.  I have a spare domain configured with a swag reverse proxy etc etc as noted above that is 'ready to go' but I wondered if there is anything more.

I notice the AWS Cloudfront allows 1TB transfer a month on their free plan, which is more than enough.  It has DDOS protection, geo filtering etc.  So..... I had a play with it. 

Signed up for a free account.  Created a dns entry to my test domain called 'embyorigin.mydomain.com', then created a cloudfront 'distribution' - which was very easy, just enter the subdomain, a few clicks and it was deployed to all their servers.  you get a random domain name linked to your origin host (i.e like abcdef12345.cloudfront.net.  I hit that URL and got the emby remote login screen - which is great, but the password seems to always be incorrect (when I know it is correct).  So hit a wall on that front.  If anyone with more skills/experience than me can get this working, then this may be a great CF replacement ?  You can link your own subdomain name to it also (i did not do that until I get the base working).

I setup the geograhic restrictions :

I found you can create 'invalidations' - which are the equivalent of the non cache page rules we set up in cloudflare.  Not sure if this would work - but if you have 1TB a month, that would cover all my video anyway.

 

the main thing I can't seem to find with any other CDN service is that cloudflare proxies all of your connections..  meaning your IP is totally hidden (assuming things are setup correctly) I don't know (haven't looked lately) of any other CDN that proxies all traffic through their servers and can do it for free like CF, which allows for the seamless ssl cert and all of that good stuff...  let us know how you make out!

Link to comment
Share on other sites

muzicman0

I've been playing around with Tailscale for the last couple days.  Easy enough to set up, and for me, the 20 device limit on the free account is no issue, I only share with a few family members.  I am most concerned with bandwidth and having to have an app on the client device.  Going to test this Sunday with my brother who lives 1600 miles away.  

Big thing for me is that I am behind CGNAT, so I can't use most of the solutions in this post.  I have been using Cloudflare Tunnels, and it works great, but like others, I want a backup, just in case!

Link to comment
Share on other sites

30 minutes ago, muzicman0 said:

I've been playing around with Tailscale for the last couple days.  Easy enough to set up, and for me, the 20 device limit on the free account is no issue, I only share with a few family members.  I am most concerned with bandwidth and having to have an app on the client device.  Going to test this Sunday with my brother who lives 1600 miles away.  

Big thing for me is that I am behind CGNAT, so I can't use most of the solutions in this post.  I have been using Cloudflare Tunnels, and it works great, but like others, I want a backup, just in case!

I also use tailscale with advertise routes for access to everything else but emby.  Its great that my synchronized chrome bookmarks work the same out of the house as they do in the house for all my local services (arr's, cctv, ssh, work servers).  Not so much for emby however as all the friends/families use roku's and other smart devices with no tailscale client.  I have two tailscale devices with the same advertise routes, should one go down, I should be able to still connect to stuff.  It works so well, I removed all my subdomains for the 'arrs - no need anymore.

Link to comment
Share on other sites

58 minutes ago, pir8radio said:

the main thing I can't seem to find with any other CDN service is that cloudflare proxies all of your connections..  meaning your IP is totally hidden (assuming things are setup correctly) I don't know (haven't looked lately) of any other CDN that proxies all traffic through their servers and can do it for free like CF, which allows for the seamless ssl cert and all of that good stuff...  let us know how you make out!

I agree, nothing can match CF - but we seem to all be just waiting for the axe to drop - and Murphies Law say it will be at the worse time possible......Like when i am overseas!

  • Haha 1
Link to comment
Share on other sites

I am still struggling with the password on AWS Cloudfront.  

I have set it up with am AWS SSL, with my normal domain emby.mydomain.com.  The website comes up ok, but just cant get past the password screen (note, this works to the normal emby server url) - The password is correct - has to be something on the AWS Caching side ?

image.png.ee10a48515bf239c85f5784ece11e90f.png

In debugging the console, maybe it is something to do with this - this is not normal right ?:

image.png.6ae5149c8bf222de08336e02dc6650bb.png

Link to comment
Share on other sites

Flintfamily
On 8/16/2022 at 10:46 PM, pir8radio said:

odd that they blocked you on the pro plan..  i would think unless you were doing something crazy they would leave you alone.  

Sounds like I saved myself $20 given I was on free and thought perhaps pro plan would have more flexibility, seems not. Would an option not be running some script to generate html page calls thus bumping the html to video ratio? Similar to your cypto services.

Link to comment
Share on other sites

Flintfamily
16 minutes ago, Flexeire said:

Mine still working for now 

 

8TB average per month traffic on free tier

caching images and ignoring video 

Makes no sense, I'm nowhere near 8TB.

Link to comment
Share on other sites

1 hour ago, Flexeire said:

Mine still working for now 

 

8TB average per month traffic on free tier

caching images and ignoring video 

OMG!!!!!!  8TB!  You must be running a netflix service 😉 

Link to comment
Share on other sites

Just now, vaise said:

OMG!!!!!!  8TB!  You must be running a netflix service 😉 

Honestly no! 
 

8 family and friends! 

Link to comment
Share on other sites

Jeez.  I limit my out of network connections to 5Mbps.  I only have 20Mbps upload.   25 named users in emby - family and friends, usually about 8 a day regular watchers of stuff......

Link to comment
Share on other sites

All of us use 4K so that’s probably the reason. Plus I have 1Gb symmetrical so all good. 
 

tbf 8 users but each has a limit of 3 set  but max at any one time I ever seen is 10 concurrent. 

Edited by Flexeire
Link to comment
Share on other sites

8 minutes ago, Flexeire said:

All of us use 4K so that’s probably the reason. Plus I have 1Gb symmetrical so all good. 
 

tbf 8 users but each has a limit of 3 set  but max at any one time I ever seen is 10 concurrent. 

The stuff dreams are made of in Australia!!!!!  Third world in seems like most of the time.  Our PM once said ‘No one needs faster that 25mbps’

  • Like 1
  • Haha 1
Link to comment
Share on other sites

Soooooo.  Sorry in advance for this long post. 

In the never ending task of getting my 'backup when cloudflare bans' new nginx system (using swag), but matching CF where possible, I have a small issue.

I have geoIP2 database in nginx for the same geo blocking CF provides (not at the router like CF did) and I have fail2ban, and added jail for emby - should help with DDOS (but also not at the router level like CF did).

In the CF WAF, I had 'block all bots', and that never caused an issue.  For this new backup nginx system, I got a load of bot lists from online here - https://www.knthost.com/nginx/blocking-bots-with-nginx and entered all that into nginx.  Once this went live last night, I had a few Google TV users (CCWGTV) report they cant connect to emby.  In a process of elimination, this was tracked down to 'data_collectors.conf', which has 273 bots in it.  That is way to many to go through one by one.  Does anyone know off hand what bots below that emby via a google TV device to connect ?  I cant seem to find anything in the logs that is helping.

map $http_user_agent $data_collectors {
default 0;
"~007ac9" 1;
"~008" 1;
"~200PleaseBot" 1;
"~A6-Indexer" 1;
"~activesearchresults.com" 1;
"~adbeat_bot" 1;
"~AddThis.com" 1;
"~ADmantX" 1;
"~AhrefsBot" 1;
"~aiHitBot" 1;
"~AlphaBot" 1;
"~archive.org_bot" 1;
"~Attentio/Nutch" 1;
"~BacklinkCrawler" 1;
"~BiggerBetter" 1;
"~BIXOCRAWLER" 1;
"~Blackboard Safeassign" 1;
"~BLEXBot" 1;
"~BPImageWalker" 1;
"~Barkrowler/0.7" 1;
"~BDCbot/1.0" 1;
"~Brodie/1.0" 1;
"~BUbiNG" 1;
"~Buck/2.2" 1;
"~BusinessSeek.biz_Spider" 1;
"~Buzzbot/1.0" 1;
"~BuzzSumo" 1;
"~calculon spider" 1;
"~CareerBot" 1;
"~CATExplorador" 1;
"~CCBot" 1;
"~CheckMarkNetwork" 1;
"~changedetection.com" 1;
"~CipaCrawler" 1;
"~CityGridMedia/1.0" 1;
"~Clickagy Intelligence Bot v2" 1;
"~CloudServerMarketSpider" 1;
"~CMS Crawler" 1;
"~cmscrawler" 1;
"~cognitiveseo" 1;
"~collection@infegy.com" 1;
"~CommonCrawler Node" 1;
"~Companybook-Crawler" 1;
"~ContextAd Bot" 1;
"~Contacts Crawler" 1;
"~CoPubbot" 1;
"~Corax" 1;
"~crawler@fast.no" 1;
"~Crawlera/" 1;
"~CRAZYWEBCRAWLER" 1;
"~CSS Certificate Spider" 1;
"~CybEye" 1;
"~datagnionbot" 1;
"~Datanyze" 1;
"~Dataprovider" 1;
"~datasift.com/bot" 1;
"~DBot" 1;
"~Diffbot" 1;
"~Digincore" 1;
"~dlvr.it" 1;
"~Domain Re-Animator Bot" 1;
"~DomainAppender" 1;
"~DomainCheck.io Crawler" 1;
"~DomainCrawler/3.0" 1;
"~DomainMacroCrawler/0.1" 1;
"~domainsigma.com/whois" 1;
"~DotBot/1.1" 1;
"~eCairn-Grabber/1.0" 1;
"~eContext/1.0" 1;
"~electricmonk/3.2.0" 1;
"~EasyBib AutoCite" 1;
"~Elmer, the Thinglink ImageBot" 1;
"~Embedly" 1;
"~etoolsbot/1.0" 1;
"~EveryoneSocialBot/1.0" 1;
"~exabot" 1;
"~Exabot/3.0" 1;
"~Experibot_v1" 1;
"~ExtLinksBot/1.5" 1;
"~FactSet Research Systems" 1;
"~Researchscan/t12sns" 1;
"~Faveeo/1.0" 1;
"~FeedChecker-Zocle" 1;
"~FlipboardProxy" 1;
"~GarlikCrawler" 1;
"~Genieo" 1;
"~GetIntent Crawler" 1;
"~glindahl-cocrawler" 1;
"~Gluten Free Crawler/1.0" 1;
"~golabbot" 1;
"~Grammarly/1.0" 1;
"~GrapeshotCrawler" 1;
"~GroupHigh/1.0" 1;
"~HubSpot Webcrawler" 1;
"~HubSpot Links Crawler" 1;
"~HubSpot Connect" 1;
"~http://search.thunderstone.com/texis/websearch/about.html" 1;
"~ia_archiver" 1;
"~ICC-Crawler" 1;
"~IDBot" 1;
"~https://www.inboundlinks.win/InboundLinks/" 1;
"~IndeedBot" 1;
"~Infegy" 1;
"~InfegyAtlas/1.0" 1;
"~ips-agent" 1;
"~it2media-domain-crawler" 1;
"~James" 1;
"~Kraken/0.1" 1;
"~kdslbot" 1;
"~LightspeedSystemsCrawler" 1;
"~LinkCheck by Siteimprove.com" 1;
"~linkdexbot" 1;
"~LinkisBot/1.0" 1;
"~LinkpadBot" 1;
"~LinkWalker/3.0" 1;
"~Lipperhey-Kaus-Australis/5.0" 1;
"~Lipperhey" 1;
"~LivelapBot" 1;
"~LoadTimeBot" 1;
"~ltbot/0.3.6" 1;
"~MacInroy" 1;
"~magpie-crawler" 1;
"~MauiBot (crawler.feedback+wc@gmail.com)" 1;
"~*MauiBot (crawler.feedback+wc@gmail.com)" 1;
"~MauiBot" 1;
"~MaxPointCrawler" 1;
"~MBCrawler" 1;
"~Mediametric" 1;
"~meanpathbot" 1;
"~Mediatoolkitbot" 1;
"~MegaIndex.com/2.0" 1;
"~MegaIndex.ru/2.0" 1;
"~MerchantCentricBot" 1;
"~MixBot" 1;
"~MixrankBot" 1;
"~MJ12bot" 1;
"~MXT/Nutch-1.12-SNAPSHOT" 1;
"~MuckRackFeedParser" 1;
"~NetcraftSurveyAgent" 1;
"~netcraft" 1;
"~netEstate FOAF crawler" 1;
"~netEstate NE Crawler" 1;
"~netEstate" 1;
"~NetLyzer FastProbe" 1;
"~NetSeer" 1;
"~NextGenSearchBot" 1;
"~NewShareCounts.com" 1;
"~nlpproject.info" 1;
"~nsrbot/1.0" 1;
"~oBot/2.3.1" 1;
"~online-webceo-bot" 1;
"~OnPageBot" 1;
"~OpenWebSpider" 1;
"~Opendi Screenshot Bot" 1;
"~OrgProbe/0.9.4" 1;
"~OutclicksBot" 1;
"~PageFreezer" 1;
"~PagesInventory" 1;
"~panscient.com" 1;
"~PaperLiBot" 1;
"~pdffillerbot/1.0" 1;
"~PercolateCrawler" 1;
"~PiplBot" 1;
"~PocketParser/2.0" 1;
"~PopScreen Bot" 1;
"~Primalbot" 1;
"~PrivacyAwareBot" 1;
"~PrivateSearch" 1;
"~probethenet.com scanner" 1;
"~profound.net" 1;
"~ProdvigatorBot/0.1" 1;
"~ProspectB2B/2.0" 1;
"~proximic" 1;
"~Pu_iN Crawler" 1;
"~R6_CommentReader" 1;
"~R6_FeedFetcher" 1;
"~RankActiveLinkBot" 1;
"~RavenCrawler" 1;
"~rbot.info" 1;
"~RebelMouse" 1;
"~redditbot" 1;
"~ReplazBot" 1;
"~RobotsChecker/0.5" 1;
"~rogerbot" 1;
"~RSSGraffiti" 1;
"~SafeDNSBot" 1;
"~SalesIntelligent" 1;
"~SCFCrawler" 1;
"~ScoutJet" 1;
"~Scrapyproject" 1;
"~Screaming Frog SEO Spider" 1;
"~scripted.com" 1;
"~SearchmetricsBot" 1;
"~semalt.com" 1;
"~semanticbot" 1;
"~SemrushBot" 1;
"~SEOdiver/1.0" 1;
"~seoengbot" 1;
"~seoengworldbot" 1;
"~SEOkicks" 1;
"~SEOkicks-Robot" 1;
"~Seologies/0.1" 1;
"~SEOlyticsCrawler" 1;
"~seoscanners.net" 1;
"~SEOstats" 1;
"~Shareaholicbot" 1;
"~ShowyouBot" 1;
"~SimplePie" 1;
"~simplereach/1.1" 1;
"~SISTRIX" 1;
"~SiteAnalyzerBot" 1;
"~SiteExplorer" 1;
"~Siteimprove.com" 1;
"~Slack-ImgProxy" 1;
"~Slackbot-LinkExpanding" 1;
"~Slackbot" 1;
"~SmabblerBot" 1;
"~SMTBot/1.0" 1;
"~socialmediascanner.eset.com" 1;
"~Socialradarbot" 1;
"~SocialRankIOBot" 1;
"~SocialSearcher" 1;
"~SolomonoBot" 1;
"~sourcescrub.com" 1;
"~spbot" 1;
"~Spiderbook" 1;
"~SpiderLing" 1;
"~SpiderTest" 1;
"~Spin3r" 1;
"~spotinfluence" 1;
"~Springshare Link Checker" 1;
"~sproutsocial.com" 1;
"~ssearch_bot" 1;
"~startmebot/1.0" 1;
"~Steeler" 1;
"~Steeler/3.5" 1;
"~StorygizeBot" 1;
"~SurdotlyBot/1.0" 1;
"~SurveyBot" 1;
"~Sysomos" 1;
"~TeeRaidBot" 1;
"~Telesphoreo" 1;
"~The Knowledge AI" 1;
"~thumbshots-de-bot" 1;
"~Traackr.com" 1;
"~tracemyfile" 1;
"~trendictionbot" 1;
"~TsolCrawler" 1;
"~TurnitinBot" 1;
"~TweetedTimes" 1;
"~TweetmemeBot" 1;
"~TwitterFeed 3" 1;
"~um-LN/1.0" 1;
"~uMBOT-LN/1.0 " 1;
"~UnwindFetchor" 1;
"~URLAppendBot" 1;
"~VegeBot" 1;
"~VelenPublicWebCrawler" 1;
"~VoilaBot" 1;
"~voltron" 1;
"~Wappalyzer" 1;
"~WebmasterCoffee" 1;
"~websays" 1;
"~WebTarantula.com" 1;
"~WeSEE" 1;
"~whois.domaintools.com" 1;
"~WikiDo" 1;
"~WorldBrewBot/2.1" 1;
"~wscheck" 1;
"~XoviBot" 1;
"~YaK/1.0" 1;
"~Ziba/Ziba-1.0.0" 1;
"~ZoominfoBot" 1;
"~subscriber.zoominfo.com/zoominfo" 1;
}
 

 

Link to comment
Share on other sites

13 hours ago, vaise said:

Soooooo.  Sorry in advance for this long post. 

In the never ending task of getting my 'backup when cloudflare bans' new nginx system (using swag), but matching CF where possible, I have a small issue.

I have geoIP2 database in nginx for the same geo blocking CF provides (not at the router like CF did) and I have fail2ban, and added jail for emby - should help with DDOS (but also not at the router level like CF did).

In the CF WAF, I had 'block all bots', and that never caused an issue.  For this new backup nginx system, I got a load of bot lists from online here - https://www.knthost.com/nginx/blocking-bots-with-nginx and entered all that into nginx.  Once this went live last night, I had a few Google TV users (CCWGTV) report they cant connect to emby.  In a process of elimination, this was tracked down to 'data_collectors.conf', which has 273 bots in it.  That is way to many to go through one by one.  Does anyone know off hand what bots below that emby via a google TV device to connect ?  I cant seem to find anything in the logs that is helping.

 

I see you got no responses on the Google TV issue and I don't have a clue either. But to troubleshoot the problem you don't have to examine all of the entries. Instead cut the list in half and follow the path that doesn't work, cutting again in half each time.  Much faster way to reach the few which might cause problems. Now, if it's more than one entry and they happen to be in different halves you'll have to run through it more than once.

For example, if the top half of the list works then split the bottom half again in half and try it. If that bottom quarter doesn't work then split it again and try. In five or six tries you should have it down to the one or two causing problems.

 

Link to comment
Share on other sites

3 hours ago, Q-Droid said:

I see you got no responses on the Google TV issue and I don't have a clue either. But to troubleshoot the problem you don't have to examine all of the entries. Instead cut the list in half and follow the path that doesn't work, cutting again in half each time.  Much faster way to reach the few which might cause problems. Now, if it's more than one entry and they happen to be in different halves you'll have to run through it more than once.

For example, if the top half of the list works then split the bottom half again in half and try it. If that bottom quarter doesn't work then split it again and try. In five or six tries you should have it down to the one or two causing problems.

 

Ah, thanks for the tip.  I may yet so that.  Can do it over time too.

Link to comment
Share on other sites

pir8radio
8 minutes ago, Flexeire said:

and I jinxed it.

 

Got my strike today with CF. Also looking for replacement. I have heard of a crowd Called Everly?

oh no...    that sucks...       do any of you that got "the video block" have your domain names transfered/registered with cloudflare?   I moved all of my domain names to them and pay them yearly for my domain names.  just wondering if that is another factor that is keeping me under the radar....

 

Link to comment
Share on other sites

3 minutes ago, pir8radio said:

oh no...    that sucks...       do any of you that got "the video block" have your domain names transfered/registered with cloudflare?   I moved all of my domain names to them and pay them yearly for my domain names.  just wondering if that is another factor that is keeping me under the radar....

 

I have name cheap for the domain, might be worth a shout to do that,

Separately I started a new thread with a query regarding removing the CDN settings from the server temporarily and I am stuck if you had any opinions on that one?

Edited by Flexeire
Link to comment
Share on other sites

My domain is also with CF.  I am still running on my backup non cf system currently.  I get Unify firewall emails about 6/day with block from dshield.  Known bad IP’s.  I will move back one day soon to a cf tunnel to make the switch easier.  if need be (process then is to shutdown one docker, start the other and flip off the dns in cf).

Link to comment
Share on other sites

giankoski68
16 hours ago, Flexeire said:

I have name cheap for the domain, might be worth a shout to do that,

Separately I started a new thread with a query regarding removing the CDN settings from the server temporarily and I am stuck if you had any opinions on that one?

How much traffic is being cached on your site ?  I've heard of Everly but they are not open for business or taking customers

. I think the bans are not permanent . 

 

Link to comment
Share on other sites

2 hours ago, giankoski68 said:

How much traffic is being cached on your site ?  I've heard of Everly but they are not open for business or taking customers

. I think the bans are not permanent . 

 

Some times up to 10TB. 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...