pir8radio 1289 Posted January 5, 2016 Share Posted January 5, 2016 (edited) This is kind of a minor petty request... But can there be a root (http://embyserver.com/robots.txt) /robots.txt file added to the server? With default disallow settings. User-agent: *Disallow: / I get quite a few spider hits each day, and all of them first look for /robots.txt , according to my logs. Any little bit would help... Thanks! Edited January 5, 2016 by pir8radio 2 Link to comment Share on other sites More sharing options...
Luke 36876 Posted January 5, 2016 Share Posted January 5, 2016 We use a meta tag in the markup to accomplish the same thing <meta name="robots" content="noindex, nofollow, noarchive"> 1 Link to comment Share on other sites More sharing options...
pir8radio 1289 Posted January 5, 2016 Author Share Posted January 5, 2016 (edited) @@Luke I know, but meta tag is more page/file specific, it still allows spiders to "look around" my emby server hitting every page then just not listing it in their search engine.. I want to prevent them from crawling my page in the first place... Take a look at this from midnight to 8am today, 500+ now imagine, days, weeks, months.. Edited September 9, 2018 by pir8radio 1 Link to comment Share on other sites More sharing options...
Luke 36876 Posted January 5, 2016 Share Posted January 5, 2016 then design a robots.txt file that is equivalent to my post #3 and i'll add it Link to comment Share on other sites More sharing options...
Koleckai Silvestri 1150 Posted January 5, 2016 Share Posted January 5, 2016 then design a robots.txt file that is equivalent to my post #3 and i'll add it It is in the first post. Should be in the HTML root. Would really only affect people using Ports 80 or 443 though. User-agent: * Disallow: / 1 Link to comment Share on other sites More sharing options...
pir8radio 1289 Posted January 6, 2016 Author Share Posted January 6, 2016 (edited) Yea, robots.txt wouldn't be a replacement for the meta tags you are already using... You kind of need both to cover the good and bad bots.. Well i guess the bad bots will ignore anything... but some do ignore robots.txt, however the heavy hitters (GOOGLE) obey the robots.txt @@Luke it should just be a text file with the following in it (thank you sir): User-agent: * Disallow: / Edited January 6, 2016 by pir8radio 1 Link to comment Share on other sites More sharing options...
CBers 6742 Posted January 6, 2016 Share Posted January 6, 2016 Should be in the HTML root. Would really only affect people using Ports 80 or 443 though. So this can be done manually? Will it get overwritten by updates? Which specific folder? Does this work with, or is it not required for, DDNS addresses? Link to comment Share on other sites More sharing options...
pir8radio 1289 Posted January 7, 2016 Author Share Posted January 7, 2016 (edited) @@Luke FYI the robots.txt needs to be at the root level... http://embyserver/robots.txt wont work if it is at http://embyserver/web/robots.txt Thanks again for hearing me out!! Edited January 7, 2016 by pir8radio Link to comment Share on other sites More sharing options...
mastrmind11 717 Posted August 30, 2018 Share Posted August 30, 2018 Resurrecting this because I'd like to know if it's implemented. I assume so given https://github.com/MediaBrowser/Emby/blob/master/MediaBrowser.WebDashboard/dashboard-ui/robots.txt ?? Link to comment Share on other sites More sharing options...
Happy2Play 8138 Posted August 31, 2018 Share Posted August 31, 2018 Resurrecting this because I'd like to know if it's implemented. I assume so given https://github.com/MediaBrowser/Emby/blob/master/MediaBrowser.WebDashboard/dashboard-ui/robots.txt ?? Have you checked? http://localhost:8096/robots.txt Link to comment Share on other sites More sharing options...
mastrmind11 717 Posted August 31, 2018 Share Posted August 31, 2018 Have you checked? http://localhost:8096/robots.txt I have now. Thanks Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now