Jump to content

/robots.txt built into Emby


pir8radio

Recommended Posts

pir8radio

This is kind of a minor petty request...  But can there be a root (http://embyserver.com/robots.txt) /robots.txt file added to the server? With default disallow settings.   

 

User-agent: *
Disallow: /

 

 

I get quite a few spider hits each day, and all of them first look for /robots.txt , according to my logs.

 

Any little bit would help...

 

Thanks!

Edited by pir8radio
  • Like 2
Link to comment
Share on other sites

We use a meta tag in the markup to accomplish the same thing

<meta name="robots" content="noindex, nofollow, noarchive">
  • Like 1
Link to comment
Share on other sites

pir8radio

@@Luke  I know, but meta tag is more page/file specific, it still allows spiders to "look around" my emby server hitting every page then just not listing it in their search engine..    I want to prevent them from crawling my page in the first place...   Take a look at this from midnight to 8am today, 500+ now imagine, days, weeks, months..

 

5b952052b1de2_568bcadcd68ca_Capture.png

Edited by pir8radio
  • Like 1
Link to comment
Share on other sites

Koleckai Silvestri

then design a robots.txt file that is equivalent to my post #3 and i'll add it

 

It is in the first post. Should be in the HTML root. Would really only affect people using Ports 80 or 443 though.

 

User-agent: *

Disallow: /

  • Like 1
Link to comment
Share on other sites

pir8radio

Yea, robots.txt wouldn't be a replacement for the meta tags you are already using...  You kind of need both to cover the good and bad bots..   Well i guess the bad bots will ignore anything...   but some do ignore robots.txt, however the heavy hitters (GOOGLE) obey the robots.txt

 

@@Luke it should just be a text file with the following in it (thank you sir):

 

User-agent: *

Disallow: /

Edited by pir8radio
  • Like 1
Link to comment
Share on other sites

CBers

Should be in the HTML root. Would really only affect people using Ports 80 or 443 though.

So this can be done manually? Will it get overwritten by updates? Which specific folder? Does this work with, or is it not required for, DDNS addresses?

Link to comment
Share on other sites

  • 2 years later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...