Completed /robots.txt built into Emby

[pir8radio 1304]

This is kind of a minor petty request...  But can there be a root (http://embyserver.com/robots.txt) /robots.txt file added to the server? With default disallow settings.   

 

User-agent: *
Disallow: /

 

 

I get quite a few spider hits each day, and all of them first look for /robots.txt , according to my logs.

 

Any little bit would help...

 

Thanks!

Edited January 5, 2016 by pir8radio

[Luke 39386]

We use a meta tag in the markup to accomplish the same thing

<meta name="robots" content="noindex, nofollow, noarchive">

[pir8radio 1304]

@@Luke  I know, but meta tag is more page/file specific, it still allows spiders to "look around" my emby server hitting every page then just not listing it in their search engine..    I want to prevent them from crawling my page in the first place...   Take a look at this from midnight to 8am today, 500+ now imagine, days, weeks, months..

 

Edited September 9, 2018 by pir8radio

[Luke 39386]

then design a robots.txt file that is equivalent to my post #3 and i'll add it

[Koleckai Silvestri 1150]

then design a robots.txt file that is equivalent to my post #3 and i'll add it

 

It is in the first post. Should be in the HTML root. Would really only affect people using Ports 80 or 443 though.

 

User-agent: *

Disallow: /

[pir8radio 1304]

Yea, robots.txt wouldn't be a replacement for the meta tags you are already using...  You kind of need both to cover the good and bad bots..   Well i guess the bad bots will ignore anything...   but some do ignore robots.txt, however the heavy hitters (GOOGLE) obey the robots.txt

 

@@Luke it should just be a text file with the following in it (thank you sir):

 

User-agent: *

Disallow: /

Edited January 6, 2016 by pir8radio

[CBers 7021]

Should be in the HTML root. Would really only affect people using Ports 80 or 443 though.

So this can be done manually? Will it get overwritten by updates? Which specific folder? Does this work with, or is it not required for, DDNS addresses?

[pir8radio 1304]

@@Luke  FYI the robots.txt needs to be at the root level...    http://embyserver/robots.txt    wont work if it is at http://embyserver/web/robots.txt    Thanks again for hearing me out!!  

Edited January 7, 2016 by pir8radio

[mastrmind11 720]

Resurrecting this because I'd like to know if it's implemented.  I assume so given https://github.com/MediaBrowser/Emby/blob/master/MediaBrowser.WebDashboard/dashboard-ui/robots.txt ??

[Happy2Play 9307]

Resurrecting this because I'd like to know if it's implemented.  I assume so given https://github.com/MediaBrowser/Emby/blob/master/MediaBrowser.WebDashboard/dashboard-ui/robots.txt ??

 

Have you checked?    http://localhost:8096/robots.txt

[mastrmind11 720]

Have you checked?    http://localhost:8096/robots.txt

I have now.  Thanks