You are going to block this site. This will do the following:
- You will no longer see this site in searches.
- Site will no longer see your site in searches.
- Site will not be able to comment on your site profile.
- Any comments this site has posted to your profile will not be displayed.
Are you sure you want to do this?
u can opt out though! you need to create a robots.txt file inside ur site folder and then copy and paste the contents of this pastebin inside and remove all the # symbols in the list to stop said crawlers from whipping shitties all over ur webzone: https://raw.githubusercontent.com/neocities/neocities/refs/heads/master/views/templates/robots.txt
i'm slightly concerned that if another crawler comes out that's not on this list i'll have to update it manually, but if i just do the User-agent: * and Disallow: \ that should work on everything right? not concerned with showing up on search engines personally
i have a robots.txt but do the #'s need to be removed to work? just to double check on some things
@mr-pibbs-site um! i believe if you remove the # at the very bottom with the "Disallow: /" it should block all crawling, yes. you may wanna remove the "Allow: /" line near the top as well!!
@seashrine yes, the # symbols are commenting out those lines! removing them will make the robots.txt block the bot on that line (i removed all of them, but left "Disallow: /" with its # so my website will still show up on google searches)
@daikonet thank you! i saw a different one going around and wasn't sure if they were to be removed or not, thanks for clearing that up!