You are going to block this site. This will do the following:
- You will no longer see this site in searches.
- Site will no longer see your site in searches.
- Site will not be able to comment on your site profile.
- Any comments this site has posted to your profile will not be displayed.
Are you sure you want to do this?
i'm sure you already know! but i just wanted to leave the note! openai seems to be ignoring it for example: https://www.businessinsider.com/openai-anthropic-ai-ignore-rule-scraping-web-contect-robotstxt
Yeah it is very unfortunate that they will not follow it. I know this doesn't apply to neocities, but for people that host their own sites I recently heard cloudflare released a feature to block AI bots.
Absolutely, all the robots file does is state that you do not consent - whether companies listen to that (often, they don't) is another matter. I think it's worth doing, but I didn't mean to create a false sense of security.