Neocities.org

Mike Grindle's Webpage

mikegrindle.com

577,124 views
106 followers
2,749 updates
0 tips
heya, i remember you posted something about robots.txt earlier -- that's mostly a convention with no guarantee anyone on the web will follow it. you can request x-bot not to crawl your site but they still can if they want to.
3 likes
sorbier 4 months ago

i'm sure you already know! but i just wanted to leave the note! openai seems to be ignoring it for example: https://www.businessinsider.com/openai-anthropic-ai-ignore-rule-scraping-web-contect-robotstxt

2 likes
colexdev 4 months ago

Yeah it is very unfortunate that they will not follow it. I know this doesn't apply to neocities, but for people that host their own sites I recently heard cloudflare released a feature to block AI bots.

3 likes
mikegrindle 4 months ago

Absolutely, all the robots file does is state that you do not consent - whether companies listen to that (often, they don't) is another matter. I think it's worth doing, but I didn't mean to create a false sense of security.

2 likes

Website Stats

Last updated 3 hours ago
CreatedNov 3, 2022
Site Traffic Stats

Tags

writing blogging technology links essays