can anyone check if i've set up my robots.txt file properly? i'm not entirely sure how to do it but i do NOT want AI feasting on this site https://lapistoria.neocities.org/robots.txt
1 like
Delete
Are you sure you want to remove this comment from the news feed?
Yeah I think you did it right! That's pretty much how mine looks except I allowed Googlebot and DuckDuckbot so my site is indexed by search engines. I would also recommend making a sitemap.xml file if you allow those bots. It allows them to find and index your web pages more easily :3
Yeah I think you did it right! That's pretty much how mine looks except I allowed Googlebot and DuckDuckbot so my site is indexed by search engines. I would also recommend making a sitemap.xml file if you allow those bots. It allows them to find and index your web pages more easily :3
noted! thank you so much!!