Here’s a friendly reminder to all you wonderful web masters out there to add/update a robots .txt file to your site if you don’t fancy bots scraping everything you’ve made.
7 likes
Delete
Are you sure you want to remove this comment from the news feed?
Thanks! I have one that is pretty extensive so far, I will add anything from yours that I do not have as well as the TurnitinBot that @owlroost mentioned! Thanks all
Thanks for the recommendation, Owlroost. I’ll be sure to add that. @punk-tart, it should go in your site’ root directory (so, not in a “folder”, but the same place as you homepage - hope that makes sense). Cheers all.
A new enabler of mediocrity to add to your file: Applebot-Extended
Feel free to pinch mine: https://mikegrindle.com/robots.txt
Thank you, keep forgetting to create one and just did it!
One to add for you: User-agent: TurnitinBot
Thanks! I have one that is pretty extensive so far, I will add anything from yours that I do not have as well as the TurnitinBot that @owlroost mentioned! Thanks all
Do we just add it anywhere on our website?
Thanks for the recommendation, Owlroost. I’ll be sure to add that. @punk-tart, it should go in your site’ root directory (so, not in a “folder”, but the same place as you homepage - hope that makes sense). Cheers all.
Thanks for the clarification, added and good to go! Always appreciate dodging AI