Robots.txt and sitemaps already exist and have been web standards forever. The AI crawler bots need to learn how to crawl better, not make website owners do extra work.
Nice writeup from Ryan Law
This post was originally shared by on Linkedin.
Robots.txt and sitemaps already exist and have been web standards forever. The AI crawler bots need to learn how to crawl better, not make website owners do extra work.
Nice writeup from Ryan Law