Robots.txt and crawl access

Understand how robots.txt guides crawlers and the limits of using it.