Google Wants To Stop Supporting 'Noindex', 'Nofollow' And 'Crawl-Delay' In Robots.txt On the web, there are countless of crawlers that relentlessly scout for contents.