Understanding the Robots File Search engines have what they call spiders or web crawlers. These computer programs browse the World Wide Web in a methodical, automated manner or in an orderly fashion, commonly known as web crawling or spidering.
