A text file placed in a website directory that directs search engine robots (spiders and crawlers) which pages to index and which to omit.