A web crawler, also known as a web spider or web robot, is a program, software package, or automated script which browses the Global Web in a systematic and automated method. Web crawlers are mostly used to generate a duplicate of all the pages they visit, then processing them throughout a search engine that will file the copied pages to deliver faster search results.
