Web Crawling : the act of automatically downloading data from a web page, extracting the hyperlinks contained therein and following them. The downloaded data is usually stored in an index or database to facilitate its search. Web crawling, also known as indexing, is used to index information on a web page using bots, also called crawlers. Web Crawlers are basically used by major search engines like Google, Bing and Yahoo.
Based on the above,.........