Factoring in the mechanism of result generation by any search engine, the role of a robot or web crawler is critical. A website crawler is fundamentally an automated software tool that visits individual page of a website and derives a certain amount of data from it. That packet of information is subsequently hived away in an enormous database. This technique is termed as indexing. At the time, a user looks for any specific search term or keyword, the search engines fit those keywords in their.........