Having a look at the mechanism of the results generated by any search engine, the role of robots or web crawlers cannot be downplayed. A website crawler is fundamentally an automated program that visits every page of the website and picks out some data from it. That data is subsequently hived away in a large database. This technique is called indexing. When a user looks for any specific search term or keyword, the search engines fit the specific keywords in their database and churn out the re.........