Three tips to be an ethical web scraper

  • 28/09/2017

Quite often many of us are so engrossed in achieving the goal that we forget about being fair in the means. Web scraping as a service is almost indispensable especially to organisations operating in a data sensitive organization. Succumbing to the pressure of being on top of the data game takes such a toll that the lines between ethical web scraping and intrusive web scraping is almost blurred. Web scraping and data extraction as services may be one of the most profitable value propositions. Web scraping is a powerful strategy to excel in number sensitive businesses, but as the saying goes – with great power comes great responsibility. There are always two ways to go about a process – the right way and the brute force way. Obviously, you know what’s better – the right way!

Pay attention to frequency

Imagine, your neighbor usually comes knocking onto your door to borrow sugar, condiments or any spice for that matter. You, being a good guy would help him, right? Now, what happens to your state of mind if he keeps coming everyday – even worse, twice a day? You will obviously not like it. You will find a way to refuse help after a point in time. Web scraping is a very similar process where you go to a website and request it for data. The more frequently you go, the more of a pain you become to the site administrator. Don’t be that guy.

You need to understand that web servers have only a limited capacity to entertain requests. The more frequent your requests get, higher the probability is that the server may slow down or even worse – just crash. The website administrator, in most cases, set up a website for human interaction and not for bot requests. A slowdown or a crash of the website makes the entire user experience fail miserably. While requesting a website for data, avoid parallel requests. It is recommended to have a reasonable gap between requests and if possible, give it more time than you deem fit.

The website is your golden egg-laying hen, don’t kill it at once, you will not get anything. Give it time and space, it will keep laying golden eggs for you.

Robots.txt is there for a reason

Let’s suppose that you have drawn the curtains of your bedroom because, obviously, you do not want anybody to peek or seek any information from your bedroom. Now, what happens if a ‘peeking tom’ tries to pull your curtains aside to try and gather information about whatever is happening inside. Would you let such a behaviour pass? Probably not. Robots.txt is essentially the curtain to a website. The website administrator may wish to hold some information private and it is expected that no bot should try and forcefully intrude the web page’s privacy.

Almost every website has a Robots.txt file which suggests how should bots behave with the website. Going against this file is highly unethical. These files would mostly have a set of predefined set of guidelines as to what can be the visiting frequency, which pages can be parsed and which pages are restricted. It is best to steer clear of restricted pages at least. If a website has placed restrictions against crawling by bots, it is best to not scrape them at all as it would not just be unethical, but also illegal. Yes, just like being a peeking tom is.

Choose your time wisely

As suggested above, websites are primarily for human interaction and a web scraping process should not in any way affect the user experience of websites. It is always advisable to study the key traffic hours of the website and stay away from it for at least the hours with the highest traffic. You may always choose to scrape the site when the server has some breathing space and there are not too many humans requesting information and access.

These tips should keep you safe in the web scraping and data extraction game. If this is too much for you, BotScraper is here to help you. Botscraper uses web-friendly scraping technology for all of its web scraping processes.

Get A Quote