Arguments against legal allegations on web crawling and scraping (Part 2)

  • 26/11/2017

While the previous blog in the series covered many popular arguments, here are a few more which made it to the list. In case you missed the previous list, click here to read it.

  1. “Google has been scraping and crawling the hell out of the web since almost eternity”

Honestly, it does not even make sense since google has ultra-sophisticated bots – known as googlebots which are significantly compliant with websites’ terms of use. Also, to face facts, who does not want to be scraped by google?


  1. “I can deal with a lawsuit”

    You may want to step back and think about it. A lawsuit comes along with a lot of burden – the physical and mental agony of diverting yourself and resources towards a petty lawsuit instead of your core business would just amount to a significant loss in opportunity cost. Also, unless you are a master at understanding the art and craft of the governing law and can twist your way with legal jargons, you may need a classy black-suit wearing attorney to save you there. And these expensive suits are sponsored by client money.
  2. “If I’m using an automated programme, technically I haven’t agreed to any contract with the website”

This again, is an ambiguous zone.

In a case from Internet Archive vs. Mrs. Shell, Internet Archive had leveraged its web crawlers to copy content off Mrs. Shell’s website. Now, the crafty Mrs. Shell had an evil trick up her sleeve – on the website she had explicitly mentioned a warning which typically read as anybody visiting her website will automatically be binded by a contract which bars them from copying content and the defaulter will have to pay USD 5,000 per page copied as indemnity to Mrs. Shell. Finally, they did achieve a middle-ground in terms of compensation.

In a similar case where BoardFirst used web crawlers to download information from SouthWest Airline’s to assist passengers in choosing preferable seats, BoardFirst was deemed guilty


  1. “Are ‘Terms of Service’ even enforceable?”

The Bingham McCutchen LLP law firm published a pretty extensive article on this matter and they state that:

“As is the general rule with any contract, a website's terms of use will generally be deemed enforceable if mutually agreed to by the parties. [...] Regardless of whether a website's terms of use are clickwrap or browsewrap, the defendant's failure to read those terms is generally found irrelevant to the enforceability of its terms. One court disregarded arguments that awareness of a website's terms of use could not be imputed to a party who accessed that website using a web crawling or scraping tool that is unable to detect, let alone agree, to such terms. Similarly, one court imputed knowledge of a website's terms of use to a defendant who had repeatedly accessed that website using such tools. Nevertheless, these cases are, again, intensely factually driven, and courts have also declined to enforce terms of use where a plaintiff has failed to sufficiently establish that the defendant knew or should have known of those terms (e.g., because the terms are inconspicuous), even where the defendant repeatedly accessed a website using web crawling and scraping tools.”

TL,DR; If the court finds that there was sufficient reason for you to be aware of the existence of ‘Terms of Service’, it is legally enforceable by all means.

In other words, Terms of Service (ToS) will be legally enforced depending on the court, and if there's sufficient proof that you were aware of them.

Get A Quote