Web crawling and scraping: Growth Hacks

  • 07/12/2017

If you’ve been following our posts all this while, you would have gained at least an elementary knowledge on web scraping and web crawling. While a majority of folks out there may liken the use case for web scraping and crawling to merely setting up a few title and meta tags right, the scope is well much beyond just that. 

The following post is going to cover a few nifty hacks to extract maximum value from your web scraping and crawling ambitions.

  1. Get in touch with your readership base by scraping existing website comments
    Generally, people who comment on your blogs do so with their name (and website link embedded). These are generally easy to spot among a list of comments given that these are hyperlinked.
    How to go about from there?
    Simple, if you are posting a new content and have posted content related to a similar subject matter earlier, just go to the earlier post and check for people who have commented on the post. The people who have commented on that post typically will have a genuine interest in knowing more on the subject, i.e., your new post.

    One of the reasons why this is much more than a cold pitch is that since these guys have already engaged on your post and expressed interest on a particular topic, the chances that your new post may interest them goes up significantly. This, in comparison to pitching absolute strangers, will contribute to much better stickiness of readership and engagement.

    If you have a horde of comments, the simplest way to scrape this list is by leveraging the ‘scraper’ extension offered by ‘dvhtn’ on chrome.

    The simple steps to use this extension is as follows-
    1. Mark the hyperlinked text (typically, the username) and right-click on it and click on “scrape similar”.
    2. Create a copy of this on google sheets and hit ‘copy to clipboard’ and paste it into the tab – “Start Here”.
    3. If you have many pages, you may need to repeat this.
    4. Go to the tab and use the add-in to find the email addresses.

    One major note of caution here would be to acknowledge the fact that these folks have not signed up for your mailing list; so, be careful so as to not spam them and just shoot a subtle informational mail.
  2. Reach out to influencers by scraping existing expert opinions
    Having a quote and link to an expert’s opinion within your content may help in more than one way. However, the tricky part is getting in touch with influencers in your industry who would be willing to provide knowledge bytes for you to include and publish in your content. While some may be too busy to even respond to your requests, there would be many who would detest plain cold pitches. The key lies in moderation and approaching only those who are genuinely inclined towards helping fellow netizens.

The same can be found and fetched through the above steps as done for guest posts.


These are 2 out of at least many more web crawling and scraping hacks.

Get A Quote