Google Scraper 101: How to Scrape Google SERPs
The importance of scraping Google for SEO research basis cannot be overemphasized. Come in now and discover the top Google scrapers in the market — and how to crea·1te one yourself.
Google is the most popular website on the Internet and the site where most people start their search. Currently, Google’s share of the global search market is 87.35 percent. It receives over 2 trillion searches yearly and has over 130 trillion pages in its index. Because of the number of people using Google and the number of pages listed on it, it has become the single most important search engine of interest among Internet marketers, and they are all out looking for information to help them rank higher for keywords they have interest in.
Not only Internet marketers, Bing, the biggest competitor of Google, has been caught spying on Google SERPs to provide better ranking for their own listing. The thing is, Google has lots of data publicly available on their SERPs that’s of interest to the Internet market -and they will do anything to get their hands on those data. On the other hand, Google does not provide an option for getting that information free of charge, and as such, marketers have to look for an alternative, and this alternative is achieved only through using automated tools known as web scrapers.
The web scrapers that can be used for scraping Google SERPs are known as Google Scrapers. In this article, you will be learning about the best Google Scrapers in the market — and how to build one for your specific need as a coder. Before then, let take a look at an overview of scraping Google.
Google Scraping — an Overview
Google’s business model depends largely on crawling websites on the Internet. However, unlike other websites that allow it to scrap their web pages and use it for their search engine system, Google does not allow scraping data off its SERPs for free.
I have tried it a good number of times, and you need to know that you will get hit by Captchas and blocks after a few requests. And mind you, Google has got one of the best anti-scraping systems in the industry, and as such, you need to know what you are doing and how to evade it anti-spam checks to be able to scrape data from the Google SERPs.
Generally, there are different reasons why you will want to scrap Google. The most popular reasons among marketers are that they want to extract keyword-based data, as well as ranking data for web pages for some specific keywords.
Some can also use it in search of expired domains and web 2.0 blogs. When it comes to gathering this data, you might not even need to do them yourself as there are some already-made solutions such as Semrush, Ahref, and Moz, among others, that can help you with this. However, if you want a more specialized tool or want to avoid paying for the prices labeled on these already-made tools, then you just have to scrape yourself.
How to Scrape Google Using Python, Requests, and BeautifulSoup
I don’t know about you, but I do know that as an Internet marketer myself, I find myself interested in a good number of data publicly available on the Google Search Engine Result Pages (SERPs) and I try to keep the cost as low as possible — fortunately, I am a coder. If you are like me and want to scrape Google by building your own Google scraper, then this section has been written for you. It will contain mostly advice and a code sample below to show you how to get it done.
The Google SERPs layout and design are different across devices and platforms, and as such, setting header most especially the user-agent header is very important. I tried running a Google scraping script I wrote on a Windows computer, with the HTML inspected using Chrome on my mobile IDE and the code break — until I same headers before it worked. Aside from this, you also need to put checks in place to notify you if there is a layout change on the Google SERP as it changes a lot — and as such, you need to prepare for it.
I will advise you not to use Selenium for the scraping as it is easily detectable and also allows Google to create a fingerprint of you. The duo of Requests and BeautifulSoup will work just fine if you want to use the Python programming language.
You need to use high-quality proxies that will not leak your IP address and aren’t detectable as proxies. When it comes to scraping Google, residential proxies are the best in the market. You also have to take care of rotating your proxies, but using a web scraping API or a proxy pool can relieve you of this duty. Aside from proxies, there are many other things you need to take care of, which includes setting headers and randomizing timing between requests.
Below is a code sample that scrapes the keyword suggestions displayed at the bottom of the Google SERP. This tool is basic and a proof of concept; you need to incorporate HTML checks to verify layout consistency and change, as well as exception handling, and proxies if you really need to use this for a big project.
import requests
from bs4 import BeautifulSoupdef add_plus(keywords):
keywords = keywords.split()
keyword_edited = ""
for i in keywords:
keyword_edited += i + "+"
keyword_edited = keyword_edited[:-1]
return keyword_edited
class KeywordScraper:
def __init__(self, keyword):
self.keyword = keyword
plusified_keyword = add_plus(keyword)
self.keywords_scraped = []
self.search_string = "https://www.google.com/search?q=" +
plusified_keyword
def scrape_SERP(self):
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36'}
content = requests.get(self.search_string, headers=headers).text
soup = BeautifulSoup(content, "html.parser")
related_keyword_section = soup.find("div", {"class":"card-section"})
keywords_cols = related_keyword_section.find_all("div",
{"class":"brs_col"}) for col in keywords_cols:
list_of_keywords = col.find_all("p", {"class":"nVcaUb"})
for i in list_of_keywords:
self.keywords_scraped.append(i.find("a").text)
def write_to_file(self):
for keyword in self.keywords_scraped:
with open("scraped keywords.txt", "a") as f:
f.write(keyword + "\n")
print("keywords related to " + self.keyword + " scraped successfully")
s = KeywordScraper("Best gaming pc")
s.scrape_SERP()
s.write_to_file()
Best Google Scrapers in the Market
There are many Google scrapers in the market you can use for scraping data publicly available on the Google SERPs. However, their effectiveness, pricing, and ease of usage are not the same. Some of them have proven to be the best when it comes to getting the work done while evading blocks. Some of these best Google Scrapers will be discussed below.
Octoparse
- Pricing: Starts at $75 per month
- Free Trials: 14 days of free trial with limitations
- Data Output Format: CSV, Excel, JSON, MySQL, SQLServer
- Supported Platform: Cloud, Desktop
Octoparse is a general web scraper that you can use for scraping Google — and one of the best Google scraper out there. Octoparse does not only have the capabilities of scraping Google result pages but also can be used for scraping data from Google Maps.
One thing I have come to like about Octoparse is that it is a very smart web scraper that intelligently avoids anti-scraping systems put in place by websites. Octoparse does not require you to be a coder in other to make use of it as it is a visual scraping tool. Octoparse is easy to use and comes as both an installable software and a cloud-based solution.
ScrapeBox
- Pricing: One-time payment of $97
- Free Trials: Yes
- Data Output Format: CSV, TXT, etc
- Supported Platform: Desktop
If there is one tool you need for scraping Google, then that tool will be ScrapeBox. It is not just meant for Google search engine scraping but for general SEO related scraping tasks — and other Internet marketing activities. Regarded as the Swiss Army Knife of SEO, ScrapeBox has got a good number of tools that will help you in carrying out your Internet marketing tasks, including Search Engine Harvesters and Keyword Harvester, which are perfect for scraping publicly available data on Google SERPs. You need proxies in other to use ScrapeBox successfully as it will help hide your IP footprint — in other to evade IP tracking. It is a Windows-based tool.
Webscraper.io
- Pricing: Browser extension is free
- Free Trials: Browser extension is free
- Data Output Format: CSV
- Supported Platform: Chrome
Web scrapers that work well come with a price tag on them, and that include every other web scraper on the list except Webscraper.io, which is completely free to use except if you are interested in using their cloud-based platform. Webscraper.io is available as a Google Chrome browser extension and can be used for extracting data from Google web pages, including Google SERPs and Google Maps. Webscraper.io works on other websites tools and with it, you can convert into a database. Because this tool is free, you will have to take care of proxies yourself. Make no mistake about it, even without a price tag on it, Webscraper.io works.
Apify Google Search Result Scraper
- Pricing: Starts at $49 per month for 100 Actor compute units
- Free Trials: Starter plan comes with 10 Actor compute units
- Data Output Format: JSON
- Supported OS: cloud-based — accessed via API
Unlike the other Google scrapers discussed above, the Apify Google Search Result Scraper was built for coders to use as an API, and as such, it is not a visual tool like the rest — you must know how to code for you to be able to harness its full potentials. With this Google scraper, you only need to send API requests, and the required data is returned in a JSON format. This scraper will help you scrape publicly available data on Google SERPs, ranging from ads, pages listed, and keyword related data. As stated earlier, this tool is for developers and can be used as a scraping API.
Proxycrawl Google Scraper
- Pricing: Starts at $29 per month for 50,000 credits
- Free Trials: first 1000 requests
- Data Output Format: JSON
- Supported Platforms: cloud-based — accessed via API
Proxycrawl Google Scraper was not originally made as a regular web scraper, but as a scraping API, you can use it to extract structured data from Google search engine result pages. Some of the information you can scrape includes keyword related information such as people also ask, related search result, ads, and many more. This means that the Proxycrawl Google Scraper is not meant to be used by non-coders but by coders who are trying to avoid handling proxies, Captchas, and blocks. It is easy to use and very much effective.
ConclusionGoogle SERPs hold a lot of keyword and page ranking based data that is of interest to Internet marketers and researchers, and as such, even though Google is against scraping them, they have not been able to prevent scraping it completely. Some of the Google scraper that has been proven to work excellently well have been discussed above.