Scrape a website using python
In this generation DATA is the key of market analysis. Every entrepreneur or business startup need the market research. So they want the formatted data for market research. For example if any entrepreneur wants to start a new ecommerce then he want to know the all product prices for complete the market. For this type of information gathering we want to scrape the website. We can scrape any website with the help of python.
Python provides many packages for web scraping. I am a python developer and share my experience while working on web scraping. Let’s start discussing on packages who is helping for scraping.
- Session Requests
- Beautiful soap
- Fake User Agent
- Proxies
Session Requests: Session requests is the python module for sending HTTP request for get the response in html like a browser with the help of add headers, SSL verification, Proxy and cookies. This is the code for install session requests:
Installing The Requests module
$ pip install requests
Example of code:
|
1 2 |
import requests req = requests.get('https://www.example.com') |
You can use post instead of get when it required with add headers etc.
Beautiful soap:
Beautiful soap is another tool for web scraping to find the informational data from html. For example, you need to get the price from html then you can pass class or id attribute in beautiful soap code and get easily data. Example code is as following:
|
1 2 3 4 5 |
from bs4 import BeautifulSoup import requests req = requests.get('https://www.example.com') soup = BeautifulSoup(req.content, 'html.parser') |
Fake User Agent: Many times some standard websites blocks your ip while using most of requests in little time. So you need to rotate the user agent for risk of blockage. Import package for fake user agent:
$ pip install fake-useragent
|
1 2 3 4 5 |
from fake_useragent import UserAgent import requests ua = UserAgent().random headers = {'User-Agent':ua} req = requests.get('https://www.example.com',headers=headers) |
For more unique user agents you can add random numbers with user agent function for you can’t block any more with the same user agent. Like this code:
|
1 2 3 4 5 |
import random from fake_useragent import UserAgent import requests headers = {'User-Agent':UserAgent().random+".{0}".format(intt)} req = requests.get('https://www.example.com',headers=headers) |
Proxy server: Some times with the use of all these methods still we blocked. So we use last tool proxy server details in headers of request session or urllib3 library. I am sharing with you the full code for get the data with the help of proxy. In market many of free proxy servers available but they don’t work. So you need the dedicated proxy server for bypass the captcha or ip blockage. I am using apify.com proxy server for rotating ips and bypass the captcha. There are a lot of others but they are costly as compare apify.com. This is my personal experience.
In this code I have used urllib for get the response instead session.requests.
Full Code
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
import time import sys import datetime from random import randint import urllib.request from bs4 import BeautifulSoup import random from fake_useragent import UserAgent sys.setrecursionlimit(1000000000) username = 'xxxxxx' password = 'xxxxxx' port = 22225 super_proxy_url = ('http://%s:%s@proxy.com:%d' %(username, password, port)) proxy_handler = urllib.request.ProxyHandler({ 'http': super_proxy_url, 'https': super_proxy_url, }) intt=random.randrange(0,1999) agent = UserAgent().random+".{0}".format(intt) opener = urllib.request.build_opener(proxy_handler) opener.addheaders = [('User-agent', agent),('Accept','text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8')] htm = opener.open('https://example.com').read() soup = BeautifulSoup(htm, 'lxml') captcha = soup.find('input', {'id': 'twotabsearchtextbox'}) # print(captcha) if(captcha is not None): print(htm) |
User Review
( votes)
About Author
@rahul
Website designer, Full stack developer, Blogger with 10+ years of experience.