Scrape a website using python

In this generation DATA is the key of market analysis. Every entrepreneur or business startup need the market research. So they want the formatted data for market research. For example if any entrepreneur wants to start a new ecommerce then he want to know the all product prices for complete the market. For this type of information gathering we want to scrape the website. We can scrape any website with the help of python. 

Python provides many packages for web scraping. I am a python developer and share my experience while working on web scraping. Let’s start discussing on packages who is helping for scraping.

  1. Session Requests
  2. Beautiful soap
  3. Fake User Agent
  4. Proxies

Session Requests: Session requests is the python module for sending HTTP request for get the response in html like a browser with the help of add headers, SSL verification, Proxy and cookies. This is the code for install session requests:

Installing The Requests module

$ pip install requests

Example of code:

You can use post instead of get when it required with add headers etc. 


Beautiful soap:

Beautiful soap is another tool for web scraping to find the informational data from html. For example, you need to get the price from html then you can pass class or id attribute in beautiful soap code and get easily data. Example code is as following:

Fake User Agent: Many times some standard websites blocks your ip while using most of requests in little time. So you need to rotate the user agent for risk of blockage. Import package for fake user agent:

$ pip install fake-useragent 

For more unique user agents you can add random numbers with user agent function for you can’t block any more with the same user agent. Like this code:

Proxy server: Some times with the use of all these methods still we blocked. So we use last tool proxy server details in headers of request session or urllib3 library. I am sharing with you the full code for get the data with the help of proxy. In market many of free proxy servers available but they don’t work. So you need the dedicated proxy server for bypass the captcha or ip blockage. I am using proxy server for rotating ips and bypass the captcha. There are a lot of others but they are costly as compare This is my personal experience.

In this code I have used urllib for get the response instead session.requests.

Full Code


User Review
5 (2 votes)

Add a Comment

Your email address will not be published. Required fields are marked *