I keep on receiving different error messages while scraping with python-bs4 with requests


Beta member
Every time I run the program I get a different attribute or error or the code runs perfectly fine.

I am scraping the price, reviews, title, and other attributes from Amazon.com. It was running fine with 1 URL, however, I embedded four into a list and wrote a for loop to scrape all of their attributes at once.

This is where I ran into some errors. So I took the list out of the for-loop and ran the scraping function manually through each part of the list. However, I am still getting what seem to be random errors and success after each time I run the program with no altercations.

Not sure what to do or how to fix this from here. Again I get a different error code every time.

from bs4 import BeautifulSoup
import requests

def get_info(url):
headers = {"User-Agent" :'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36'}
page = requests.get(url, headers=headers)
soup = BeautifulSoup(page.content, 'html.parser')
title = soup.find(id='productTitle').get_text()
review_score = soup.find(id='acrPopover').get_text()
Quantity_of_Reviews = soup.find(id='acrCustomerReviewText').get_text()
Shipping_Details = soup.find(id='price-shipping-message').get_text()
price = soup.find(id='priceblock_ourprice').get_text()
Strippers = price.strip()
print((review_score.strip()), (Quantity_of_Reviews.strip()), (Shipping_Details.strip()), )

url_list = ('https://www.amazon.com/CaiFang-Port...t=&hvlocphy=9007236&hvtargid=pla-844935074316',


The errors are different. However.

Here it has ran successfully
Peacock Blue
4.7 out of 5 stars 123 ratings
Grand Patio Steel Patio Side Table, Weather Resistant Outdoor Round End Table, Peacock Blue
4.7 out of 5 stars 123 ratings & FREE Shipping
CERBIOR Tray End Table, Round Metal Tray Table Side Sofa Table Anti-Rust and Waterproof Outdoor & Indoor Snack Table Accent Coffee Table Black
4.4 out of 5 stars 204 ratings & FREE Shipping. Details
HollyHOME Small Round Patio Metal Side Snack Table, Accent Anti-Rust Steel Coffee Table for Garden, Modern Weatherproof Outdoor End Table, (H) 17.55" x(D) 15.60", Navy Blue
4.6 out of 5 stars 357 ratings & FREE Shipping. Details
Top Bottom