I keep on receiving different error messages while scraping with python-bs4 with requests

kavyasahu

Beta member
Messages
1
Location
uk
Every time I run the program I get a different attribute or error or the code runs perfectly fine.

I am scraping the price, reviews, title, and other attributes from Amazon.com. It was running fine with 1 URL, however, I embedded four into a list and wrote a for loop to scrape all of their attributes at once.

This is where I ran into some errors. So I took the list out of the for-loop and ran the scraping function manually through each part of the list. However, I am still getting what seem to be random errors and success after each time I run the program with no altercations.

Not sure what to do or how to fix this from here. Again I get a different error code every time.


from bs4 import BeautifulSoup
import requests

def get_info(url):
headers = {"User-Agent" :'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36'}
page = requests.get(url, headers=headers)
soup = BeautifulSoup(page.content, 'html.parser')
title = soup.find(id='productTitle').get_text()
review_score = soup.find(id='acrPopover').get_text()
Quantity_of_Reviews = soup.find(id='acrCustomerReviewText').get_text()
Shipping_Details = soup.find(id='price-shipping-message').get_text()
price = soup.find(id='priceblock_ourprice').get_text()
print(title.strip())
Strippers = price.strip()
print(Strippers[1:])
print((review_score.strip()), (Quantity_of_Reviews.strip()), (Shipping_Details.strip()), )

url_list = ('https://www.amazon.com/CaiFang-Port...t=&hvlocphy=9007236&hvtargid=pla-844935074316',
'https://www.amazon.com/Grand-Patio-...ble&qid=1600363264&s=lawn-garden&sr=1-10&th=1',
'https://www.amazon.com/dp/B085NBYH8...jbGlja1JlZGlyZWN0JmRvTm90TG9nQ2xpY2s9dHJ1ZQ==',
'https://www.amazon.com/HollyHOME-Ac...XFEEX7C7F66&psc=1&refRID=F92DXW1YFXFEEX7C7F66')

get_info(url_list[1])
get_info(url_list[1])
get_info(url_list[2])
get_info(url_list[3])

The errors are different. However.

Here it has ran successfully
Peacock Blue
39.99
4.7 out of 5 stars 123 ratings
Grand Patio Steel Patio Side Table, Weather Resistant Outdoor Round End Table, Peacock Blue
39.99
4.7 out of 5 stars 123 ratings & FREE Shipping
CERBIOR Tray End Table, Round Metal Tray Table Side Sofa Table Anti-Rust and Waterproof Outdoor & Indoor Snack Table Accent Coffee Table Black
37.99
4.4 out of 5 stars 204 ratings & FREE Shipping. Details
HollyHOME Small Round Patio Metal Side Snack Table, Accent Anti-Rust Steel Coffee Table for Garden, Modern Weatherproof Outdoor End Table, (H) 17.55" x(D) 15.60", Navy Blue
39.99
4.6 out of 5 stars 357 ratings & FREE Shipping. Details
 
Top Bottom