Scraping the most in-demand domains just got easier.
Get structured JSON data using the standard API you already know and love.
Our advanced Auto Parsing feature allows you to collect JSON data from Amazon, Google Search, and Google Shopping (more domains to come) using the standard API endpoint. When using this feature, ScraperAPI will parse the raw HTML and return all relevant information in a structured format.
Our team will make sure to identify and handle any page structure changes and ensure you’re receiving the accurate and constant flow of data you need without needing to build and maintain your own parsers.
Benefits of Auto Parsing: More Data, Less Work
If you need to retrieve data from any of our supported domains, Auto Parsing can make your life easier by:
- Allowing you to work with JSON data, making your workflow more efficient
- Managing the parsers for you, saving engineering hours
- Increasing your scrapers’ success rate to almost 100%
- Providing reliable and consistent data at the scale you need
- Scrape structured data from highly custom URLs to fit your unique project, no matter how complex
Note: Auto Parsing is best used when you need to work with custom URLs to fit your unique project. To take URL generation out of your already complex scraping pipeline, check our Domain Specific endpoints.
Getting Started with ScraperAPI’s Auto Parsing [Amazon Example]
Using the Auto Parsing feature is as simple as using our standard API endpoint. To see it in action, let’s scrape https://www.amazon.com/dp/B09R93MDJX/ using ScraperAPI’s sync endpoint and adding the autoparse parameter.
We’ll use Requests in Python for this simple script and print the response to the terminal:
import requests
url = "https://api.scraperapi.com?api_key=YOUR_API_KEY&autoparse=true&url=https://www.amazon.com/dp/B09R93MDJX/"
response = requests.get(url)
print(response.text)
Note: Although it’s not always the case, the parameters on the target URL may conflict with the API’s parameters. To avoid any potential issues, it’s a best practice to add the url parameter last.
After running our script, here’s the returned JSON:
{
"name": "Ampere Shower Power - Hydropower Bluetooth Shower Speaker, Waterproof Detachable Portable, Rechargeable Mini Speaker (Chrome)",
"product_information": {
"Package Dimensions": "7.76 x 4.41 x 4.09 inches",
"Item Weight": "1.95 pounds",
"ASIN": "B09R93MDJX",
"Batteries": "1 Lithium Ion batteries required. (included)",
"Customer Reviews": {
"ratings_count": 34,
"stars": "3.7 out of 5 stars"
},
"Best Sellers Rank": [
"#2,182 in Portable Bluetooth Speakers",
"#12,927 in MP3 & MP4 Player Accessories"
],
"Date First Available": "January 27, 2022",
"Manufacturer": "Ampere"
},
"brand": "Visit the Ampere Store",
"brand_url": "https://www.amazon.com/stores/Ampere/page/9C39E858-3C64-4A00-8874-12B8AADCAB6B?ref_=ast_bln",
"full_description": "Product Description\nRead more\n360\u00b0 SPEAKERS The cylindrical design delivers immersive audio that plays evenly in all directions.\nBIG BUTTONS Play, pause and skip tracks directly on the speaker with the big easy-to-touch buttons.\nALWAYS HAVE MUSIC IN THE SHOWER Once installed, Shower Power never needs to be removed.\nRead more\nRead more",
"pricing": "$99.00",
"list_price": "",
"shipping_price": "FREE",
"availability_status": "In Stock. In Stock.",
"images": [
"https://m.media-amazon.com/images/I/31KKeBcl-NL.jpg",
"https://m.media-amazon.com/images/I/41gv1XZqBmL.jpg",
"https://m.media-amazon.com/images/I/41Qw+17FCEL.jpg",
"https://m.media-amazon.com/images/I/41sU4zf0Y6L.jpg",
"https://m.media-amazon.com/images/I/519iXpFvQJL.jpg",
"https://m.media-amazon.com/images/I/41nl2K4wu+L.jpg"
],
"product_category": "Electronics \u203a Portable Audio & Video \u203a Portable Speakers & Docks \u203a Portable Bluetooth Speakers",
"average_rating": 3.7,
"small_description": "About this item \n \nPOWERED BY WATER: Our proprietary hydropower system uses the water you are already using in the shower for power so you always have a fully charged speaker. SIMPLE TO USE: Shower Power seamlessly integrates into your daily routine. It turns on automatically, automatically pairs, and automatically turns off. BIG SOUND, small size: The cylindrical design delivers big immersive audio that plays evenly in all directions and is designed for bathroom acoustics. MADE FROM RECYCLED OCEAN PLASTIC: Shower Power is made from recovered ocean plastic but that\u2019s not all, it is also delivered in 100% plastic-free packaging. WORKS ON (ALMOST) ANY SHOWER: Shower Power installs onto nearly any showerhead \u2014 rain, fixed, or handheld.",
"feature_bullets": [
"POWERED BY WATER: Our proprietary hydropower system uses the water you are already using in the shower for power so you always have a fully charged speaker.",
"SIMPLE TO USE: Shower Power seamlessly integrates into your daily routine. It turns on automatically, automatically pairs, and automatically turns off.",
"BIG SOUND, small size: The cylindrical design delivers big immersive audio that plays evenly in all directions and is designed for bathroom acoustics.",
"MADE FROM RECYCLED OCEAN PLASTIC: Shower Power is made from recovered ocean plastic but that\u2019s not all, it is also delivered in 100% plastic-free packaging.",
"WORKS ON (ALMOST) ANY SHOWER: Shower Power installs onto nearly any showerhead \u2014 rain, fixed, or handheld."
],
"total_reviews": 34,
"total_answered_questions": 5,
"customization_options": {
"Color": [
{
"asin": "B09R93MDJX",
"is_selected": true,
"value": "Chrome",
"image": "https://m.media-amazon.com/images/I/31KKeBcl-NL.jpg"
},
{
"asin": "B09R9LDKZP",
"is_selected": false,
"value": "Black",
"image": "https://m.media-amazon.com/images/I/31m-FVxKB6L.jpg"
}
],
"Style": [
{
"asin": "B09R93MDJX",
"is_selected": true,
"value": "Shower Power",
"image": null
},
{
"asin": "B09R95153K",
"is_selected": false,
"value": "Shower Power Pro",
"image": null
}
]
},
"seller_id": "A1PZ4VLRD8PLH6",
"seller_name": "Ampere Tech",
"fulfilled_by_amazon": true,
"fast_track_message": "",
"aplus_present": true
}
By enabling Auto Parsing, ScraperAPI will provide you with all relevant data points including, but not limited to:
| Data Point | JSON Name |
|---|---|
| Product’s name | name |
| Product’s price | price |
| Product information like weight, dimensions and ASIN code | package dimensions, item weight, ASIN |
| Product images | images |
| Customization options like color and variants | color, style |
This is a standard response when using the Auto Parsing with Amazon product pages, so take your time to understand how data is organized to make the most out of our structured data feature.
Side Note:
Just for contrast, here’s a portion of the same response but without Auto Parsing enabled:
HTML responses are loaded with unnecessary tags, scripts, and unreadable data that makes them harder to work with – you can learn more about scraping Amazon in our tutorial.
Extracting Specific Data Points from the Response
Picking specific fields from the response is as easy as using the .json() method – in the case of Python – on the response and traversing the data using the designated pairs.
Here’s a simple example of extracting the product’s name, price, and ASIN:
import requests
product_data = []
payload = {
'api_key': 'YOUR_API_KEY',
'url':'https://www.amazon.com/dp/B09R93MDJX/?th=1',
'autoparse': 'true',
}
response = requests.get('https://api.scraperapi.com', params=payload)
product = response.json()
product_data.append({
"product_name": product["name"],
"product_price": product["pricing"],
"product_ASIN": product["product_information"]["ASIN"]
})
print(product_data)
It returns the following:
[
{
"product_name": "Ampere Shower Power - Hydropower Bluetooth Shower Speaker, Waterproof Detachable Portable, Rechargeable Mini Speaker (Chrome)",
"product_price": "$99.00",
"product_ASIN": "B09R93MDJX"
}
]
Get Data from Any Amazon Page Type
Just like for product pages, you can use Auto Parsing to get structured data from offer listings, reviews and search results. All you have to do is pass the URLs you want to parse, and ScraperAPI will take care of the rest.
For example, we can use a listing URL like:
import requests
payload = {
'api_key': 'YOUR_API_KEY',
'url':'https://www.amazon.com/s?k=gaming+controller&pf_rd_i=23508887011&pf_rd_m=ATVPDKIKX0DER&pf_rd_p=434db2ed-6d53-4c59-b173-e8cd550a2e4f&pf_rd_r=YDRBNXKBQYE60DWFAR2Y&pf_rd_s=merchandised-search-5&pf_rd_t=101&ref=nb_sb_noss_1',
'autoparse': 'true'
}
response = requests.get('https://api.scraperapi.com', params=payload)
print(response.text)
Note: We used a different syntax to show the different ways to construct the request in Python. Check our documentation to learn to use the tool with your preferred programming language. Also, when using this built-in functionality, there’s no risk of conflicting parameters between the API and the target URL.
And it’ll return all relevant data (including pagination links) in JSON:
JSON Product Example
"amazons_choice": [],
"results": [
{
"type": "search_product",
"position": 1,
"name": "PowerA Spectra Infinity Enhanced Wired Controller for Xbox Series X|S- Black",
"image": "https://m.media-amazon.com/images/I/71XhN3d9GoL.jpg",
"has_prime": true,
"is_best_seller": false,
"is_amazon_choice": true,
"is_limited_deal": false,
"stars": 4.4,
"total_reviews": 14014,
"url": "https://www.amazon.com/Infinity-Enhanced-Controller-Officially-Licensed-x/dp/B095JNZGC4/ref=sr_1_1?keywords=gaming+controller&pf_rd_i=23508887011&pf_rd_m=ATVPDKIKX0DER&pf_rd_p=434db2ed-6d53-4c59-b173-e8cd550a2e4f&pf_rd_r=YDRBNXKBQYE60DWFAR2Y&pf_rd_s=merchandised-search-5&pf_rd_t=101&qid=1674205926&sr=8-1",
"availability_quantity": null,
"spec": {},
"price_string": "$38.99",
"price_symbol": "$",
"price": 38.99
},
.
.
.
JSON Pagination Example
"pagination": [
"https://www.amazon.com/s?k=gaming+controller&page=2&pf_rd_i=23508887011&pf_rd_m=ATVPDKIKX0DER&pf_rd_p=434db2ed-6d53-4c59-b173-e8cd550a2e4f&pf_rd_r=YDRBNXKBQYE60DWFAR2Y&pf_rd_s=merchandised-search-5&pf_rd_t=101&qid=1674205926&ref=sr_pg_2",
"https://www.amazon.com/s?k=gaming+controller&page=3&pf_rd_i=23508887011&pf_rd_m=ATVPDKIKX0DER&pf_rd_p=434db2ed-6d53-4c59-b173-e8cd550a2e4f&pf_rd_r=YDRBNXKBQYE60DWFAR2Y&pf_rd_s=merchandised-search-5&pf_rd_t=101&qid=1674205926&ref=sr_pg_3",
.
.
.
Collect Localized Structured Data from Anywhere in the World [Google Search Example]
With Auto Parsing being an additional feature to the standard API, you can use all ScraperAPI’s functionalities including geotargeting to control from where your requests are getting sent. Just add the country_code parameter to your payload and set it to your target country.
import requests
payload = {
'api_key': 'YOUR_API_KEY',
'url':'https://www.google.com/search?q=project+management+software',
'country_code': 'us',
'autoparse': 'true',
}
response = requests.get('https://api.scraperapi.com', params=payload)
print(response.text)
Here’s how a standard organic result will look like in JSON:
Organic Result Example
"organic_results":[
{
"position":4,
"title":"The Best Project Management Software for 2023",
"tags":"",
"snippet":"The Best Project Management Software for 2023 · OUR TOP TESTED PRODUCTS · GanttPro · Teamwork · Zoho Projects · Celoxis · LiquidPlanner · ProofHub · Redmine.",
"highlighs":[
"Project Management Software"
],
"link":"https://www.pcmag.com/picks/the-best-project-management-software",
"date":"",
"displayed_link":"https://www.pcmag.com › ... › Project Management",
"thumbnail":null,
"sitelinks":{
"inline":[
],
"block":[
]
}
},
.
.
.
The response will also contain all elements in the SERP, including the pagination:
{
"search_information": {
"total_results": 2470000000,
"time_taken_displayed": 0.37,
"query_displayed": "project management software"
},
"shopping_link": "https://www.google.com/search?q=project+management+software&gl=US&hl=en&source=lnms&tbm=shop&sa=X&ved=2ahUKEwj1pdX-7dX8AhXSEFkFHTwTABIQ_AUoA3oECAEQBQ",
"ads": [...],
"organic_results": [...],
"related_questions": [...],
"related_searches": [...],
"pagination": {...}
}
For comparison, let’s see what happens when we changed the country code to the UK:
"organic_results":[
{
"position":1,
"title":"The Best Project Management Software for 2023 - PCMag",
"tags":"",
"snippet":"The Best Project Management Software for 2023 · OUR TOP TESTED PRODUCTS · GanttPro · Teamwork · Zoho Projects · Celoxis · LiquidPlanner · ProofHub · Redmine.",
"highlighs":[
"Project Management Software"
],
"link":"https://www.pcmag.com/picks/the-best-project-management-software",
"date":"",
"displayed_link":"https://www.pcmag.com › ... › Project Management",
"thumbnail":null,
"sitelinks":{
"inline":[
],
"block":[
]
}
},
.
.
.
In the UK version of the SERP, there’s no ad before the first organic result, and positions 2 – 5 are taken by the People Also Ask box.
Resource: You can find full JSON response samples for Google Search and Google Shopping in these links.
💡 Pro Tip
Although we’re targeting Google Search in this example, you can also use geotargeting with Amazon pages. This is especially useful in cases where, for example, you’re interested in US prices (so amazon.com is used), but you want to be specific with sellers that deliver to Canada (so CA geotargeting is enabled).
Using ScraperAPI’s functionalities alongside Auto Parsing, you’ll be able to scrape accurate localized data from any location of the world without the risk of getting banned, blocked, or missing information.
If you want to try Auto Parsing by yourself, create a free ScraperAPI account and experience all our features for seven days without any commitment. Alternatively, subscribe to one of our paid plans to take full advantage of ScraperAPI in a few minutes.
