How to Gain a Competitive Edge with Amazon Data (+ Templates)

In this article, we’ll explore how to collect data from the biggest ecommerce marketplace, how to unlock your products’ growth potential, and a simple way to scrape Amazon data at scale.

How Analyzing Amazon Data Gives You a Competitive Edge

According to AMZscout, “there are over 600 million products listed on the marketplace, and 12 million of these items are sold by Amazon itself, [dominating] 37.8% of the ecommerce market share in the US. ”These numbers and the different types of information created (reviews, products, pricing, categories, etc.) on the platform make Amazon a goldmine of data.

By collecting and analyzing these data points, you’re able to eliminate the guesswork and start uncovering powerful insights that’ll help you:

• Launch successful new products
• Grow your sales
• Understand how to make your offerings unique
• Stay ahead of the curve by spotting trends

Although this is not an exhaustive list, we will show you five powerful strategies made possible by extracting Amazon data and a simple method to collect this data.

1. Monitor Pricing Trends in Your Market

Your pricing strategy is critical for product success. And thanks to the internet, it is easier than ever for consumers to compare prices. In fact, “Nearly 70 percent of online shoppers use Amazon to compare products found on a brand’s website,” according to Episerver.

By keeping an eye on your competitors’ price points, you can inform yours. Let’s say you’re about to launch a new product. Now, you can collect price data from products in the same category or products that satisfy a similar pain and set a competitive price based on this information.

However, by monitoring price data within your niche (extracting price data periodically), you can build a historical dataset that can help you find openings for offers – like periods when your competitors’ prices go up – or find shifts in pricing most or all competitors are following.

Price data can be useful in many ways, so keep your mind open.

But of course, we need to scrape Amazon product pages to extract price data before we make our analysis.

Automating Price Monitoring

To automate this process, we’ll use ScraperAPI’s DataPipeline, a low-code solution to:

• Bypass Amazon’s anti-scraping mechanisms
• Schedule our requests with a couple of clicks
• Export our data to a Webhook or download it as a JSON file
• Automate geotargeting

And simplify the development process.

1. Create Your Account

First, let’s create a free ScraperAPI account to get access to the tool. Once on your dashboard, click on “Create new no-code project.”

This will open a new window where you can name the project and choose the way you want to interact with the tool.

For this particular case, we’ll be using the “Amazon Product Pages” template.

2. Add Your Target Product ASINs

Inside the template, it’ll ask for some input to understand how you want the job to be done, starting with the ASIN numbers we want to target. (*Note: ASIN stands for Amazon Standard Identification Number, a unique number assigned to every product.)

To speed things up, here’s a list of ASIN numbers in our scraping Amazon ASINs at scale tutorial. It has a simple code snippet you can use to collect hundreds of product ASIN numbers from Amazon searches, so we suggest you check it out.

You can use this list to test the tool

For transparency, the tool will calculate how many API credits you’ll use when running the job. You can see this information on the right.

3. Set your Geotargeting Parameters

The next parameters are for geotargeting. The country code will define the location of the proxies you want ScraperAPI to use when running the job, while the TLD will define what domain the job will target (.com, .es, .ca, .uk, etc.).

Here’s a list of accepted values

TLD will default to the US version (.com) if it is not defined. However, if the country code is not defined, it will use any proxy, regardless of location.

A better practice is to have control over the proxies you’re using and avoid getting different data based on the location of the request.

4. Choose Your Preferred Output Method

Select download for the output, but even if you choose webhook, you can download the JSON file later, so don’t worry.

Note: You’ll receive a JSON response to your Webhook.

5. Schedule Your Job

Here’s where the magic happens.

Click on schedule, and set the frequency you want ScraperAPI to run this project. You’ll get a new file – or response to your Webhook – in every run, so you can integrate the data easily.

For advanced users, there’s an option to use the CRON format.

After the schedule is set, you’ll see the next three dates the job will run.

6. Choose How Often You Want to Be Notified

For this test, we are running it once a week, so we’ll receive a notification after a run is completed.

7. Enjoy Your Data

Click on “START SCRAPING” at the bottom right corner and…

There was an error! The tool charges you for every request it sends, so to avoid overcharging you and wasting API credits, it’ll tell you if you have any duplicates. Here’s the revised list with the duplicates deleted. Run the job now, and it’ll take you to the “all projects” view.

Of course, you can click on the project to see its status. (This was a small project, so it took a few minutes.)

It also turned every product page into a structured JSON file.

Including the price.

"pricing":"$61.99","list_price":"$72.99$72.99","shipping_price":"FREE"

2. Discover How to Differentiate Yourself

Messaging is a powerful tool to make your products and offerings unique, and the fastest way to create something original is by knowing what’s already out there. How can we use Amazon data for this? Well, a fundamental part of Amazon product listings is descriptions.

Like this one:

And this one:

And we can find this information within the “full_description” and “small_description” values. You can find these key pairs within the file you just downloaded.

"small_description":"About this itemCUSTOM DYNAMIC CAPSULE: The built-in dynamic mic capsule helps you capture deep lows, delicate highs, and crisp mid-tones. Ultra wide frequency response of 40Hz-16KHz present crystal clear and clean sound. Cardioid polar patterns reject any unwanted background noise in your recording and streaming. UNLOCK THE POTENTIAL IN MAONO LINK: Choose the perfect tone from your microphone, and you can turn on/off the RGB light or switch in 9 different colors, also supports EQ, limiter, and compressor adjustment in USB mode. USB AND XLR CONNECTIVITIES: Plug-and-play when recording on desktop or laptop by plugging the 8.2ft USB cable directly. XLR analog output for connecting to all professional interfaces, mixer or portable PA system when podcasting in your studio. 2-in-1 SMART KNOB: Everything in your control, the podcast microphone have a 2-in-1 smart knob to adjust mic gain, headphone volume, one click mute key with LED indicator help you instantly mute the mic, and real-time monitoring of 3.5 mm headphone jack monitor output voice as desired by recording professionals. CONTROLLED RGB LIGHTING: The gaming microphone has dynamic multi-color gradients and 8 static solid-color effects, which creates a personalized gaming atmosphere and makes your streaming more appealing. You can press the button at the bottom of the microphone to switch the different color and turn on/off the RGB, turn off the RGB need 2 seconds. METAL CONSTRUCTION: Higher standard microphone materials and craftsmanship to protect the microphone from being hurt easily as well as the internal structure. WHAT'S IN THE BOX: Dynamic Microphone, Shock Mount, 2.5m USB-C to USB-A/C Cable, 5/8'' to 3/8'' Adapter, User's Manual, 18 Months Free Warranty (NOTE: XLR CABLE NOT INCLUDED)"

With this data, you can analyze all this text to find common words used and the number of times they appear in descriptions, how many products use these words and phrases, etc.

To help your writers use this data, you can build word clouds that help them to see what’s being overused, and help them figure out new ways to talk about your products.

3. Find Out What Consumers Think About Your Product/s

Wouldn’t it be great to know how people feel about a particular product or category?

Usually, companies tend to use surveys and customer interviews to get this kind of information, but it can take a lot of time and effort. An alternative road is collecting product reviews and analyzing those texts using sentiment analysis. This process can be done at a larger scale than surveys and interviews and can uncover impressive insights you won’t get from raw numbers.

Some applications for review data are:

• Shifts in market feelings – for example, you can spot negative trends if negative sentiment is growing within a product category.
• Monitor how customers are responding to your own products – if you are promoting products on Amazon, you can extract your products’ reviews and spot unsatisfied customers or rising negative trends, so your team can work more proactively.
• Understand what’s working for your competitors – you can find common words and phrases in positive reviews to understand what’s making their products so popular.

Automating Product Review Scraping

1. Open Amazon Reviews Structured Data Source

In this opportunity, we’ll be using the “Amazon Reviews” template to gather all the reviews from our previous list of ASINs.

2. Paste Your Target ASINs and Set Geotargeting

Just like we did before, let’s paste our list of ASINs and make sure to target the US within the additional options and filters section.

3. Set Output and Schedule Your Review Scraper Job

We’ll go with weekly again, but if you’re targeting a highly-sold product, maybe daily would be a great option.

But you can also use the cron format to provide more specific scheduling – like every eight hours or every three days.

4. Run Your Scraping Project

When you run your projects using the Review endpoint, you’ll get all the reviews associated with the targeted products instead of their product page details.

Basically, turning the dedicated review pages into JSON data.

To avoid embedding the entire JSON, here’s what a single review from the response looks like:

{
"stars":5,
"date":"Reviewed in the United States 🇺🇸 on July 15, 2023",
"verified_purchase":true,
"manufacturer_replied":false,
"username":"Alan Jackson",
"user_url":"https://www.amazon.com/gp/profile/amzn1.account.AHX2YT7Z53OFPGA3CNCE3N7XNAWQ/ref=cm_cr_arp_d_gw_btm?ie=UTF8",
"title":"5.0 out of 5 stars\n\n\n\n\n\n\n\n \n \n A Nice Surprise",
"review":"A search during Prime Day showed this product as a result. The price was amazing so I jumped on the deal. When the package arrived a few days later I suddenly realized it was a refurb. Surprise! My heart sank. Before opening the box I went back to the site to check. Sure enough. That fact wasn't in the item description, but was instead shown above the add to cart button. Not exactly cool in my book.Feeling more than a little frustrated I opened the box. That's when the 2nd surprise, and nicer, surprise happened. The boom stand was in perfect condition. No scratches, no missing parts, like new, and ready to use. Gator makes quality gear. So new or used it gets 5-stars from me.",
"review_url":"https://www.amazon.com/gp/customer-reviews/RWB5Y9NA16TNN/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B082YHPC3Z",
"total_found_helpful":null,
"images":[
],
"variation":{
"style":"4000 Series",
"pattern_name":"Stand"
},
"video_url":null
}

Use this link to check the entire JSON response, or download your file once it’s ready.

4. Identify Common Threads Around Your or Similar Products

When it comes to data analysis, more data can be a critical component in uncovering important insights and drawing powerful conclusions.

By combining the data from both reviews and product pages details, you can answer very complex questions, like:

• What are the most reviewed Amazon products?
• What are the initial and current number of customer reviews for each product?
• How do the reviews in the first 90 days after a product launch compare to the price of the product?
• How do the reviews in the first 90 days after a product launch compare to the days available for sale?

You can even map the keywords in the reviews against the review ratings to help train sentiment models or find opportunities to improve your offerings and products by identifying common negative words and looking for them within your reviews.

All this information will allow you to develop more accurate strategies based on what consumers are actually feeling, using the words they use to express themselves.

5. Find New Competitors Entering Your Niche

Another Amazon space worth monitoring is its search engine. Although you might not think too much about it, many pieces of research have found that “50% of product searches start on Amazon”.

Just as you keep an eye on your competitors on Google search results, you must be analyzing Amazon search results to stay on top of new players entering the field, what products are getting on the top of searches, and why.

It’s also a great way to discover new products to monitor and get messaging ideas.

Automating Amazon Search Monitoring

You know the drill. We’ll be using ScraperAPI’s Amazon Search endpoint to schedule regular scraping jobs without having to write the code ourselves.

1. Add Your List of Search Terms and Specified Geotargeting

Instead of product ASINs, the Amazon Search endpoint uses a list of search terms as input.

For this example, we used a short list of search terms around kitchen knives and set our geotargeting to the US.

You can build a list of search terms (also known as keywords in the SEO space) based on the keywords people use to find you on other search engines like Google.

Note: If you don’t have any experience doing keyword research, a great place to start is by checking your ranking information in Google Search Console.

2. Set Your Output Options and Scheduling

From there, it’s the same as before: set the way you want to receive your data, add the scheduling parameters you prefer, and get started.

Of course, you don’t have to enable scheduling if you’re looking for one-time data, so don’t feel pressured to use it if you don’t need to.

3. Enjoy Your Data

After the project runs, it’ll return all the elements within an Amazon search result – for each search term used – in JSON format.

Turning this:

Into this:

{
"type":"search_product",
"position":1,
"name":"Kitchen Knife Set with Block, Full Tang 15 Pcs Professional Chef Knife Set with Knife Sharpener, Food Grade German Stainless Steel Knife Block Set, Steel-king Series with Gift Box",
"image":"https://m.media-amazon.com/images/I/71qXyojrOIL.jpg",
"has_prime":true,
"is_best_seller":false,
"is_amazon_choice":false,
"is_limited_deal":false,
"stars":4.7,
"total_reviews":243,
"url":"https://www.amazon.com/BRODARK-Professional-Sharpener-Certified-Steel-king/dp/B0B974GRLY/ref=sr_1_1?keywords=kitchen+knives+with+block&qid=1689748067&sr=8-1",
"availability_quantity":null,
"spec":{
},
"price_string":"$153.79",
"price_symbol":"$",
"price":153.79
}

Download the file after your project is done running, or check the full response in this link.

Wrapping Up

ScraperAPI takes care of all the complexity and automates extracting the data periodically for you in just a couple of clicks.

Best of all, it returns all this data in a predictable and easy-to-manipulate JSON format you can feed to your models with almost no modifications beyond – maybe – picking the specific key pairs you want to use.

If you want to try it yourself, create a free ScraperAPI account and get 5,000 free API credits for seven days – after subscribing, the trial won’t start until you’re ready.

Until next time, happy scraping!