How To Do An Amazon Competitive Analysis: Step-by-Step Guide

Why Competitive Analysis is Key to Winning on Amazon

In Amazon’s digital marketplace, competitive analysis is crucial, particularly, for business owners introducing new products.

At its core, this analysis is about understanding your place in the market; it’s about gaining insights into market share, determining brand positioning, and evaluating customer acquisition costs.

To fully grasp the impact of competitive analysis, let’s break down these components:

• Understanding market share: Competitive analysis helps in assessing market share. Knowing how much of the market your competitors command gives you a realistic picture of the potential customer base. For a new product like wireless earphones, it’s essential to understand whether the market is saturated or if a gap is waiting for your product to fill.
• Assessing Brand Positioning: This analysis allows you to see how competitors position themselves. Are they premium, budget-friendly, or somewhere in between? Understanding this helps you position your product to fill a market need and differentiate it from existing offerings.
• Understanding the customer acquisition cost: Competitive analysis sheds light on the investment required to attract and retain customers. In the competitive world of Amazon, where customer loyalty can be fleeting, knowing how much your competitors spend on acquiring customers can guide your marketing strategies.

Essentially, competitive analysis on Amazon goes beyond observing competitors. It’s a strategic approach that informs vital business decisions, helping you navigate the competitive market with data-driven insights.

By leveraging these insights, you can make informed decisions about pricing and positioning, thus setting the stage for a successful product launch and long-term market presence.

Extracting Data for Competitive Analysis on Amazon using ScraperAPI

Now, let’s put theory into practice; we will use the scenario of our company preparing to launch wireless earphones on Amazon. The challenge? Navigating through the sea of competitors and carving out a unique space in the market.

Here, ScraperAPI comes into play as our navigational tool. We’ll use ScraperAPI and Python’s requests library to gather critical product data from Amazon—a process vital for an in-depth understanding of the competition.

From price points to customer ratings, we’ll extract the essentials and then, using pandas, transform this raw data into a structured, analysis-ready format, which we will store in a CSV. 

Lastly, using Excel, we’ll extract meaningful insights from the data to complete our analysis.

Ready? Let’s get started!

Requirements

Before jumping into our analysis, setting up your environment with the necessary tools and libraries is crucial.

Here’s a step-by-step guide to getting you started:

1. Python and Library Setup: First, ensure you have Python installed on your system, ideally version 3.8 or newer, as it’s compatible with most libraries used in data analysis.
2. Get Your API Key: Create a free ScraperAPI account to get access to your API key, the Amazon Structured Endpoint, and 5,000 API credits to test the tool.
3. Installing Required Libraries: Next, open your terminal or command prompt. To install the libraries necessary for working with ScraperAPI and handling data, execute the following commands:

	pip install requests pandas

4. Setting Up Your Project: Now, let’s create a dedicated directory and a Python script where your code will reside. In your terminal or command prompt, run these commands:

mkdir amazon_analysis
touch amazon_analysis/app.py

Now that you have set up your environment let’s dive into coding and analyzing data to uncover Amazon market insights!

Understanding Amazon Search Page Layout

The first step in scoping out our competitors is to do a simple search on Amazon for our product, “wireless earphones”, and filter the featured products that are the most recommended; this is where we want to end up when launching our new product.

This is a snapshot of what you’d see when searching for “wireless earphones” on Amazon sorted by featured:

From this page, we’ll extract each product’s ASINs (Amazon Standard Identification Numbers), which will allow us to use ScraperAPI’s Amazon Product endpoint to access structured information like prices, ratings, and reviews.

Important Update

To simplify the process, you can now use our Amazon Search endpoint to collect all product ASINs from any Amazon search result page.

Just send your get() request to the endpoint alongside your API key and query, and let us do the rest. The endpoint will return all products listed in JSON format, allowing you to extract specific data points using key-value pairs.

To get the name and ASIN from the products, target the keys name and asin. To test this, let’s use the same query (wireless earphones) as the rest of the tutorial:

import requests
import json


asins = []


payload = {
   'api_key': 'YOUR_API_KEY',
   'query': 'wireless earphones',
   'country': 'us',
}


response = requests.get('https://api.scraperapi.com/structured/amazon/search', params=payload)
earphones = response.json()


all_products = earphones['results']
for product in all_products:
   if product['asin'] != None:
       asins.append(product['asin'])
   else:
       print('not found')


with open('asins.json', 'w') as f:
   json.dump(asins, f)

Here’s what you’ll get:

[
   "B0C1ZF8TVF",
   "B097F4V3FJ",
   "B0CJCHDKLF",
   "B0BYSQDWRT",
   "B0CTS23YQ8",
   "B07RGZ5NKS",
   "B0BTYCRJSS",
   "B09T5Z8L9G",
   "B096SV8SJG",
   "B0BQPNMXQV",
   "B09CY995KT",
   "B0CPJG173J",
   "B0CJDSNWRQ",
   "B0CJJHSFFP",
   "B0CSPGM8PX",
   "B0BTLVHMHW",
   "B09FT58QQP",
   "B095BV8R27"
]

Check the Amazon Search Endpoint documentation to learn more.

First, to find the ASINs on the page, let’s inspect the page. We can see that every product card is wrapped by the following div:

The product’s ASIN code is stored inside a data-asin attribute within each div element. We will use this to extract the ASIN codes of each product.

Now that we’ve figured out the HTML elements we need to target, let’s start with the actual coding part and extract some data!

Step 1: Importing Libraries

First up, import all the necessary libraries. This includes:

• requests for web scraping using Scraper API.
• csv to manage data files.
• BeautifulSoup for HTML parsing.
  sleep to pace our requests.
• pandas for data cleaning.

These Python libraries are essential for exploring and extracting the data we need from Amazon.

import requests
import csv
from bs4 import BeautifulSoup
from time import sleep
import pandas as pd

Step 2: Initializing Constants

We’ll first input our ScraperAPI key as API_KEY, and define BASE_URL with the Amazon search URL we got from our initial search for wireless earphones, and then, we create an empty list called ASINS for storing the ASINs we’ll gather.

Note: Don’t have an API key? Create a free ScraperAPI account to get 5,000 API credits to try all our tools for 7 days.

We’re using a base URL because we will be going through the first 5 pages to collect the product ASINs and will be adding onto the URL to do that.

API_KEY = 'YOUR_API_KEY'
BASE_URL = 'https://www.amazon.com/s?k=wireless+earphones'
ASINS = []

Step 3: Scraping Amazon Product ASINs

Our next step is to navigate through Amazon’s search results and extract the product ASINs from them. We aim to identify and extract ASINs from the most visible products our future customers will likely encounter first.

We start a loop to go through the first five pages of the Amazon search results for ‘wireless earphones.’

For each iteration, we construct the URL for the current page by appending '&page='  and the page number to the base URL.

for page in range(1, 6):
    url = f'{BASE_URL}&page={page}'

We prepare a payload dictating our request. It includes our API key (to authenticate with ScraperAPI) and the constructed URL.

payload = {
        'api_key': API_KEY,
        'url': url
    }

Next, we send a get() request to ScraperAPI with the payload. If the response is successful (status code 200), we parse the page content.

We’re using BeautifulSoup to parse the HTML content of the response, allowing us to navigate and extract data from it.

try:
        response = requests.get('https://api.scraperapi.com', params=payload)
        if response.status_code == 200:
            soup = BeautifulSoup(response.content, 'html.parser')

We’ll use BeautifulSoup’s select method to find all div elements with a 'data-asin' attribute containing the products’ ASINs.

We extract the ASIN and add each product ASIN found to the ASINS list, provided it’s not an empty string.

products = soup.select('div[data-asin]')
            for product in products:
                asin = product.attrs['data-asin']
                if asin != '':
                    ASINS.append(asin)

We also handle any potential errors. If the response status code is not 200, we log an error. We also catch any exceptions that might occur during the request.

After processing each page, we pause for one second using sleep(1). This is a good practice to avoid sending requests too quickly and to mimic human-like access patterns.

else:
            print(f"Error on page {page}: Received status code {response.status_code}")
    except requests.RequestException as e:
        print(f"Request failed on page {page}: {e}")
    sleep(1)