How to Scrape Quora Q&As [Full Guide]

TL;DR: Full TechCrunch Scraper

Here’s the completed code for those in a hurry:

import time
import csv
from bs4 import BeautifulSoup
from seleniumwire import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

def configure_selenium_wire_proxy(api_key):
    """Configures selenium wire to use a specified proxy with the provided API key."""
    proxy_options = {
        'proxy': {
            'http': f'http://scraperapi.render=true:{api_key}@proxy-server.scraperapi.com:8001',
            'https': f'http://scraperapi.render=true.country_code=us.premium=true:{api_key}@proxy-server.scraperapi.com:8001',
            'no_proxy': 'localhost,127.0.0.1'
        }
    }
    return proxy_options

def scroll_to_bottom(driver):
    """Scrolls the webdriver instance to the bottom of the page."""
    last_height = driver.execute_script("return document.body.scrollHeight")
    while True:
        driver.find_element(By.TAG_NAME, 'body').send_keys(Keys.END)
        time.sleep(10)  # Wait for page to load
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:  # Check if the bottom has been reached
            break
        last_height = new_height

def scrape_answers(driver):
    """Scrapes non-ad answers from a Quora page."""
    soup = BeautifulSoup(driver.page_source, 'html.parser')
    main_content = soup.find("div", attrs={"id": "mainContent"})
    all_results = main_content.find_all("div", class_="q-click-wrapper qu-display--block qu-tapHighlight--none qu-cursor--pointer ClickWrapper___StyledClickWrapperBox-zoqi4f-0 iyYUZT")
    
    # Class name indicating ads to avoid
    ad_class = "q-box dom_annotate_ad_promoted_answer"
    non_ad_results = [result for result in all_results if not result.find_parents(class_=ad_class)]
    
    return non_ad_results

def write_to_csv(results, query):
    """Writes the scraped results to a CSV file."""
    with open('quora_answers.csv', 'w', newline='', encoding='utf-8') as file:
        writer = csv.writer(file)
        writer.writerow(["Question", "Answer", "Upvotes"])
        
        for result in results:
            question = result.find('div', class_="q-text qu-dynamicFontSize--regular_title qu-fontWeight--bold qu-color--gray_dark_dim qu-passColorToLinks qu-lineHeight--regular qu-wordBreak--break-word")
            answer = result.find('div', class_="q-box spacing_log_answer_content puppeteer_test_answer_content")
            
            parent = result.find_parent("div")
            
            upvote = parent.find("span", class_="q-text qu-whiteSpace--nowrap qu-display--inline-flex qu-alignItems--center qu-justifyContent--center")

            
            answer_text = answer.text if answer else "No answer text available"
            question_text = question.text.replace("Related", "") if question else query
            upvote = upvote.text if upvote else "0"
            
            writer.writerow([question_text, answer_text, upvote])
        print(f"Written {len(results)} answers to quora_answers.csv")   

# Configure the webdriver with proxy support
chrome_options = webdriver.ChromeOptions()
API_KEY = "YOUR_API_KEY"
proxy_options = configure_selenium_wire_proxy(API_KEY)
driver = webdriver.Chrome(options=chrome_options, seleniumwire_options=proxy_options)

try:
    query = "What is the easiest way to learn to code?"
    driver.get(f"https://www.quora.com/{query.replace(' ', '-')}-q{query.replace(' ', '-')}")

    time.sleep(100)  # Initial wait for page to load
    scroll_to_bottom(driver)  # Scroll to the bottom of the page
    answers = scrape_answers(driver)  # Scrape non-ad answers
    write_to_csv(answers, query)  # Write the answers to a CSV file
except Exception as e:
    print("An error occurred:", e)
finally:
    driver.quit()  # Ensure the driver is quit properly

Before running the code, add your API key to the API_key parameter within the payload.

Note: Don’t have an API key? Create a free ScraperAPI account to get 5,000 API credits to try all our tools for 7 days.

Want to see how we built it? Keep reading and join us on this exciting scraping journey!

Why Scrape Quora?

Scraping Quora Q&As helps you get lots of useful information for different reasons:

• Stay Informed on Diverse Topics: Quora has answers and opinions on almost everything. This can help you stay updated, see what’s new, and learn from experts in all kinds of areas.
• Market Research and Competitive Analysis: Seeing what people ask and say about products or services can give you great ideas for your business and help you understand what your competitors are doing.
• Content Creation and Strategy: Find out what questions people are really interested in and what answers they like. This can guide you to create content that people will want to read or watch.

Scraping Quora the Right Way

When scraping Quora, it’s important to do it responsibly:

• Respecting Quora’s Terms of Service: Familiarize yourself with and adhere to Quora’s guidelines to avoid violating them while scraping.
• Prioritizing User Privacy: Be mindful of the privacy of Quora’s users. Avoid collecting or disseminating personal data without consent, and consider anonymizing data where possible.
• Legally Sound Practices: Ensure your scraping activities are legal. This means strictly avoiding any content not meant to be publicly accessed or shared, and bypassing login walls.

By keeping these points in mind, you can collect valuable information from Quora in a way that’s fair and respects everyone involved.

Scrape Quora Q&As with Python

In this tutorial; we will navigate through the question’s page to scrape Quora answers, methodically extracting the answers and their corresponding upvote counts. 

We will utilize automated browser simulation to ensure we capture the data as a user would see it, enabling us to capture all the dynamically loaded content on the webpage.

Requirements

Before we dive into the scraping of Quora Q&As, we must prepare our environment with the necessary tools and libraries. 

Here’s how to set everything up:

1. Python Installation: Make sure Python is installed on your system, preferably version 3.10 or later. You can download it from the official Python website.
2. Library Installations: For this guide, we will use several Python libraries – Selenium, BeautifulSoup (bs4), and Selenium-wire. Install them using the following command in your terminal or command prompt:

pip install selenium beautifulsoup4 selenium-wire

• selenium: This tool allows us to automate web browser interaction, enabling us to navigate through Quora’s pages like a real user. It’s crucial for simulating human-like behavior and dealing with dynamic content.
• bs4 (Beautiful soup): BeautifulSoup aids in parsing HTML content. It helps us sift through Quora’s complex page structure to extract the questions and answers we’re interested in. This library makes it easier to navigate and parse the web pages, extracting the necessary data efficiently.
• seleniumwire: We’ll use Selenium Wire to extend Selenium’s functionality, enabling proxy management for our web requests through ScraperAPI. This integration is vital for smoothly bypassing Quora’s anti-scraping measures, ensuring our scraping process remains efficient and uninterrupted by leveraging ScraperAPI’s Standard API in proxy mode.

Understanding Quora’s Website Layout

With our environment ready for scraping, it’s time to explore Quora’s website layout to understand how information is structured. Identifying key HTML elements that contain the Q&A data is crucial. We will extract essential details from discussions, including questions, answers, and upvote counts.

In this article, we’re focusing on the  Quora question page for the question: “What is the easiest way to learn to code?”.

This is what the page looks like on Quora:

We’ll be scraping the individual answers provided to the question on Quora. We’ll use the developer tools (right-click on the webpage and select ‘inspect’) to examine the HTML structure and locate the HTML elements holding the data we need.

This div tag contains all the answers present on the page; it’s the main tag we will be targeting first: #mainContent

This div tag contains the question:

.q-text qu-dynamicFontSize--regular_title qu-fontWeight--bold qu-color--gray_dark_dim qu-passColorToLinks qu-lineHeight--regular qu-wordBreak--break-word

Extracting answers from Quora is essential for understanding its community’s collective intelligence and insights.

By extracting the answers, we can compile comprehensive knowledge bases, identify common themes or advice, and analyze the depth of user engagement around specific inquiries.

The key to our data extraction process is the HTML element containing the answer text.

This div tag is the tag that contains the answer:

.q-box spacing_log_answer_content puppeteer_test_answer_content

Capturing the number of upvotes on answers lets us know which ones people find most helpful. It’s a quick way to see what information stands out on Quora.

This span tag contains the upvote:

.q-text qu-whiteSpace--nowrap qu-display--inline-flex qu-alignItems--center qu-justifyContent--center

The last element we need to identify to make our scraping more effective is the element that contains ad answers; we need to know this to eliminate them from our results and make them more accurate.

This div contains the ad answers:

.q-box dom_annotate_ad_promoted_answer

Note: Deleting ad answers ensures we’re only considering organic user responses.

Now we’re all set to start scraping!

Step 1: Import Libraries

First, we import all necessary Python libraries:

• time: Adds delays between actions to mimic human behavior.
• csv: Manages CSV file operations for storing our scraped data.
• BeautifulSoup (bs4): Parses HTML content for data extraction.
• seleniumwire: Enables web automation with proxy support, which is crucial for using ScraperAPI in proxy mode.
• Keys from selenium: Simulates keyboard actions (e.g., End key for page scrolling).
• By from selenium: Specifies methods for locating web elements (tags, IDs, class names), which is helpful for precise data selection during scraping.

import time
import csv
from bs4 import BeautifulSoup
from seleniumwire import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

Step 2: Configuring Proxy with Selenium Wire

Next, to enable Selenium Wire to use ScraperAPI as a proxy, we define a function configure_selenium_wire_proxy() that sets up proxy options.

This setup is needed to bypass web scraping restrictions by making our script’s requests appear as legitimate browser activity.

The function will take our Scraper API key as an argument and return a dictionary of proxy options configured for HTTP and HTTPS protocols.

We include the render=true parameter in the proxy options for ScraperAPI, which ensures the webpage is fully rendered, including JavaScript-generated content, before content is fetched.

Note: Adding the render=true would increase your API credits cost, so test the scraper without the parameter to make sure you actually need it. In most cases, Selenium can render the page on its own.

def configure_selenium_wire_proxy(api_key):
    """Configures selenium wire to use a specified proxy with the provided API key."""
    proxy_options = {
        'proxy': {
            'http': f'http://scraperapi.render=true:{api_key}@proxy-server.scraperapi.com:8001',
            'https': f'http://scraperapi.render=true.country_code=us.premium=true:{api_key}@proxy-server.scraperapi.com:8001',
            'no_proxy': 'localhost,127.0.0.1'
        }
    }
    return proxy_options

This function ensures that all web traffic from our Selenium Wire-driven browser goes through ScraperAPI, leveraging its capabilities to navigate around anti-scraping measures seamlessly.