I’ll never forget the first time I tried scraping a modern website. My selectors worked perfectly in the browser’s inspector. But when I ran my spider, everything returned None.

I checked my CSS selectors ten times. I tried XPath. Still nothing. I was going crazy.

Then I viewed the page source (Ctrl+U) and realized: the content wasn’t there. The HTML was nearly empty. Everything was loaded by JavaScript after the page rendered.

That’s when I learned: Scrapy doesn’t run JavaScript. It only sees the initial HTML. Let me show you how to handle JavaScript-heavy sites.

---

The Problem: Scrapy Doesn’t Run JavaScript

When you visit a website in a browser:

1. Browser downloads HTML
2. Browser runs JavaScript
3. JavaScript fetches data (AJAX)
4. JavaScript builds the page
5. You see the final result

When Scrapy visits the same site:

1. Scrapy downloads HTML
2. Scrapy stops here
3. JavaScript never runs
4. Dynamic content never loads
5. Your selectors find nothing

---

How to Detect JavaScript-Heavy Sites

Test 1: View Page Source vs Inspect Element

In your browser:

1. Right-click → "Inspect Element"
2. Find the element you want to scrape
3. Note its HTML structure

Then:

1. Press Ctrl+U (or Cmd+Option+U on Mac)
2. Search for the same content
3. Is it there?

If NO: Content is JavaScript-loaded.

If YES: Content is in HTML, Scrapy will work.

Test 2: Use Scrapy Shell

scrapy shell "https://example.com"

>>> response.css('.product-name::text').get()
None  # Uh oh, JavaScript site!

Test 3: Disable JavaScript in Browser

1. Open Chrome DevTools (F12)
2. Press Ctrl+Shift+P
3. Type "disable javascript"
4. Select "Disable JavaScript"
5. Refresh the page

If the page is now empty or broken, it’s JavaScript-heavy.

---

Solution 1: Find the API (Best Approach)

JavaScript-heavy sites load data from APIs. Find these APIs and scrape them directly.

How to Find APIs

Step 1: Open Network Tab

1. Open DevTools (F12)
2. Click "Network" tab
3. Filter by "XHR" or "Fetch"
4. Refresh the page

Step 2: Look for JSON Responses

Watch the network requests. Look for:

• Requests to /api/
• Requests returning JSON
• Requests with product data

Step 3: Click on Interesting Requests

Click a request → "Preview" tab

If you see your data in JSON format, you found it!

Example: Scraping the API Directly

Let’s say you find this API:

https://example.com/api/products?page=1&limit=20

Returns:

{
"products": [
{"id": 1, "name": "Product 1", "price": 29.99},
{"id": 2, "name": "Product 2", "price": 39.99}
],
"total": 100
}

Your Spider:

import scrapy
import json

class ApiSpider(scrapy.Spider):
name = 'api'

def start_requests(self):
# Scrape the API directly, not the webpage
url = 'https://example.com/api/products?page=1&limit=20'
yield scrapy.Request(url, callback=self.parse)

def parse(self, response):
data = json.loads(response.text)

for product in data['products']:
yield {
'name': product['name'],
'price': product['price']
}

# Pagination
current_page = int(response.url.split('page=')[1].split('&')[0])
total = data['total']
items_per_page = 20

if current_page * items_per_page < total:
next_page = current_page + 1
next_url = f'https://example.com/api/products?page={next_page}&limit=20'
yield scrapy.Request(next_url, callback=self.parse)

Benefits:

• Much faster (no rendering)
• Clean JSON data (no HTML parsing)
• More reliable
• Often has more data than the webpage

What the docs don’t tell you:

• APIs often have rate limiting (go slower)
• Some APIs require authentication (check headers)
• APIs might have different pagination than the website

---

Solution 2: Scrapy-Playwright (Modern Browser Automation)

When you can’t find the API, use Scrapy-Playwright to render JavaScript.

Installation

pip install scrapy-playwright
playwright install

Basic Setup

settings.py:

# Enable Playwright
DOWNLOAD_HANDLERS = {
"http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
"https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

# Optional: Use Chromium, Firefox, or Webkit
PLAYWRIGHT_BROWSER_TYPE = 'chromium'

# Launch options
PLAYWRIGHT_LAUNCH_OPTIONS = {
'headless': True,  # Run without visible browser
'timeout': 60000,  # 60 seconds timeout
}

Basic Spider

import scrapy

class PlaywrightSpider(scrapy.Spider):
name = 'playwright'

def start_requests(self):
yield scrapy.Request(
'https://example.com',
meta={'playwright': True}  # Enable Playwright for this request
)

def parse(self, response):
# JavaScript has run! Content is now in response
for product in response.css('.product'):
yield {
'name': product.css('h2::text').get(),
'price': product.css('.price::text').get()
}

Wait for Elements to Load

def start_requests(self):
yield scrapy.Request(
'https://example.com',
meta={
'playwright': True,
'playwright_page_methods': [
# Wait for specific element
{'wait_for_selector': '.product-name'},
]
}
)

Wait for Network Idle

def start_requests(self):
yield scrapy.Request(
'https://example.com',
meta={
'playwright': True,
'playwright_page_methods': [
# Wait until no network requests for 500ms
{'wait_for_load_state': 'networkidle'},
]
}
)

Scrolling (For Infinite Scroll)

def start_requests(self):
yield scrapy.Request(
'https://example.com',
meta={
'playwright': True,
'playwright_page_methods': [
# Scroll down multiple times
{'evaluate': 'window.scrollBy(0, 1000)'},
{'wait_for_timeout': 2000},
{'evaluate': 'window.scrollBy(0, 1000)'},
{'wait_for_timeout': 2000},
{'evaluate': 'window.scrollBy(0, 1000)'},
]
}
)

Click Buttons

def start_requests(self):
yield scrapy.Request(
'https://example.com',
meta={
'playwright': True,
'playwright_page_methods': [
# Click "Load More" button
{'click': 'button.load-more'},
{'wait_for_selector': '.new-products'},
]
}
)

Screenshots (Debugging)

def start_requests(self):
yield scrapy.Request(
'https://example.com',
meta={
'playwright': True,
'playwright_page_methods': [
{'screenshot': {'path': 'page.png', 'fullPage': True}},
]
}
)

---

Solution 3: Scrapy-Selenium (Older but Still Works)

Selenium has been around longer and has more examples online.

Installation

pip install scrapy-selenium

Download ChromeDriver from: https://chromedriver.chromium.org/

Setup

settings.py:

from shutil import which

SELENIUM_DRIVER_NAME = 'chrome'
SELENIUM_DRIVER_EXECUTABLE_PATH = which('chromedriver')
SELENIUM_DRIVER_ARGUMENTS = ['--headless']  # Run without visible browser

DOWNLOADER_MIDDLEWARES = {
'scrapy_selenium.SeleniumMiddleware': 800
}

Basic Spider

from scrapy_selenium import SeleniumRequest

class SeleniumSpider(scrapy.Spider):
name = 'selenium'

def start_requests(self):
yield SeleniumRequest(
url='https://example.com',
callback=self.parse
)

def parse(self, response):
# JavaScript has run!
for product in response.css('.product'):
yield {
'name': product.css('h2::text').get(),
'price': product.css('.price::text').get()
}

Wait for Elements

from scrapy_selenium import SeleniumRequest

def start_requests(self):
yield SeleniumRequest(
url='https://example.com',
callback=self.parse,
wait_time=10,  # Wait up to 10 seconds
wait_until=EC.presence_of_element_located((By.CLASS_NAME, 'product'))
)

Execute JavaScript

def parse(self, response):
driver = response.meta['driver']

# Scroll down
driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')

# Wait a bit
import time
time.sleep(2)

# Now scrape
for product in response.css('.product'):
yield {...}

---

Solution 4: Splash (Lightweight Rendering)

Splash is a lightweight JavaScript rendering service.

Installation

# Run Splash with Docker
docker run -p 8050:8050 scrapinghub/splash

Setup

settings.py:

SPLASH_URL = 'http://localhost:8050'

DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}

SPIDER_MIDDLEWARES = {
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}

DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'

Basic Spider

from scrapy_splash import SplashRequest

class SplashSpider(scrapy.Spider):
name = 'splash'

def start_requests(self):
yield SplashRequest(
url='https://example.com',
callback=self.parse,
args={'wait': 2}  # Wait 2 seconds
)

def parse(self, response):
for product in response.css('.product'):
yield {...}

---

Comparison: Which Solution to Use?

Use API Scraping When:

• You can find the API endpoints
• API has all the data you need
• You want maximum speed
• You want clean JSON data

Pros: Fast, reliable, clean data Cons: Requires finding API, might need authentication

Use Scrapy-Playwright When:

• Modern sites (2020+)
• Need full browser features
• Complex JavaScript interactions
• Want the best tool

Pros: Modern, fast, feature-rich Cons: Requires Playwright installation

Use Scrapy-Selenium When:

• Older sites
• Need Selenium-specific features
• More examples available online
• Already familiar with Selenium

Pros: Mature, lots of examples Cons: Slower, resource-heavy

Use Splash When:

• Want lightweight rendering
• Already using Scrapinghub services
• Need something between Scrapy and full browsers

Pros: Lightweight, separate service Cons: Extra infrastructure, learning curve

---

Real-World Example: Infinite Scroll Site

Many modern sites use infinite scroll. Here’s how to handle it:

With Playwright

import scrapy

class InfiniteScrollSpider(scrapy.Spider):
name = 'infinite'

def start_requests(self):
yield scrapy.Request(
'https://example.com/products',
meta={
'playwright': True,
'playwright_include_page': True,  # Keep page object
},
callback=self.parse
)

async def parse(self, response):
page = response.meta['playwright_page']

# Scroll and wait multiple times
for i in range(10):  # Scroll 10 times
# Scroll to bottom
await page.evaluate('window.scrollTo(0, document.body.scrollHeight)')

# Wait for new content
await page.wait_for_timeout(2000)

# Get the fully loaded HTML
content = await page.content()

# Close page
await page.close()

# Parse with Scrapy selectors
from scrapy.http import HtmlResponse
new_response = HtmlResponse(
url=response.url,
body=content.encode('utf-8')
)

for product in new_response.css('.product'):
yield {
'name': product.css('h2::text').get(),
'price': product.css('.price::text').get()
}

---

Performance Considerations

JavaScript Rendering is SLOW

Normal Scrapy:

• 10-50 pages per second

With Playwright/Selenium:

• 1-5 pages per second

Rendering JavaScript is 10-50x slower!

Optimization Strategies

1. Only render when necessary

def start_requests(self):
# Check if page needs JavaScript
if self.needs_javascript(url):
yield scrapy.Request(url, meta={'playwright': True})
else:
yield scrapy.Request(url)  # Normal request

2. Use concurrent browsers

# settings.py
PLAYWRIGHT_MAX_CONTEXTS = 4  # Run 4 browsers in parallel

3. Cache rendered pages

HTTPCACHE_ENABLED = True  # Don't re-render during development

4. Prefer APIs

Always try to find APIs first. They’re much faster.

---

Common Mistakes

Mistake #1: Not Waiting Long Enough

# BAD (content might not load)
meta={'playwright': True}

# GOOD (wait for content)
meta={
'playwright': True,
'playwright_page_methods': [
{'wait_for_selector': '.products-loaded'}
]
}

Mistake #2: Forgetting to Enable Playwright

# BAD (won't render JavaScript)
yield scrapy.Request(url)

# GOOD
yield scrapy.Request(url, meta={'playwright': True})

Mistake #3: Using Rendering for Everything

# BAD (slow for no reason)
for url in urls:
yield scrapy.Request(url, meta={'playwright': True})

# GOOD (only render when needed)
for url in urls:
if 'product' in url:
yield scrapy.Request(url, meta={'playwright': True})
else:
yield scrapy.Request(url)

---

Quick Decision Tree

Is content in page source (Ctrl+U)?
├─ Yes → Use normal Scrapy
└─ No → JavaScript content
│
Can you find the API?
├─ Yes → Scrape API directly (BEST)
└─ No → Need browser
│
Modern site (2020+)?
├─ Yes → Use Scrapy-Playwright
└─ No → Use Scrapy-Selenium

---

Summary

JavaScript content isn’t in initial HTML:

• Scrapy doesn’t run JavaScript
• Need special tools to render pages

Four solutions:

1. Find API (best, fastest)
2. Scrapy-Playwright (modern, recommended)
3. Scrapy-Selenium (older, still works)
4. Splash (lightweight alternative)

Always try APIs first:

• 10-50x faster
• Cleaner data
• More reliable

When rendering JavaScript:

• Wait for content to load
• Only render when necessary
• Use concurrent browsers
• Cache during development

Key insight:

• View page source (Ctrl+U) to check if content is there
• Inspect Element shows final result after JavaScript
• These are different!

Start by checking page source. If content is there, use normal Scrapy. If not, find the API or use Playwright.

Happy scraping! 🕷️