Debugging 403, 429 and CAPTCHA Errors with Proxies

Your scraper worked fine on 50 requests. At 500, it started returning 403 Forbidden. At 5,000, you're staring at 429 Too Many Requests and CAPTCHA pages with a 200 status code. Adding more proxies didn't help, or helped for ten minutes.

The mistake most developers make here is changing several things at once: new proxies, new headers, more retries. This guide gives you a repeatable way to find out why you're being blocked so that you can fix the right thing.

What we'll cover

• What 403, 429, and CAPTCHA responses actually mean
• A step-by-step isolation process (and a small Python probe script)
• Detecting "soft blocks" that return HTTP 200
• Backoff, Retry-After, and pacing done properly
• Tracking proxy health so bad IPs get benched automatically
• When the problem isn't the IP at all

Examples use Python and requests, with placeholder proxies from the documentation range (203.0.113.x).

What each signal usually means

Note that 407 is not a block at all. It's your proxy configuration, and it's the first thing to rule out.

Step 1: Isolate where the block happens

Before touching your scraper, test the target with the simplest possible client. If curl through the proxy works but your script fails, the problem is in your code. If both fail, it's the IP or the target.

# Direct (no proxy)
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/products

# Through the proxy
curl -s -o /dev/null -w "%{http_code}\n" -x http://user:pass@203.0.113.10:8080 https://example.com/products

Then reproduce the comparison in Python with a small probe that varies one thing at a time:

import requests

TARGET = 'https://example.com/products'
PROXY = 'http://user:pass@203.0.113.10:8080'
PROXIES = {'http': PROXY, 'https': PROXY}

BROWSER_HEADERS = {
   'User-Agent': (
       'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
       '(KHTML, like Gecko) Chrome/124.0 Safari/537.36'
   ),
   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
   'Accept-Language': 'en-US,en;q=0.9',
}

def probe(label, **kwargs):
   try:
       r = requests.get(TARGET, timeout=(5, 15), **kwargs)
       print(f'{label:<28} {r.status_code}  {len(r.content):>8} bytes')
   except requests.RequestException as exc:
       print(f'{label:<28} ERROR {type(exc).__name__}')

probe('direct, default headers')
probe('direct, browser headers', headers=BROWSER_HEADERS)
probe('proxy, default headers', proxies=PROXIES)
probe('proxy, browser headers', proxies=PROXIES, headers=BROWSER_HEADERS)

Read the results like a truth table:

• Direct works, proxy fails: the proxy IP is the problem (reputation, type, or geo).
• Default headers fail, browser headers pass: the site checks the User-Agent and friends. The default python-requests/x.y agent is an easy signal to filter.
• Everything fails: the page may need JavaScript, a cookie, or a login. The proxy is not the issue.
• Everything passes: your problem is volume or pacing, so move on to Step 3.

Also confirm the proxy really is being used by calling an IP-echo endpoint such as https://api.ipify.org?format=json through it. A surprising number of "proxy problems" are requests silently going out directly.

Step 2: Detect soft blocks

A response code of 200 doesn't mean you got the page. Many sites serve a CAPTCHA or an "access denied" page with a success status, and your parser then silently stores garbage. Classify every response before you trust it:

BLOCK_MARKERS = (
   'captcha',
   'unusual traffic',
   'access denied',
   'verify you are human',
   'are you a robot',
)

def classify(response):
   if response.status_code == 407:
       return 'proxy_auth'
   if response.status_code == 429:
       return 'rate_limited'
   if response.status_code in (401, 403):
       return 'blocked'

   snippet = response.text[:5000].lower()
   if any(marker in snippet for marker in BLOCK_MARKERS):
       return 'challenge'
   return 'ok'

The marker list is site-specific, so tune it to the challenge pages you actually see. A cheaper complementary check is to assert that the content you expect (a product title, a JSON key) is present, and treat its absence as a failure.

Step 3: Fix 429s with backoff and pacing

A 429 is the friendliest error you'll get, because the server is telling you exactly what to do: slow down. Many servers include a Retry-After header, either as several seconds or as an HTTP date. Honor it:

import random
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

def retry_after_seconds(response, default=30):
   value = response.headers.get('Retry-After')
   if not value:
       return default
   if value.isdigit():
       return int(value)
   try:
       when = parsedate_to_datetime(value)
       return max(0, (when - datetime.now(timezone.utc)).total_seconds())
   except (TypeError, ValueError):
       return default

def polite_sleep(base=1.0, jitter=1.5):
   # randomized delay between requests, so traffic isn't perfectly periodic
   time.sleep(base + random.uniform(0, jitter))

Three principles prevent most 429s:

1. Limit concurrency per IP, not just overall. Ten threads on one IP is a burst. Ten threads across ten IPs is not.
2. Add jitter. A request every exactly 1.000 seconds looks robotic.
3. Use exponential backoff on repeated failures. If a 429 comes back after waiting, wait longer next time instead of retrying at the same speed.

If you're still rate-limited at a sensible pace, the limit probably applies per IP, and the fix is a larger pool rather than a faster loop.

Step 4: Track proxy health and bench bad IPs

Random rotation keeps sending traffic to IPs that are already burned. Instead, record outcomes and give failing proxies a cooldown:

import random
import time
from collections import defaultdict

class ProxyPool:
   def __init__(self, proxies, cooldown=300, max_failures=3):
       self.proxies = list(proxies)
       self.cooldown = cooldown
       self.max_failures = max_failures
       self.failures = defaultdict(int)
       self.benched_until = {}

   def get(self):
       now = time.time()
       available = [p for p in self.proxies if self.benched_until.get(p, 0) <= now]
       if not available:
           raise RuntimeError('No healthy proxies: slow down or add more IPs')
       return random.choice(available)

   def report(self, proxy, outcome):
       if outcome == 'ok':
           self.failures[proxy] = 0
           return
       self.failures[proxy] += 1
       # a 429 benches the IP immediately; other failures need a streak
       if outcome == 'rate_limited' or self.failures[proxy] >= self.max_failures:
           self.benched_until[proxy] = time.time() + self.cooldown
           self.failures[proxy] = 0

And a fetch function that ties the pieces together:

import requests

def fetch(url, pool, attempts=4):
   for _ in range(attempts):
       proxy = pool.get()
       try:
           r = requests.get(
               url,
               proxies={'http': proxy, 'https': proxy},
               headers=BROWSER_HEADERS,
               timeout=(5, 15),
           )
       except requests.RequestException:
           pool.report(proxy, 'error')
           continue

       outcome = classify(r)
       pool.report(proxy, outcome)

       if outcome == 'ok':
           return r
       if outcome == 'rate_limited':
           time.sleep(retry_after_seconds(r))
       elif outcome == 'proxy_auth':
           raise RuntimeError('Proxy credentials rejected (407). Check auth or IP whitelist.')

       polite_sleep()

   raise RuntimeError(f'Failed to fetch {url} after {attempts} attempts')

Log the outcome, proxy, and URL for every request. After a few hundred requests, you'll be able to see whether failures cluster on specific IPs (bad IPs), specific URLs (protected pages), or specific times (rate-limit windows). That pattern tells you which fix to apply.

Step 5: When it's the IP type

If your probe showed that browser-like headers pass directly but fail through the proxy, you're dealing with IP reputation. Different proxy types carry different levels of trust:

A few related checks:

• Dedicated vs shared: a shared IP can be burned by someone else before you send a single request.
• Geo mismatch: if your Accept-Language says en-US but your exit IP is in another country, or your IP is in Germany while you request a US-only price page, you will get blocked or served different content.
• Burned static IPs: a static IP that has been hammered for days may need a rest or a replacement. Rotation isn't just for volume; it's hygiene.

Step 6: When it isn't the IP at all

If you've tried clean IPs and sensible pacing and still get 403s, look beyond the proxy:

• Session and cookies: Many sites set a cookie on the first page and expect it on the next. Use a requests.Session and keep the same IP for the whole flow (a sticky session if your provider supports one).
• Inconsistent headers: A Chrome User-Agent with a missing Accept-Language, or headers in an unusual order, is a signal. Keep the set realistic and consistent per session.
• TLS fingerprinting: Some bot-protection systems inspect the TLS handshake itself, and Python's default stack looks different from a real browser's. If headers and IPs are fine but a site still refuses you, a real browser (Playwright or Selenium) or an HTTP client that mimics browser TLS such as curl_cffi may be the next thing to test.
• JavaScript-rendered content: If the data is loaded by JavaScript, plain requests get an empty shell. Look for the JSON API call the page makes in your browser's network tab, or use a headless browser.
• Behavioral signals: Hitting deep URLs with no prior navigation, or requesting pages at inhuman speed, can raise a risk score even from a perfect IP.

A word on CAPTCHAs

A CAPTCHA is not an error to defeat. It's a signal that your traffic looks risky. The most reliable strategy is to trigger fewer of them: better IP quality, lower rate, consistent sessions, realistic headers. If you're seeing them constantly, treat that as a symptom and work back through the steps above.

It's also worth asking whether you should be scraping that page at all. Check for an official API or data export first, read the site's terms of service and robots.txt, and only collect data you're permitted to use. Circumventing access controls on sites that forbid automated access can create legal problems that no proxy will solve.

Choosing proxies for testing

To follow along, you need proxies that let you compare types on the same target. I used a provider that offers datacenter, ISP, residential, and mobile IPs with both HTTP/HTTPS and SOCKS5 so that I could swap types without changing code. Disclosure: [add your affiliate or sponsorship note here if applicable]. Whichever service you use, buy a small amount of each type you're considering, run the probe script from Step 1, and compare success rate and latency on your target before committing to a larger plan.

Quick reference

• 407? Fix proxy auth first. It isn't a block.
• Isolate: compare direct vs proxy, default vs browser headers.
• Classify every response, including 200 pages that are really CAPTCHAs.
• 429: honor Retry-After, add jitter, limit per-IP concurrency, then grow the pool.
• Bench bad IPs with a cooldown instead of rotating unquestioningly.
• 403 on clean IPs: check sessions, headers, TLS fingerprint, and JavaScript rendering.
• Constant CAPTCHAs: reduce the triggers, and check whether an official API exists.

Got a blocking pattern that isn't covered here? Share it in the comments, and I'll add it to the guide.