Added

• Adaptive rate limiting (auto-enabled)
  Per-domain smart throttle on every stealth driver — no settings or meta flags.

  • Tracks HTTP 429, Retry-After, and response latency per domain + driver.
  • AIMD spacing: backs off on rate limits, eases delay after success streaks.
  • Behavioral timing jitter on basic/turbo is folded into the throttle wait.
  • Stats: stealth/throttle/waits, stealth/throttle/wait_ms,
    stealth/throttle/rate_limited, stealth/throttle/retry_after
    (each with a {driver} breakdown where applicable).

  New module: scrapy_stealth/strategies/throttle.py.

Added

• Behavioral fingerprinting engine (auto-enabled)
  Human-like interaction runs automatically on every stealth driver.

  • browser — after each GET navigation: viewport emulation from the active
    fingerprint profile (desktop vs mobile), Bezier-curved CDP mouse paths via
    Input.dispatchMouseEvent (mouseMoved), CDP scroll (mouseWheel), and an
    occasional keyboard nudge.
  • basic / turbo — profile-seeded pre-request timing jitter (~30–350 ms);
    no DOM is available on HTTP drivers, so mouse/scroll are not simulated there.

  New package: scrapy_stealth/behaviors/ (engine, patterns, viewport,
  noise, timing). Exports include simulate_hover() for custom CDP mouse paths.

Changed

• Browser behavioral input — CDP instead of JavaScript
  Mouse and scroll replay previously used document.dispatchEvent() (untrusted,
  invisible to the OS cursor). They now use Chrome DevTools Protocol input events,
  which anti-bot behavioral checks are more likely to treat as real browser input.

Added

• Smart Proxy Management
  Per-domain proxy health tracking for turbo/basic engines: dead proxies (407,
  CONNECT aborted, tunnel errors) and repeated blocks (403 by default) open a
  temporary cooldown, skip the bad entry during rotation, and automatically
  switch to the next proxy in STEALTH_PROXIES. Telemetry is exposed in
  crawler.stats: stealth/proxy/connection_failures,
  stealth/proxy/cooldowns, stealth/proxy/rotations (each with a {driver}
  breakdown), plus stealth/proxy/last_connection_failure and
  stealth/proxy/last_cooldown (host:port only — credentials never appear
  in stats). Controlled by STEALTH_PROXY_HEALTH, STEALTH_PROXY_CIRCUIT_AFTER,
  STEALTH_PROXY_COOLDOWN_S, and STEALTH_PROXY_CIRCUIT_CODES.

• STEALTH_RECYCLE_AFTER_BANS in Scrapy settings
  The middleware now loads STEALTH_RECYCLE_AFTER_BANS from settings.py or
  spider custom_settings on spider open (same pattern as STEALTH_DRIVER and
  STEALTH_PROXIES).

Fixed

• Smart Proxy Management — cooldown log spam
  Repeated failures on a proxy already in cooldown no longer re-print the
  cooldown warning on every request.

Changed

• Random default browser profile
  Removed static DEFAULT_PROFILE (chrome_147). When no profile is set on a
  request, engines pick a weighted random profile from the fingerprint pool via
  ProfileRotator. Pin a profile with meta["stealth"]["profile"] or
  BasicEngine(profile="chrome_147").

Fixed

• Browser driver — Cloudflare challenge wait on 403/503
  The browser engine now runs the JS challenge wait loop on 403/503 interstitials
  (e.g. Cloudflare “Just a moment” or “Performing security verification”), not
  only on HTTP 2xx. Challenge pages poll for up to BROWSER_CHALLENGE_TIMEOUT_S
  (default 30s) instead of returning challenge HTML immediately.

• Browser driver — JPG/PNG/binary asset bodies
  Chrome’s built-in image viewer returns HTML (<img src="...jpg">) in the DOM.
  Direct GET/HEAD to asset URLs (.jpg, .png, .gif, .pdf, …) now return
  raw bytes: CDP Network.getResponseBody first, then in-page fetch() when the
  network/DOM response is HTML. Prefers the latest 2xx network response over an
  earlier 403 challenge body. Fixes CDN assets behind Cloudflare (e.g.
  scdn.autodoc.de/.../*.jpg).

• wreq.emulation import typo
  Fixed from wreq.eulation import Profile in profile resolution that caused
  startup failure with a misleading Visual C++ runtime error on Windows.

Added

• BROWSER_CHALLENGE_TIMEOUT_S — max seconds to wait on JS challenge /
  Cloudflare interstitial pages (default 30.0). Configurable via settings /
  scrapy_stealth.config.config.

• Cloudflare Turnstile / managed-challenge detection — expanded signatures
  for challenges.cloudflare.com, cf-turnstile, “Verify you are human”, and
  “Performing security verification” titles.

• Turbo driver — HTTP/3 (QUIC) support
  Opt-in via config.HTTP3 = True or meta["stealth"]["http3"] = True.
  Uses curl_cffi CurlHttpVersion.V3 with HTTP/3-capable impersonate presets
  (e.g. chrome150). Requires a UDP-capable proxy for QUIC.

• Turbo driver — browser header order
  Turbo sends cookies through curl_cffi’s cookies API (not a raw Cookie
  header) so they don’t disrupt the header order applied by the impersonate
  preset.

• Turbo impersonate presets bumped to chrome150
  Chromium-family profiles now map to curl_cffi’s latest Chrome preset.

• Dependency: curl_cffi>=0.16.1
  Required for HTTP/3 options and updated curl-impersonate backend.

[0.6.16] - 2026-08-19

Changed

• Browser driver defaults to visible Chrome (headless=False)
  Explicit driver="browser" and driver="auto" browser fallback now open a visible
  window by default. Set meta={"stealth": {"headless": True}} or BROWSER_HEADLESS = True
  to opt into headless mode.

Added

• Browser cookie handoff
  After each browser response, tab cookies are read via CDP and exposed on the response as
  meta["stealth"]["browser_cookies"] and meta["stealth"]["browser_cookie_header"].
  When COOKIES_ENABLED and BROWSER_EXPORT_COOKIES are on (both default), cookies merge
  into Scrapy's jar so follow-up basic/turbo requests reuse the session (login with
  browser → scrape with turbo). Stats: stealth/browser_cookies_exported.

• Browser form POST — hidden field merge
  Urlencoded POST bodies on the browser driver automatically merge hidden <form> fields
  (e.g. csrf_token) from the loaded page before in-page fetch().

• driver="auto" POST fallback
  When turbo/basic POST gets a JS challenge or session ban (403/429/503, Cloudflare, etc.),
  middleware retries once with the browser driver using the same method, body, and headers.
  Stats: stealth/fallbacks/method/post (and put, patch, delete).

• Proxy-Seller sponsor
  README and AGENTS.md now include Proxy-Seller with affiliate link, promo code FAWAD15, and logo assets under docs/static/sponsors/.

Added

• POST / PUT / PATCH / DELETE on all drivers
  basic, turbo, and browser honor the same Scrapy Request fields — method,
  body, Cookie, and custom headers (Content-Type, Authorization, etc.).

• Single request builder for all drivers
  build_stealth_request() in scrapy_stealth.utils.network.request validates
  and normalizes method, URL, body, Cookie, and custom headers once. Browser
  POST uses in-page fetch() via browser_http_fetch().

• README and example spider
  New “POST, headers, and cookies” section with live test URLs
  (postman-echo.com, quotes.toscrape.com, jsonplaceholder.typicode.com).
  examples/full_spider.py demonstrates JSON POST on all three drivers and form
  login via browser.

Fixed

• Browser POST — same-origin setup
  Load the target URL (GET), not the site root, before in-page fetch(). Fixes
  TypeError: Failed to fetch when the root redirects elsewhere
  (e.g. postman-echo.com → www.postman.com).

• Browser POST — brotli decode error in Scrapy
  Strip content-encoding and content-length from browser fetch responses; the
  body from arrayBuffer() is already decoded.

• Basic driver — POST body dropped
  wreq expects raw bytes as body=, not data= (turbo/curl_cffi uses data=).
  Added StealthRequestPayload.basic_http_kwargs() for the basic engine.

• Browser CDP headers on POST setup
  Do not send Content-Type / Content-Length via CDP extra headers during
  origin setup; they are set only on the in-page fetch() call.

Changed

• Browser POST context verifies same-origin after navigation and checks for Chrome
  error pages before running fetch().

Changed

• PyPI wheel/sdist packaging
  Ship only scrapy_stealth and docs/static/logo.png (browser splash). Exclude
  examples/, scripts/, sponsor assets, and other docs from installs.

• STEALTH_ENABLED uses smart driver selection by default
  When global stealth is on, the middleware injects meta["stealth"]["driver"] = "auto"
  on requests that do not already specify a driver. HTTP impersonation (turbo by default,
  or STEALTH_DRIVER) runs first; JS challenges and session bans retry once with the
  browser driver.

• STEALTH_DRIVER default is now "turbo"
driver="auto" and global stealth now start with the turbo driver for stronger TLS
  impersonation. Set STEALTH_DRIVER = "basic" for the lighter HTTP driver.

Removed

• STEALTH_AUTO_FALLBACK setting
  Browser fallback is controlled solely by driver="auto" (injected automatically when
  STEALTH_ENABLED = True, or set per-request). Use meta["stealth"]["fallback"] = False
  to opt out for a single URL.

Fixed

• Browser splash logo showed a blank tab on startup
  _splash_url() loads docs/static/logo.png (included in PyPI wheels for splash).

Changed

Changed

• STEALTH_ENABLED uses smart driver selection by default
  When global stealth is on, the middleware injects meta["stealth"]["driver"] = "auto"
  on requests that do not already specify a driver. HTTP impersonation (turbo by default,
  or STEALTH_DRIVER) runs first; JS challenges and session bans retry once with the
  browser driver.

• STEALTH_DRIVER default is now "turbo"
driver="auto" and global stealth now start with the turbo driver for stronger TLS
  impersonation. Set STEALTH_DRIVER = "basic" for the lighter HTTP driver.

Removed

• STEALTH_AUTO_FALLBACK setting
  Browser fallback is controlled solely by driver="auto" (injected automatically when
  STEALTH_ENABLED = True, or set per-request). Use meta["stealth"]["fallback"] = False
  to opt out for a single URL.

Fixed

• Browser splash logo showed a blank tab on startup
  _splash_url() loads docs/static/logo.png only (PyPI via MANIFEST.in graft docs).

• NodeMaven materials
  Updated README and AGENTS.md with new copy, tracking links (Fawadss1readmegh, Fawadss1agentmdgh, Fawadss1tools), and the new horizontal banner (docs/static/sponsors/nodemaven-banner.png).

• Utils package layout
  Reorganised scrapy_stealth.utils into subpackages: core, detection,
  network, browser, engine, and telemetry. Import paths updated
  (e.g. scrapy_stealth.utils.core.meta, scrapy_stealth.utils.network.proxy).

Changed

• Middleware — drop deprecated spider arg from process_request
  Matches current Scrapy downloader middleware API: the spider is read from the
  crawler saved in from_crawler() (crawler.spider) instead of a method
  argument. Removes the ScrapyDeprecationWarning about
  StealthDownloaderMiddleware.process_request().

Added

• Smart browser selection (STEALTH_AUTO_FALLBACK, driver="auto")
  When basic or turbo returns a JS challenge or session ban, the middleware
  retries once with the browser driver. The fallback always runs with
  headless=False for better evasion. Opt in globally with
  STEALTH_AUTO_FALLBACK = True, per-request with
  meta["stealth"]["driver"] = "auto", or opt out with
  meta["stealth"]["fallback"] = False. Fallback counters appear under
  stealth/fallbacks in crawler.stats.