This Scrapy downloader middleware adds some settings to configure how to work with your Zyte proxy service.
Default API key for your Zyte proxy service.
Note that Zyte API and Zyte Smart Proxy Manager have different API keys.
Default endpoint for your Zyte proxy service.
For guidelines on setting a value, see the initial configuration instructions.
Number of consecutive bans necessary to stop the spider.
Timeout for processing proxied requests. It overrides Scrapy’s
False sets Scrapy’s
0, making the spider to crawl faster. If set to
True, it will
respect the provided
DOWNLOAD_DELAY from Scrapy.
Default headers added only to proxied requests. Headers defined on
DEFAULT_REQUEST_HEADERS will take precedence as long as the
ZyteSmartProxyMiddleware is placed after the
DefaultHeadersMiddleware. Headers set on the requests have precedence over the two settings.
This is the default behavior,
DefaultHeadersMiddlewaredefault priority is
400and we recommend
ZyteSmartProxyMiddlewarepriority to be
Step size used for calculating exponential backoff according to the formula:
random.uniform(0, min(max, step * 2 ** attempt)).
Max value for exponential backoff as showed in the formula above.
List of HTTP response status codes that warrant enabling your Zyte proxy service for the corresponding domain.
When a response with one of these HTTP status codes is received after an unproxied request, the request is retried with your Zyte proxy service, and any new request to the same domain is also proxied.