Settings

This Scrapy downloader middleware adds some settings to configure how to work with your Zyte proxy service.

ZYTE_SMARTPROXY_APIKEY

Default: None

Default API key for your Zyte proxy service.

Note that Zyte API and Zyte Smart Proxy Manager have different API keys.

You can override this value on specific requests.

ZYTE_SMARTPROXY_URL

Default: 'http://proxy.zyte.com:8011'

Default endpoint for your Zyte proxy service.

For guidelines on setting a value, see the initial configuration instructions.

You can override this value on specific requests.

ZYTE_SMARTPROXY_MAXBANS

Default: 400

Number of consecutive bans necessary to stop the spider.

ZYTE_SMARTPROXY_DOWNLOAD_TIMEOUT

Default: 190

Timeout for processing proxied requests. It overrides Scrapy’s DOWNLOAD_TIMEOUT.

ZYTE_SMARTPROXY_PRESERVE_DELAY

Default: False

If False sets Scrapy’s DOWNLOAD_DELAY to 0, making the spider to crawl faster. If set to True, it will respect the provided DOWNLOAD_DELAY from Scrapy.

ZYTE_SMARTPROXY_DEFAULT_HEADERS

Default: {}

Default headers added only to proxied requests. Headers defined on DEFAULT_REQUEST_HEADERS will take precedence as long as the ZyteSmartProxyMiddleware is placed after the DefaultHeadersMiddleware. Headers set on the requests have precedence over the two settings.

This is the default behavior, DefaultHeadersMiddleware default priority is 400 and we recommend ZyteSmartProxyMiddleware priority to be 610.

ZYTE_SMARTPROXY_BACKOFF_STEP

Default: 15

Step size used for calculating exponential backoff according to the formula: random.uniform(0, min(max, step * 2 ** attempt)).

ZYTE_SMARTPROXY_BACKOFF_MAX

Default: 180

Max value for exponential backoff as showed in the formula above.

ZYTE_SMARTPROXY_FORCE_ENABLE_ON_HTTP_CODES

Default: []

List of HTTP response status codes that warrant enabling your Zyte proxy service for the corresponding domain.

When a response with one of these HTTP status codes is received after an unproxied request, the request is retried with your Zyte proxy service, and any new request to the same domain is also proxied.

ZYTE_SMARTPROXY_KEEP_HEADERS

Default: False

If True, header dropping and translation is disabled.