Redis-based components for Scrapy.

• Usage: https://github.com/rmax/scrapy-redis/wiki/Usage
• Documentation: https://github.com/rmax/scrapy-redis/wiki.
• Release: https://github.com/rmax/scrapy-redis/wiki/History
• Contribution: https://github.com/rmax/scrapy-redis/wiki/Getting-Started
• LICENSE: MIT license

• Distributed crawling/scraping

  You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls.

• Distributed post-processing

  Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue.

• Scrapy plug-and-play components

  Scheduler + Duplication Filter, Item Pipeline, Base Spiders.

• In this forked version: added json supported data in Redis

  data contains url, `meta` and other optional parameters. meta is a nested json which contains sub-data. this function extract this data and send another FormRequest with url, meta and addition formdata.

  For example:

  { "url": "https://exaple.com", "meta": {"job-id":"123xsd", "start-date":"dd/mm/yy"}, "url_cookie_key":"fertxsas" }

  this data can be accessed in scrapy spider through response. like: request.url, request.meta, request.cookies

• Python 3.7+
• Redis >= 5.0
• Scrapy >= 2.0
• redis-py >= 4.2

From pip

pip install scrapy-redis

From GitHub

git clone https://github.com/darkrho/scrapy-redis.git
cd scrapy-redis
python setup.py install

pip uninstall scrapy-redis

Idle Redis queue polling can be reduced with opt-in, deadline-gated exponential backoff:

• REDIS_IDLE_BACKOFF_ENABLED (default False) enables idle-poll backoff.
• REDIS_IDLE_BACKOFF_MIN (default 1.0) is the initial delay in seconds.
• REDIS_IDLE_BACKOFF_MAX (default 30.0) caps the delay in seconds.
• REDIS_IDLE_BACKOFF_FACTOR (default 2.0) multiplies the delay after each empty poll.

When enabled, the spider never blocks the reactor: while waiting for the next poll deadline it skips all Redis calls. The close deadline is still checked on every idle callback. The tradeoff is that new work may take up to the backoff cap to be picked up, while idle Redis load is reduced when many workers share a queue.

Components of one crawler share a single Redis connection pool when using the default client class and plain connection parameters. Pools are scoped per Settings object (crawler). Configurations with custom client classes or client-only parameters (ssl, unix_socket_path, single_connection_client) keep the previous per-component behavior.

• REDIS_MAX_CONNECTIONS: Maximum connections in the shared pool (default: no override is applied — redis-py's own pool default, effectively unlimited on redis-py 4.x/5.x). It raises ConnectionError when exhausted; supply a BlockingConnectionPool via REDIS_PARAMS.connection_pool (REDIS_PARAMS["connection_pool"]) for waiting semantics.

redis-py 8 changed its default connection protocol to RESP3. Selecting a protocol changes connection setup only; it does not remove timeout risk from warm operations on already-established connections. The REDIS_PROTOCOL setting defaults to None, leaving redis-py's installed-version default unchanged.

On redis-py >= 5, the recommended, discoverable form for opting into RESP2 is:

REDIS_PROTOCOL = 2

The existing equivalent form remains supported:

REDIS_PARAMS = {"protocol": 2}

Both forms require redis-py >= 5. REDIS_PROTOCOL takes precedence when both settings specify a protocol. As optional advanced tuning on redis-py >= 5, you can also omit redis-py's connection identification metadata:

REDIS_PARAMS = {"driver_info": None}

This can reduce connection-setup work, but loses useful client-identification metadata in exchange.

The library's historical retry_on_timeout=True default is deprecated and has no effect on redis-py >= 6. On those versions, the effective retry policy comes from redis-py's retry object and its defaults. Measure the resulting behavior before lowering retries, especially under connection churn or saturation.

In testing with Redis 8.0.2, CLIENT MAINT_NOTIFICATIONS was rejected during RESP3 connection setup. This observation is scoped to the tested Redis 8.0.2 version. The rejection is not shown by MONITOR or commandstats; inspect client logs and Redis's total_error_replies counter for diagnostics.

The connection-related behavior by redis-py version is summarized below:

Frontera is a web crawling framework consisting of crawl frontier, and distribution/scaling primitives, allowing to build a large scale online web crawler.