把起始URL放到Redis中去
把起始URL放到redis中去
from scrapy_redis.spiders import RedisSpider
# 继承RedisSpider
class ChoutiSpider(RedisSpider):
name = ‘chouti‘
allowed_domains = [‘chouti.com‘]
配置文件:
REDIS_START_URLS_KEY = ‘%(name)s:start_urls‘
REDIS_START_URLS_AS_SET = False
下面是源码
# use_set = self.settings.getbool(‘REDIS_START_URLS_AS_SET‘, defaults.START_URLS_AS_SET)
# fetch_one = self.server.spop if use_set else self.server.lpop 相关推荐
王道革 2020-11-25
wangdonghello 2020-11-03
chenhualong0 2020-11-16
聚合室 2020-11-16
koushr 2020-11-12
guoyanga 2020-11-10
fackyou00 2020-11-10
Orangesss 2020-11-03
dongCSDN 2020-10-31
Quietboy 2020-10-30
liuyulong 2020-10-29
fansili 2020-10-29
温攀峰 2020-10-23
jackbon 2020-10-19
kaixinfelix 2020-10-04