使用scrapy抓取Youtube播放页数据

可参看Knowsmore

抓取Youtube播放页数据的前提是scrapy部署的机器可以正常访问Youtube网站

示例网址

抓取的原理是读取Youtube播放页桌面版网页源代码中的全局变量： ytInitialData

存取到Mongo中的数据如下：

{
    "title" : "20130410 锵锵三人行 陈平原谈中国教育问题",
    "view_count" : "12,407 views"
}

代码如下：

# -*- coding: utf-8 -*-
import scrapy
import re
import json
from scrapy import Selector
from knowsmore.items import YoutubeItem
from ..common import *

class YoutubeSpider(scrapy.Spider):
    custom_settings = {
        'DOWNLOADER_MIDDLEWARES' : {
        }
    }

    name = 'youtube'
    allowed_domains = ['www.youtube.com']
    start_urls = ['https://www.youtube.com/watch?v=3vkqOdMBP48']

    def parse(self, response):
        ytInitialData = r1(r'ytInitialData"\] = (.*?)}};', response.body)

        if ytInitialData:
            ytInitialData = '%s}}' % ytInitialData
            ytInitialDataObj = json.loads(ytInitialData)
            
            videoInfo = ytInitialDataObj['contents']['twoColumnWatchNextResults']['results']['results']['contents'][0]['videoPrimaryInfoRenderer']
            Item = YoutubeItem(
                title = videoInfo['title']['simpleText'].encode('utf-8'),
                view_count = videoInfo['viewCount']['videoViewCountRenderer']['viewCount']['simpleText']
            )
            yield Item