Scrapy实战：爬取豆瓣电影数据和评分热度排名

1

pip install scrapy

1

scrapy startproject doubanmovie

1

2

cd doubanmovie

scrapy genspider douban douban.com

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

import scrapy

class DoubanSpider(scrapy.Spider):

    name = 'douban'

    allowed_domains = ['douban.com']

    start_urls = ['https://movie.douban.com/top250']

    def parse(self, response):

        movie_list = response.xpath('//div[@class="item"]')

        for movie in movie_list:

            yield {

                'name': movie