Scrapy is a fast high-level web crawling and scraping framework written in Python used to crawl websites and extract structured data from their pages. It can be used for many purposes, from data mining to monitoring and automated testing.
This article is divided into 2 sections:
- Creating a Simple web crawler to scrape the details from a Web Scraping Sandbox website (http://books.toscrape.com/)
- Exploring how Scrapy Feed exports can be used to store the scraped data to export files in various formats.
Creating a Simple web crawler
We are going to create a web crawler to scrape all the book details(URL, Title, Price) from a Web Scraping Sandbox website
1. Installation of packages – run the following command from the terminal
pip install scrapy
2. Create a Scrapy project – run the following command from the terminal
scrapy startproject booklist
cd booklist
scrapy genspider book http://books.toscrape.com/
Here,
- Project Name: "booklist"
- Spider Name: "book"
- Domain to be Scraped: "http://books.toscrape.com/"
Directory Structure:
Directory Structure
3. Create an Item - replace the contents of "booklist\items.py" file with the below code
We define each Item scraped from the website as an object with the following 3 fields:
Python
# booklist\items.py
# Define here the models for your scraped items
from scrapy.item import Item, Field
class BooklistItem(Item):
url = Field()
title = Field()
price = Field()
4. Define the Parse function - Add the following code to "booklist\spiders\book.py"
The response from the crawler is parsed to extract the book details (i.e. URL, Title, Price) as shown in the below code
Python
# booklist\spiders\book.py
import scrapy
from booklist.items import BooklistItem
class BookSpider(scrapy.Spider):
name = 'book'
allowed_domains = ['books.toscrape.com']
start_urls = ['http://books.toscrape.com/']
def parse(self, response):
for article in response.css('article.product_pod'):
book_item = BooklistItem(
url=article.css("h3 > a::attr(href)").get(),
title=article.css("h3 > a::attr(title)").extract_first(),
price=article.css(".price_color::text").extract_first(),
)
yield book_item
5. Run the spider using following command:
scrapy crawl book
Output:
Output ItemsScrapy Feed Exports
One of the most frequently required features when implementing scrapers is being able to store the scraped data as an "export file".
Scrapy provides this functionality out of the box with the Feed Exports, which allows to generate feeds with the scraped items, using multiple serialization formats and storage backends.
The different file formats supported are:
1. Saving Files via the Command Line
The simplest way to export the file of the data scraped, is to define a output path when starting the spider in the command line.
Add the flag -o to the scrapy crawl command along with the file path you want to save the file to
- CSV - "data/book_data.csv"
- JSON - "data/book_data.json"
- JSON Lines - "data/book_data.jsonl"
- XML - "data/book_data.xml"
scrapy crawl book -o data/book_data.csv
scrapy crawl book -o data/book_data.json
scrapy crawl book -o data/book_data.jsonl
scrapy crawl book -o data/book_data.xml
There are 2 options to using this command:
Flag
| Description
|
-o
| Appends new data to an existing file. |
-O
| Overwrites any existing file with the current data. |
2. Saving Files using Feed Exports
For serializing the scraped data, the Feed Exports internally use the Item Exporters.
Saving the Data via FEEDS setting:
The scraped data can stored by defining the FEEDS setting in the "booklist\settings.py" by passing it a dictionary with the path/name of the file and the file format
Python
# booklist\settings.py
# To store in CSV format
FEEDS = {
'data/book_data.csv': {'format': 'csv', 'overwrite': True}
}
# booklist\settings.py
# To store in JSON format
FEEDS = {
'data/book_data.json': {'format': 'json', 'overwrite': True}
}
# booklist\settings.py
# To store in JSON Lines format
FEEDS = {
'data/book_data.jsonl': {'format': 'jsonlines', 'overwrite': True}
}
# booklist\settings.py
# To store in XML Lines format
FEEDS = {
'data/book_data.xml': {'format': 'xml', 'overwrite': True}
}
Saving the Data via custom_settings:
The scraped data can also be stored by configuring the FEEDS setting in each individual spider by setting a custom_setting in the spider ("booklist\spiders\book.py") for various file formats required.
Python
# booklist\spiders\book.py
import scrapy
from booklist.items import BooklistItem
class BookSpider(scrapy.Spider):
name = 'book'
allowed_domains = ['books.toscrape.com']
start_urls = ['http://books.toscrape.com/']
# To store in CSV format
custom_settings = {
'FEEDS': {'data.csv': {'format': 'csv', 'overwrite': True}}
}
def parse(self, response):
for article in response.css('article.product_pod'):
book_item = BooklistItem(
url=article.css("h3 > a::attr(href)").get(),
title=article.css("h3 > a::attr(title)").extract_first(),
price=article.css(".price_color::text").extract_first(),
)
yield book_item
# booklist\spiders\book.py
import scrapy
from booklist.items import BooklistItem
class BookSpider(scrapy.Spider):
name = 'book'
allowed_domains = ['books.toscrape.com']
start_urls = ['http://books.toscrape.com/']
# To store in JSON format
custom_settings = {
'FEEDS': {'data.json': {'format': 'json', 'overwrite': True}}
}
def parse(self, response):
for article in response.css('article.product_pod'):
book_item = BooklistItem(
url=article.css("h3 > a::attr(href)").get(),
title=article.css("h3 > a::attr(title)").extract_first(),
price=article.css(".price_color::text").extract_first(),
)
yield book_item
# booklist\spiders\book.py
import scrapy
from booklist.items import BooklistItem
class BookSpider(scrapy.Spider):
name = 'book'
allowed_domains = ['books.toscrape.com']
start_urls = ['http://books.toscrape.com/']
# To store in JSON Lines format
custom_settings = {
'FEEDS': {'data.jsonl': {'format': 'jsonlines', 'overwrite': True}}
}
def parse(self, response):
for article in response.css('article.product_pod'):
book_item = BooklistItem(
url=article.css("h3 > a::attr(href)").get(),
title=article.css("h3 > a::attr(title)").extract_first(),
price=article.css(".price_color::text").extract_first(),
)
yield book_item
# booklist\spiders\book.py
import scrapy
from booklist.items import BooklistItem
class BookSpider(scrapy.Spider):
name = 'book'
allowed_domains = ['books.toscrape.com']
start_urls = ['http://books.toscrape.com/']
# To store in XML format
custom_settings = {
'FEEDS': {'data.xml': {'format': 'xml', 'overwrite': True}}
}
def parse(self, response):
for article in response.css('article.product_pod'):
book_item = BooklistItem(
url=article.css("h3 > a::attr(href)").get(),
title=article.css("h3 > a::attr(title)").extract_first(),
price=article.css(".price_color::text").extract_first(),
)
yield book_item
Setting Dynamic File Paths/Names:
The generated data files can be stored using dynamic Path/Name as follows:
Below code creates a JSON file in the data folder, followed by the subfolder with the spiders name, and a file name that includes the spider name and date it was scraped.
Python
# settings.py
FEEDS = {
'data/%(name)s/%(name)s_%(time)s.json': {
'format': 'json', 'overwrite': True
}
}
The generated path would look something like this.
"data/book/book_2023-01-15T16-53-02.json"
Saved Files Output:
Saved Files Output
Similar Reads
Implementing Web Scraping in Python with Scrapy Nowadays data is everything and if someone wants to get data from webpages then one way to use an API or implement Web Scraping techniques. In Python, Web scraping can be done easily by using scraping tools like BeautifulSoup. But what if the user is concerned about performance of scraper or need to
5 min read
Getting Started With Scrapy
Scrapy Basics
Scrapy - Command Line ToolsPrerequisite: Implementing Web Scraping in Python with Scrapy Scrapy is a python library that is used for web scraping and searching the contents throughout the web. It uses Spiders which crawls throughout the page to find out the content specified in the selectors. Hence, it is a very handy tool to
5 min read
Scrapy - Item LoadersIn this article, we are going to discuss Item Loaders in Scrapy. Scrapy is used for extracting data, using spiders, that crawl through the website. The obtained data can also be processed, in the form, of Scrapy Items. The Item Loaders play a significant role, in parsing the data, before populating
15+ min read
Scrapy - Item PipelineScrapy is a web scraping library that is used to scrape, parse and collect web data. For all these functions we are having a pipelines.py file which is used to handle scraped data through various components (known as class) which are executed sequentially. In this article, we will be learning throug
10 min read
Scrapy - SelectorsScrapy Selectors as the name suggest are used to select some things. If we talk of CSS, then there are also selectors present that are used to select and apply CSS effects to HTML tags and text. In Scrapy we are using selectors to mention the part of the website which is to be scraped by our spiders
7 min read
Scrapy - ShellScrapy is a well-organized framework, used for large-scale web scraping. Using selectors, like XPath or CSS expressions, one can scrape data seamlessly. It allows systematic crawling, and scraping the data, and storing the content in different file formats. Scrapy comes equipped with a shell, that h
9 min read
Scrapy - SpidersScrapy is a free and open-source web-crawling framework which is written purely in python. Thus, scrapy can be installed and imported like any other python package. The name of the package is self-explanatory. It is derived from the word 'scraping' which literally means extracting desired substance
11 min read
Scrapy - Feed exportsScrapy is a fast high-level web crawling and scraping framework written in Python used to crawl websites and extract structured data from their pages. It can be used for many purposes, from data mining to monitoring and automated testing. This article is divided into 2 sections:Creating a Simple web
5 min read
Scrapy - Link ExtractorsIn this article, we are going to learn about Link Extractors in scrapy. "LinkExtractor" is a class provided by scrapy to extract links from the response we get while fetching a website. They are very easy to use which we'll see in the below post. Scrapy - Link Extractors Basically using the "LinkEx
5 min read
Scrapy - SettingsScrapy is an open-source tool built with Python Framework. It presents us with a strong and robust web crawling framework that can easily extract the info from the online page with the assistance of selectors supported by XPath. We can define the behavior of Scrapy components with the help of Scrapy
7 min read
Scrapy - Sending an E-mailPrerequisites: Scrapy Scrapy provides its own facility for sending e-mails which is extremely easy to use, and it’s implemented using Twisted non-blocking IO, to avoid interfering with the non-blocking IO of the crawler. This article discusses how mail can be sent using scrapy. For this MailSender
2 min read
Scrapy - ExceptionsPython-based Scrapy is a robust and adaptable web scraping platform. It provides a variety of tools for systematic, effective data extraction from websites. It helps us to automate data extraction from numerous websites. Scrapy Python Scrapy describes the spider that browses websites and gathers dat
7 min read
Data Collection and Management
Data Extraction and Export
How to Convert Scrapy item to JSON?Prerequisite:Â scrapyJSON Scrapy is a web scraping tool used to collect web data and can also be used to modify and store data in whatever form we want. Whenever data is being scraped by the spider of scrapy, we are converting that raw data to items of scrapy, and then we will pass that item for fur
8 min read
Saving scraped items to JSON and CSV file using ScrapyIn this article, we will see how to use crawling with Scrapy, and, Exporting data to JSON and CSV format. We will scrape data from a webpage, using a Scrapy spider, and export the same to two different file formats. Here we will extract from the link  http://quotes.toscrape.com/tag/friendship/. This
6 min read
How to get Scrapy Output File in XML File?Prerequisite: Implementing Web Scraping in Python with Scrapy Scrapy provides a fast and efficient method to scrape a website. Web Scraping is used to extract the data from websites. In Scrapy we create a spider and then use it to crawl a website. In this article, we are going to extract population
2 min read
Scraping a JSON response with ScrapyScrapy is a popular Python library for web scraping, which provides an easy and efficient way to extract data from websites for a variety of tasks including data mining and information processing. In addition to being a general-purpose web crawler, Scrapy may also be used to retrieve data via APIs.
2 min read
Logging in ScrapyScrapy is a fast high-level web crawling and scraping framework written in Python used to crawl websites and extract structured data from their pages. It can be used for many purposes, from data mining to monitoring and automated testing. As developers, we spend most of our time debugging than writi
3 min read
Appliaction And Projects
How to use Scrapy to parse PDF pages online?Prerequisite: Scrapy, PyPDF2, URLLIB In this article, we will be using Scrapy to parse any online PDF without downloading it onto the system. To do that we have to use the PDF parser or editor library of Python know as PyPDF2. PyPDF2 is a pdf parsing library of python, which provides various method
3 min read
How to download Files with Scrapy ?Scrapy is a fast high-level web crawling and web scraping framework used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing. In this tutorial, we will be exploring how to download files usi
8 min read
Automated Website Scraping using ScrapyScrapy is a Python framework for web scraping on a large scale. It provides with the tools we need to extract data from websites efficiently, processes it as we see fit, and store it in the structure and format we prefer. Zyte (formerly Scrapinghub), a web scraping development and services company,
5 min read
Writing Scrapy Python Output to JSON fileIn this article, we are going to see how to write scrapy output into a JSON file in Python. Using  scrapy command-line shell This is the easiest way to save data to JSON is by using the following command: scrapy crawl <spiderName> -O <fileName>.json This will generate a file with a provi
2 min read
Pagination using Scrapy - Web Scraping with PythonPagination using Scrapy. Web scraping is a technique to fetch information from websites. Scrapy is used as a Python framework for web scraping. Getting data from a normal website is easier, and can be just achieved by just pulling the HTML of the website and fetching data by filtering tags. But what
3 min read
Email Id Extractor Project from sites in Scrapy PythonScrapy is open-source web-crawling framework written in Python used for web scraping, it can also be used to extract data for general-purpose. First all sub pages links are taken from the main page and then email id are scraped from these sub pages using regular expression. This article shows the e
8 min read
Scraping Javascript Enabled Websites using Scrapy-SeleniumScrapy-selenium is a middleware that is used in web scraping. scrapy do not support scraping modern sites that uses javascript frameworks and this is the reason that this middleware is used with scrapy to scrape those modern sites.Scrapy-selenium provide the functionalities of selenium that help in
4 min read
How to use Scrapy Items?In this article, we will scrape Quotes data using scrapy items, from the webpage https://quotes.toscrape.com/tag/reading/. The main objective of scraping, is to prepare structured data, from unstructured resources. Scrapy Items are wrappers around, the dictionary data structures. Code can be written
9 min read
How To Follow Links With Python Scrapy ?In this article, we will use Scrapy, for scraping data, presenting on linked webpages, and, collecting the same. We will scrape data from the website 'https://quotes.toscrape.com/'. Creating a Scrapy Project Scrapy comes with an efficient command-line tool, also called the 'Scrapy tool'. Commands ar
9 min read
Difference between BeautifulSoup and Scrapy crawlerWeb scraping is a technique to fetch data from websites. While surfing on the web, many websites don’t allow the user to save data for personal use. One way is to manually copy-paste the data, which both tedious and time-consuming. Web Scraping is the automation of the data extraction process from w
3 min read
Python - How to create an ARP Spoofer using Scapy?ARP spoofing is a malicious attack in which the hacker sends falsified ARP in a network. Every node in a connected network has an ARP table through which we identify the IP address and the MAC address of the connected devices. What aim to send an ARP broadcast to find our desired IP which needs to b
6 min read