Scrapy project

An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

scrapy Public
Scrapy, a fast high-level web crawling & scraping framework for Python.

Python 44,886 BSD-3-Clause 9,757 512 (30 issues need help) 279 Updated Oct 18, 2022
scrapy.org Public
The scrapy.org website

HTML 49 154 1 (1 issue needs help) 0 Updated Oct 17, 2022
w3lib Public
Python library of web-related functions

Python 359 BSD-3-Clause 101 14 (1 issue needs help) 7 Updated Oct 14, 2022
protego Public
A pure-Python robots.txt parser with support for modern conventions.

DIGITAL Command Language 33 BSD-3-Clause 20 4 (2 issues need help) 1 Updated Oct 12, 2022
parsel Public
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

Python 822 BSD-3-Clause 130 31 18 Updated Oct 8, 2022
scrapyd Public
A service daemon to run Scrapy spiders

Python 2,533 BSD-3-Clause 551 30 19 Updated Sep 24, 2022
booksbot Public
A crawler for http://books.toscrape.com

Python 34 837 0 2 Updated Sep 13, 2022
scrapyd-client Public
Command line client for Scrapyd server

Python 673 BSD-3-Clause 138 10 1 Updated Sep 7, 2022
itemloaders Public
Library to populate items using XPath and CSS with a convenient API

Python 33 BSD-3-Clause 12 17 4 Updated Aug 29, 2022
itemadapter Public
Common interface for data container classes

Python 43 BSD-3-Clause 9 4 2 Updated Aug 2, 2022

Pinned