Scrapy project

An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

scrapy Public
Scrapy, a fast high-level web crawling & scraping framework for Python.

Python 44,638 BSD-3-Clause 9,719 519 (30 issues need help) 281 Updated Sep 27, 2022
scrapy.org Public
The scrapy.org website

HTML 48 153 0 0 Updated Sep 27, 2022
scrapyd Public
A service daemon to run Scrapy spiders

Python 2,524 BSD-3-Clause 546 30 19 Updated Sep 24, 2022
w3lib Public
Python library of web-related functions

Python 356 BSD-3-Clause 100 15 (1 issue needs help) 7 Updated Sep 23, 2022
booksbot Public
A crawler for http://books.toscrape.com

Python 34 835 0 2 Updated Sep 13, 2022
scrapyd-client Public
Command line client for Scrapyd server

Python 669 BSD-3-Clause 137 10 1 Updated Sep 7, 2022
itemloaders Public
Library to populate items using XPath and CSS with a convenient API

Python 33 BSD-3-Clause 12 17 4 Updated Aug 29, 2022
parsel Public
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

Python 820 BSD-3-Clause 131 30 17 Updated Aug 26, 2022
itemadapter Public
Common interface for data container classes

Python 42 BSD-3-Clause 7 4 2 Updated Aug 2, 2022
protego Public
A pure-Python robots.txt parser with support for modern conventions.

DIGITAL Command Language 32 BSD-3-Clause 20 3 (1 issue needs help) 1 Updated Jul 7, 2022

Pinned