Scrapy project

Repositories

• scrapy

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  python crawler framework scraping crawling hacktoberfest
  Python 9,235 40,973 497 (33 issues need help) 319 Updated Jul 4, 2021

• parsel

  Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

  css python xml scraping selectors xpath lxml
  Python 105 666 27 (1 issue needs help) 14 Updated Jul 3, 2021

• xtractmime

  https://mimesniff.spec.whatwg.org/ implementation for Python
  Python 0 3 1 2 Updated Jul 2, 2021

• w3lib

  Python library of web-related functions

  python
  Python 87 333 15 (3 issues need help) 11 Updated Jul 2, 2021

• cssselect

  CSS Selectors for Python

  css python selectors
  Python 51 239 15 9 Updated Jun 30, 2021

• itemadapter

  Common interface for data container classes

  python metadata python3 scrapy hacktoberfest python-dataclasses python-attrs
  Python BSD-3-Clause 5 21 1 3 Updated Jun 25, 2021

• scrapy.org

  The scrapy.org website

  html
  HTML 144 43 2 5 Updated Jun 21, 2021

• queuelib

  Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python

  python python3 persistent queues non-persistent
  Python 49 215 2 2 Updated Jun 20, 2021

• itemloaders

  Library to populate items using XPath and CSS with a convenient API
  Python BSD-3-Clause 10 26 14 3 Updated Jun 14, 2021

• scrapyd-client

  Command line client for Scrapyd server
  Python 128 604 25 13 Updated Jun 10, 2021

• scrapyd

  A service daemon to run Scrapy spiders
  Python 525 2,312 99 25 Updated Apr 20, 2021

• scrapy-bench

  A CLI for benchmarking Scrapy.

  python web-crawler scrapy command-line-tool benchmark-suite scrapy-bench
  Python MIT 15 26 6 2 Updated Feb 24, 2021

• loginform

  Fill HTML login forms automatically
  Python 71 241 10 2 Updated Feb 2, 2021

• protego

  A pure-Python robots.txt parser with support for modern conventions.

  python robots-txt robots-parser
  DIGITAL Command Language BSD-3-Clause 17 21 0 0 Updated Feb 1, 2021

• booksbot

  Forked from stummjr/books_crawler

  A crawler for http://books.toscrape.com
  Python 771 32 0 2 Updated Aug 8, 2020

• quotesbot

  This is a sample Scrapy project for educational purposes
  Python MIT 685 1,042 1 6 Updated Jul 13, 2020

• scrapy-itemloader Archived

  [Archived] Library to populate Scrapy items using XPath and CSS with a convenient API
  Python BSD-3-Clause 7 5 2 0 Updated May 5, 2020

• scrapely

  A pure-python HTML screen-scraping library
  HTML 255 1,756 26 5 Updated Nov 28, 2019

• scurl

  Performance-focused replacement for Python urllib

  python cython chromium gurl urlparse
  Python Apache-2.0 6 16 10 (1 issue needs help) 1 Updated Oct 2, 2018

• url-chromium

  url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url

  chromium gurl
  C++ 2 0 0 0 Updated Aug 7, 2018

• base-chromium

  base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/
  C++ 3 1 0 0 Updated Jul 31, 2018

• dirbot

  Scrapy project to scrape public web directories (educational) [DEPRECATED]
  Python 1,121 1,608 0 0 Updated Oct 27, 2017

• scrapy-bench-speedcenter

  Forked from Parth-Vader/scrapy-bench-speedcenter

  Codespeed for scrapy-bench
  Python 2 2 0 0 Updated Aug 28, 2017

• pypydispatcher

  A fork of http://pydispatcher.sourceforge.net/ with PyPy support
  Python 3 12 1 0 Updated Jul 3, 2017

• slybot
  60 222 5 0 Updated Apr 27, 2015

• gsoc2014-integration-tests

  GSoC2014 - Scrapy Integration tests project
  Shell 3 3 0 0 Updated Mar 18, 2014