Scrapy, a fast high-level web crawling & scraping framework for Python.
#
crawler
Repositories 2,732
A Powerful Spider(Web Crawler) System in Python.
Python
Updated Oct 17, 2018
A scalable web crawler framework for Java.
Java
Updated Sep 30, 2018
News, full-text, and article metadata extraction in Python 3. Advanced docs:
Python
Updated Nov 2, 2018
Elegant Scraper and Crawler Framework for Golang
[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.
crawler
spider
multi-interface
golang
distributed-crawler
high-concurrency-crawler
fastest-crawler
cross-platform-crawler
Go
Updated Jul 12, 2018
Python爬虫代理IP池(proxy pool)
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
JavaScript
Updated Aug 30, 2018
Incredibly fast crawler designed for OSINT.
Python
Updated Oct 24, 2018
Redis-based components for Scrapy.
Python
Updated May 5, 2018
Distributed crawler powered by Headless Chrome
JavaScript
Updated Nov 5, 2018
Declarative web scraping
A collection of awesome web crawler,spider in different languages
Updated Oct 9, 2018
Python
Updated Sep 13, 2018
Every web site provides APIs.
Python
Updated Nov 4, 2018
基于搜狗微信搜索的微信公众号爬虫接口
Python
Updated Oct 30, 2018
Web Application Security Scanner Framework
Intelligent proxy pool for Humans™
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, Baidu and others) by using p…
HTML
Updated Oct 30, 2018
Web crawling framework based on asyncio.
Python
Updated Mar 19, 2018
Polite, slim and concurrent web crawler.
Go
Updated Apr 29, 2018
Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机
Python
Updated Oct 31, 2018
The DomCrawler component eases DOM navigation for HTML and XML documents.
PHP
Updated Nov 3, 2018
DotnetSpider, a .NET Standard web crawling library similar to WebMagic and Scrapy. It is a lightweight ,efficient and…
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be ex…
Go
Updated Nov 16, 2017
Easy to use lightweight web crawler（易用的轻量化网络爬虫）
Java
Updated Sep 21, 2018
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
Ruby
Updated Aug 2, 2018
Crawl a website and run it through Google lighthouse
JavaScript
Updated Feb 22, 2018
简单易用的Python爬虫框架，QQ交流群：597510560
Python
Updated Oct 24, 2018