facert / awesome-spider

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

crawler scraper laravel database spider magnet-link guzzlehttp magnet adult javbus javlibrary avmoo adult-video

Updated Jul 30, 2020
PHP

s0md3v / Photon

Star

Incredibly fast crawler designed for OSINT.

python crawler osint spider information-gathering

Updated Jul 22, 2020
Python

crawlab-team / crawlab

Star

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

go docker platform crawler spider web-crawler scrapy webcrawler scrapyd-ui webspider crawling-tasks crawlab spiders-management

Updated Jul 31, 2020
Go

henrylee2cn / pholcus

Star

Pholcus is a distributed high-concurrency crawler software written in pure golang

spider crowler

Updated Feb 15, 2020
Go

luyishisi / Anti-Anti-Spider

Star

越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）（因工作原因，项目暂停）

python spider geek

Updated Jun 29, 2020
Python

bda-research / node-crawler

Star

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

nodejs javascript jquery crawler spider cheerio extract-data

Updated Jun 24, 2020
JavaScript

SpiderClub / haipproxy

Star

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

redis crawler spider scheduler distributed scrapy high-availability ipproxy

Updated Nov 1, 2019
Python

BruceDone / awesome-crawler

Star

A collection of awesome web crawler,spider in different languages

crawler scraper awesome spider web-crawler web-scraper node-crawler

Updated Aug 5, 2020

tophubs / TopList

Star

今日热榜，一个获取各大热门网站热门头条的聚合网站，使用Go语言编写，多协程异步快速抓取信息，预览:https://mo.fish

golang spider hot today-s-hot-list hotlist

Updated May 6, 2020
Go

gaojiuli / toapi

Star

Every web site provides APIs.

python html api flask json crawler web spider toapi

Updated Feb 29, 2020
Python

DormyMo / SpiderKeeper

Star

admin ui for scrapy/open source scrapinghub

spider dashboard scrapy scrapyd scrapy-ui scrapyd-ui scrapyd-dashboard

Updated Nov 4, 2019
Python

shiyanhui / dht

Star

BitTorrent DHT Protocol && DHT Spider.

go spider dht bittorrent-dht-protocol

Updated Apr 26, 2020
Go

JAVClub / core

Star

🔞 JAVClub - 让你的大姐姐不再走丢

spider japanese google-drive jav video-streaming magnet adult porn adult-content javbus javiewer

Updated Aug 4, 2020
JavaScript

Gerapy / Gerapy

Star

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

docker vuejs django spider dashboard vue distributed scrapy scrapyd webspider gerapy

Updated Aug 8, 2020
Python

jae-jae / QueryList

Star

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

crawler scraper spider querylist

Updated Jul 18, 2020
PHP

gaojiuli / gain

Star

Web crawling framework based on asyncio.

python crawler spider aiohttp asyncio uvloop

Updated Jun 1, 2019
Python

howie6879 / owllook

Star

owllook-在线网络小说阅读网站&小说搜索引擎&小说推荐系统[搜索、追书、收藏、追更、小说API]

spider schedule book sanic aiohttp reader asyncio novels uvloop novels-search owllook

Updated May 3, 2020
Python

my8100 / scrapydweb

Star

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO 👉

log-analysis spider dashboard scrapy scrapyd scrapyd-ui log-parsing scrapyd-api scrapyd-admin scrapyd-manage scrapyd-monitor scrapyd-keeper scrapyd-control scrapy-log-analysis scrapyd-log-analysis scrapy-visualization scrapyd-visualization scrapyd-cluster-management

Updated Aug 2, 2020
Python

jumper2014 / lianjia-beike-spider

Star

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 🚁，点星支持

crawler spider lianjia

Updated Jul 24, 2019
Python

sjdirect / abot

Star

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

c-sharp unit-testing crawler spider csharp parsing cross-platform web-crawler netcore log4net takes-care flexibility pluggable spiders csharp-library abot netcore2 netstandard20 netcore3 javascript-renderer netstandard21 abot-nuget icrawldecisionmaker netsta

Updated Jun 13, 2020
C#

hu17889 / go_spider

Star

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

go crawler spider schedule pipeline

Updated Nov 16, 2017
Go

xianhu / PSpider

Star

简单易用的Python爬虫框架，QQ交流群：597510560

python crawler multi-threading spider multiprocessing web-crawler proxies python-spider web-spider

Updated Mar 3, 2020
Python

wkunzhi / Python3-Spider

Star

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

python crawler spider selenium crawl scrapy splash geek taobao scrapy-crawler meituan dianping pyppeteer

Updated Jul 24, 2020
Python

JayBizzle / Crawler-Detect

Star

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

php crawler user-agent spider bots detect

Updated Jul 13, 2020
PHP

howie6879 / ruia

Star

Async Python 3.6+ web scraping micro-framework based on asyncio（Python3.6+异步爬虫框架）

crawler spider aiohttp asyncio uvloop crawling-framework asyncio-spider ruia

Updated Jun 7, 2020
Python

geziyor / geziyor

Star

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

go crawler scraper spider scraping

Updated Dec 12, 2019
Go

spider

Here are 2,110 public repositories matching this topic...

facert / awesome-spider

gocolly / colly

jhao104 / proxy_pool

shengqiangzhang / examples-of-web-crawlers

guyueyingmu / avbook

s0md3v / Photon

crawlab-team / crawlab

henrylee2cn / pholcus

luyishisi / Anti-Anti-Spider

bda-research / node-crawler

SpiderClub / haipproxy

BruceDone / awesome-crawler

tophubs / TopList

gaojiuli / toapi

DormyMo / SpiderKeeper

shiyanhui / dht

JAVClub / core

Gerapy / Gerapy

jae-jae / QueryList

gaojiuli / gain

howie6879 / owllook

my8100 / scrapydweb

jumper2014 / lianjia-beike-spider

sjdirect / abot

hu17889 / go_spider

xianhu / PSpider

wkunzhi / Python3-Spider

JayBizzle / Crawler-Detect

howie6879 / ruia

geziyor / geziyor

Improve this page

Add this topic to your repo