Build software better, together

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

crawler scraper laravel database spider magnet-link guzzlehttp magnet adult javbus javlibrary avmoo adult-video

Updated Mar 29, 2022
PHP

henrylee2cn / pholcus

Star

Pholcus is a distributed high-concurrency crawler software written in pure golang

spider crowler

Updated Sep 19, 2021
Go

luyishisi / Anti-Anti-Spider

Star

越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）（因工作原因，项目暂停）

python spider geek

Updated Oct 17, 2021
Python

ssssssss-team / spider-flow

Star

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

crawler spider web-crawler jsoup xpath webcrawler webspider web-spider spider-flow

Updated Mar 3, 2022
Java

bda-research / node-crawler

Star

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

nodejs javascript jquery crawler spider cheerio extract-data

Updated Mar 28, 2022
JavaScript

INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。

chrome automation spider selenium python3 crawl tkinter outlook wxpython csdn hotmail

Updated Mar 12, 2022
Python

SpiderClub / haipproxy

Star

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

redis crawler spider scheduler distributed scrapy high-availability ipproxy

Updated Mar 2, 2022
Python

BruceDone / awesome-crawler

Star

A collection of awesome web crawler,spider in different languages

crawler scraper awesome spider web-crawler web-scraper node-crawler

Updated Nov 3, 2021

tophubs / TopList

Star

今日热榜，一个获取各大热门网站热门头条的聚合网站，使用Go语言编写，多协程异步快速抓取信息，预览:https://mo.fish

golang spider hot today-s-hot-list hotlist

Updated Jan 22, 2022
Go

gaojiuli / toapi

Star

Every web site provides APIs.

python html api flask json crawler web spider toapi

Updated Jan 21, 2022
Python

niespodd / browser-fingerprinting

Star

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

bot crawler scraper automation recaptcha web spider detection chromium chromedriver fingerprinting browser-fingerprinting stealth webscraping bot-detection chromium-browser puppeteer

Updated Dec 4, 2021
JavaScript

201206030 / fiction_house

Star

小说精品屋是一个多平台（web、安卓app、微信小程序）、功能完善的屏幕自适应小说漫画连载系统，包含精品小说专区、轻小说专区和漫画专区。包括小说/漫画分类、小说/漫画搜索、小说/漫画排行、完本小说/漫画、小说/漫画评分、小说/漫画在线阅读、小说/漫画书架、小说/漫画阅读记录、小说下载、小说弹幕、小说/漫画自动采集/更新/纠错、小说内容自动分享到微博、邮件自动推广、链接自动推送到百度搜索引擎等功能。

cms spider book download read novel cartoon soft-novel

Updated Nov 11, 2021
Java

JAVClub / core

Star

🔞 JAVClub - 让你的大姐姐不再走丢

spider japanese google-drive jav video-streaming magnet adult porn adult-content javbus javiewer

Updated Jun 14, 2021
JavaScript

Gerapy / Gerapy

Star

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

docker vuejs django spider dashboard vue distributed scrapy scrapyd webspider gerapy

Updated Apr 1, 2022
Python

DormyMo / SpiderKeeper

Star

admin ui for scrapy/open source scrapinghub

spider dashboard scrapy scrapyd scrapy-ui scrapyd-ui scrapyd-dashboard

Updated Mar 19, 2021
Python

shiyanhui / dht

Star

BitTorrent DHT Protocol && DHT Spider.

go spider dht bittorrent-dht-protocol

Updated Aug 17, 2021
Go

my8100 / scrapydweb

Star

Open

User Guide | Q&A | 用户指南 | 问答

78

LWsmile commented Nov 27, 2018

linux：HTTPConnectionPool(host='192.168.0.24', port=6801): Max retries exceeded with url: /listprojects.json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f0a78b2d828>: Failed to establish a new connection: [Errno 111] Connection refused',))
windows：HTTPConnectionPool(host='localhost', port=6801): Max retries exceeded with url: /jobs (Caused by Ne