#

Crawler

A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).

Here are 5,925 public repositories matching this topic...

scrapy

scrapy / scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

python crawler framework scraping crawling hacktoberfest

Updated Nov 26, 2022
Python

iawia002 / lux

👾 Fast and simple video download library and CLI tool written in Go

go golang crawler scraper downloader youtube video download tumblr bilibili qq youku iqiyi

Updated Nov 17, 2022
Go

gocolly / colly

Elegant Scraper and Crawler Framework for Golang

go golang crawler scraper framework spider scraping crawling

Updated Nov 17, 2022
Go

jhao104 / proxy_pool

Python爬虫代理IP池(proxy pool)

redis flask crawler spider proxy crawl proxypool ssdb

Updated Aug 16, 2022
Python

binux / pyspider

A Powerful Spider(Web Crawler) System in Python.

Updated Nov 20, 2022
Python

codelucas / newspaper

News, full-text, and article metadata extraction in Python 3. Advanced docs:

python crawler scraper news crawling news-aggregator

Updated Nov 15, 2022
Python

shengqiangzhang / examples-of-web-crawlers

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

python crawler spider example selenium multithreading stock wechat taobao pyquery tmall fund agent-pool wechat-report wereader

Updated Apr 27, 2022
Python

code4craft / webmagic

A scalable web crawler framework for Java.

java crawler framework scraping

Updated Nov 24, 2022
Java

crawlab

crawlab-team / crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

go docker platform crawler spider web-crawler scrapy webcrawler scrapyd-ui webspider crawling-tasks crawlab spiders-management

Updated Nov 24, 2022
Go

s0md3v / Photon

Sponsor

Incredibly fast crawler designed for OSINT.

python crawler osint spider information-gathering

Updated Jun 14, 2022
Python

guyueyingmu / avbook

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

crawler scraper laravel database spider magnet-link guzzlehttp magnet adult javbus javlibrary avmoo adult-video

Updated Nov 13, 2022
PHP

injetlee / Python

Python脚本。模拟登录知乎，爬虫，操作excel，微信公众号，远程开机

python crawler excel wechat

Updated Nov 22, 2022
Python

ssssssss-team / spider-flow

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

crawler spider web-crawler jsoup xpath webcrawler webspider web-spider spider-flow

Updated Nov 20, 2022
Java

crawlee

apify / crawlee

Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.

npm crawler scraper automation web-crawler headless scraping crawling javascript-library web-scraping web-crawling headless-chrome rpa apify puppeteer

Updated Nov 24, 2022
TypeScript

bda-research / node-crawler

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

nodejs javascript jquery crawler spider cheerio extract-data

Updated Nov 21, 2022
JavaScript

lorien / awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.

crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python

Updated Nov 14, 2022
Makefile

chyroc / WechatSogou

Sponsor

基于搜狗微信搜索的微信公众号爬虫接口

python crawler pypi scrapy wechat sogou

Updated Nov 22, 2022
Python

yujiosaka / headless-chrome-crawler

Distributed crawler powered by Headless Chrome

jquery crawler chrome scraper promise scraping crawling chromium headless-chrome puppeteer

Updated Nov 11, 2022
JavaScript

BruceDone / awesome-crawler

A collection of awesome web crawler,spider in different languages

crawler scraper awesome spider web-crawler web-scraper node-crawler

Updated Nov 4, 2022

SpiderClub / haipproxy

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

redis crawler spider scheduler distributed scrapy high-availability ipproxy

Updated Nov 4, 2022
Python

Wikipedia: Wikipedia