#

Crawler

A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).

Here are 397 public repositories matching this topic...

code4craft / webmagic

A scalable web crawler framework for Java.

java crawler framework scraping

Updated Feb 10, 2023
Java

ssssssss-team / spider-flow

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

crawler spider web-crawler jsoup xpath webcrawler webspider web-spider spider-flow

Updated Nov 20, 2022
Java

xtuhcy / gecco

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

java crawler dynamic jsoup gecco fastjson

Updated Oct 23, 2022
Java

CatVodTVOfficial / CatVodTVSpider

player crawler spider tv catvod maotv

Updated Jun 10, 2022
Java

dadoonet / fscrawler

Elasticsearch File System Crawler (FS Crawler)

java elasticsearch crawler tika

Updated Mar 15, 2023
Java

wycm / zhihu-crawler

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

java crawler spider zhihu

Updated Apr 2, 2019
Java

TeamNewPipe / NewPipeExtractor

NewPipe's core library for extracting data from streaming sites

crawler scraper youtube extractor soundcloud bandcamp newpipe peertube mediaccc

Updated Mar 14, 2023
Java

codelibs / fess

Fess is very powerful and easily deployable Enterprise Search Server.

search java search-engine elasticsearch crawler full-text-search lucene fulltext-search enterprise-search

Updated Mar 16, 2023
Java

xuxueli / xxl-crawler

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

java crawler web spider flexible distributed object-oriented xxl-crawler

Updated Oct 15, 2022
Java

fengzhizi715 / NetDiscovery

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

kotlin redis middleware crawler kafka spider dsl coroutines selenium rxjava2 lettuce disruptor htmlunit vertx3

Updated Nov 28, 2020
Java

fanyong920 / jvppeteer

Headless Chrome For Java （Java 爬虫）

java crawler chrome scraper chrome-headless puppeteer jvppeteer

Updated Feb 19, 2023
Java

jaeksoft / opensearchserver

Open-source Enterprise Grade Search Engine Software

search java search-engine enterprise crawler ocr indexing synonyms lucene webcrawler custom-search webcrawling opensearchserver

Updated Sep 3, 2022
Java

crawljax / crawljax

Sponsor

Crawljax

crawler dom dynamic crawling test-generation web-testing web-analysis event-driven-crawling

Updated Mar 16, 2023
Java

smuyyh / CrawlerForReader

Android 本地网络小说爬虫，基于jsoup及xpath

android crawler jsoup xpath bookreader

Updated Sep 2, 2020
Java

yAnXImIN / weiboPicDownloader

免登录下载微博图片爬虫 Download Weibo Images without Logging-in

java crawler weibo

Updated May 20, 2022
Java

tim232385 / WebVideoBot

Web crawler.

crawler spider pornhub

Updated Dec 1, 2019
Java

commoncrawl / news-crawl

News crawling with StormCrawler - stores content as WARC

crawler news web-crawler apache-storm warc commoncrawl common-crawl

Updated Nov 16, 2022
Java

codesofun / web-bee

🐝 Web vertical crawler framework for fun

java crawler framework java-8 webbee

Updated Sep 1, 2022
Java

luohaha / jlitespider

A lite distributed Java spider framework :-)

distributed-systems crawler spider rabbitmq distributed

Updated May 3, 2017
Java

greengerong / prerender-java

java framework for prerender

java crawler seo angular1 prerender prerendered-page

Updated Sep 14, 2022
Java

Followers: 264 followers
Wikipedia: Wikipedia