CommonCrawl

cc-pyspark Public
Process Common Crawl data with Python and Spark

Python 187 MIT 64 6 4 Updated Dec 17, 2021
cc-index-table Public
Index Common Crawl archives in tabular format

Java 39 Apache-2.0 4 5 0 Updated Dec 13, 2021
cc-crawl-statistics Public
Statistics of Common Crawl monthly archives mined from URL index files

Python 45 Apache-2.0 7 0 0 Updated Dec 10, 2021
cc-notebooks Public
Various Jupyter notebooks about Common Crawl data

Jupyter Notebook 12 Apache-2.0 4 1 0 Updated Dec 7, 2021
nutch Public
Common Crawl fork of Apache Nutch

Java 19 Apache-2.0 1,184 3 (1 issue needs help) 0 Updated Dec 4, 2021
news-crawl Public
News crawling with Storm-crawler - stores content as WARC

Java 151 Apache-2.0 19 8 0 Updated Oct 29, 2021
commoncrawl-crawler Public
The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)

Java 203 63 0 1 Updated Oct 29, 2021
warcio Public
Streaming WARC/ARC library for fast web archive IO

Python 0 Apache-2.0 44 0 0 Updated Oct 20, 2021
cc-mrjob Public
Demonstration of using Python to process the Common Crawl dataset with the mrjob framework

Python 156 MIT 80 2 2 Updated Oct 18, 2021
cc-webgraph Public
Tools to construct and process webgraphs from Common Crawl data

Shell 17 Apache-2.0 2 1 0 Updated Oct 4, 2021

Pinned