webarchiving

Star

Here are 35 public repositories matching this topic...

iipc / awesome-web-archiving

Star

An Awesome List for getting started with web archiving

awesome awesome-list webarchiving

Updated Nov 9, 2020

N0taN3rd / Squidwarc

Star

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head

crawler chrome crawling chrome-headless browser-automation headless-chrome webarchiving webarchives high-fidelity-preservation puppeteer

Updated May 19, 2020
JavaScript

machawk1 / awesome-memento

Star

A list of things related to software, literature, and other content for 🕣 Memento

awesome memento awesome-list webarchiving memento-rfc

Updated Feb 25, 2020

N0taN3rd / node-warc

Star

Parse And Create Web ARChive (WARC) files with node.js

warc web-archiving webarchive web-archives webarchiving warc-files chrome-remote-interface pupeteer

Updated Dec 11, 2020
JavaScript

peterk / warcworker

Star

A dockerized, queued high fidelity web archiver based on Squidwarc

crawling archiving preservation webarchiving webarchives high-fidelity-preservation

Updated Jul 19, 2020
Python

ArchiveTeam / wget-lua

Star

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

crawler spider lua crawling archiving wget crawl zstd warc webarchiving archiveteam wget-lua wget-at

Updated Oct 29, 2020
C

ArchiveTeam / WebArchiver

Star

Decentralized web archiving

python crawler web decentralized archiving archiver warc webarchiving

Updated Aug 7, 2018
Python

WebarchivCZ / Seeder

Star

Seeder - Czech webarchive curating tool and public site

government django tools czech czech-republic archive webarchive webarchiving webarchives

Updated Dec 21, 2020
Python

peterk / munin-indexer

Star

A social media open post web archiving tool

archiving preservation webarchiving high-fidelity-preservation

Updated Sep 15, 2020
JavaScript

httpreserve / httpreserve

Star

Digital Preservation of HTTP in documentary heritage.

archives code4lib internetarchive digipres webarchiving digitalpreservation documentary-heritage digital-repositories

Updated Jan 25, 2020
Go

atomotic / pywb-recorder-tor

Star

pywb recorder over tor, anonymously records the web. (docker image)

tor webrecorder webarchiving

Updated Dec 10, 2018
CSS

atomotic / webrecorder-chrome-extension

Star

record current active tab on webrecorder.io

chrome-extension webrecorder webarchiving

Updated May 9, 2017
JavaScript

httpreserve / tikalinkextract

Star

Tika based link extractor for httpreserve

tika archives code4lib webarchiving digitalpreservation httpreserve tika-wrapper

Updated Jan 25, 2020
HTML

oduwsdl / tmvis

Star

An archival thumbnail visualization server

visualization nodejs archive memento webarchiving timemap tmvis webpage-changes

Updated Sep 2, 2020
JavaScript

httpreserve / linkscanner

Star

A helper package to tokenize textual content and retrieve hyperlinks

archives code4lib webarchiving digitalpreservation documentary-heritage httpreserve

Updated Mar 16, 2019
Go

httpreserve / workbench

Star

Client app for httpreserve pkg that generates CSV, JSON, HTTP, and BoltDB

archives boltdb code4lib internetarchive webarchiving digitalpreservation digital-repositories

Updated Mar 23, 2019
JavaScript

exponential-decay / moonshine

Star

Given four bytes, download a random file from web archives implementing the UKWA Shine interface

archives glam code4lib digipres webarchiving warclight file-formats ukwa

Updated Jan 25, 2020
Go

datacoon / metawarc

Star

metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)

metadata osint warc webarchiving warc-files osint-python

Updated Dec 4, 2020
Python

mgunn001 / WebArchiving-SeminarCourse

Star

Class page for ODU CS 791 / 891 Web Archiving Seminar

webarchiving archived-webpages

Updated Dec 6, 2017
HTML

httpreserve / phantomjsscreenshot

Star

A wrapper for phantom.js commands for headless screenshots.

code4lib webarchiving digitalpreservation websnapshot httpreserve

Updated Apr 22, 2018
Go

pierlauro / MDBubing

Star

From WARC records to MongoDB documents

crawler crawling warc webarchive webarchiving warc-files warc-format warc-record bubing

Updated Nov 3, 2020
Java

gitdev-bash / webArchiver

Star

A archiving utility that is compatible with web servers.

webserver archiving webarchive webarchiving

Updated Dec 29, 2020
Python

N0taN3rd / whatDoWebPagesDo

Star

webarchiving webarchives high-fidelity-preservation web-page-info web-archive-analysis

Updated Sep 20, 2017
JavaScript

MozillaCZ / phpbbcrawler

Star

Link crawler for a phpBB forum

crawler tool phpbb wayback-archiver webarchiving

Updated Jul 17, 2017
Java

httpreserve / wayback

Star

A restrictied API in Golang for the (semi)-exposed functions of the internet archive.

archives code4lib internetarchive webarchiving digitalpreservation

Updated Apr 26, 2017
Go

httpreserve / million-dollar-webpage

Star

HTTPreserve Analysis of Million Dollar Web Page

digital-humanities web-archiving code4lib harvard webarchiving computing-history boingboing

Updated Jul 31, 2017

ibnesayeed / awesome-web-archiving

Star

An Awesome List for getting started with web archiving

awesome awesome-list webarchiving

Updated Mar 23, 2020

N0taN3rd / node-cdxj

Star

Parse CDXJ(https://github.com/oduwsdl/ORS/wiki/CDXJ) files with node.js

webarchive web-archives webarchiving cdxj

Updated Jul 20, 2017
JavaScript

arquivo / dspace-link-extractor

Star

Extracts links from DSpace repositories

java tika sitemaps webarchiving

Updated Oct 13, 2020
Java

UbuntuCZ / archiver

Star

Nástroj pro archivaci webových stránek na Wayback Machine

crawler wayback-archiver webarchiving

Updated Dec 30, 2018
Kotlin

Improve this page

Add a description, image, and links to the webarchiving topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the webarchiving topic, visit your repo's landing page and select "manage topics."

Learn more