scraping

Apify SDK — The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

npm automation scraping crawling javascript-library web-scraping web-crawling headless-chrome rpa apify puppeteer

Updated Jul 27, 2020
JavaScript

emadehsan / thal

Star

Getting started with Puppeteer and Chrome Headless for Web Scraping

nodejs mongodb mongoose scraping chrome-headless puppeteer

Updated Jan 28, 2020
JavaScript

NikolaiT / GoogleScraper

Star

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

python search-engine crawler scraping search-engines search-engine-optimization

Updated May 11, 2020
HTML

symfony / panther

Star

A browser testing and web crawling library for PHP and Symfony

php webdriver symfony scraping selenium chromedriver selenium-webdriver e2e-testing

Updated Aug 5, 2020
PHP

oscarotero / Embed

Star

Get info from any web service or page

embeds oembed twitter-cards scraping opengraph

Updated Aug 1, 2020
PHP

transitive-bullshit / awesome-puppeteer

Star

A curated list of awesome puppeteer resources.

automation awesome scraping crawling awesome-list headless-chrome puppeteer

Updated Aug 8, 2020

geziyor / geziyor

Star

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

go crawler scraper spider scraping

Updated Dec 12, 2019
Go

medialab / artoo

Star

artoo.js - the client-side scraping companion.

scraping

Updated Jun 6, 2020
JavaScript

holgerd77 / django-dynamic-scraper

Star

Creating Scrapy scrapers via the Django admin interface

python scraper django spider scraping scrapy webscraping

Updated Mar 25, 2020
Python

meetmangukiya / instagram-scraper

Star

Scrape the Instagram frontend. Inspired from twitter-scraper by @kennethreitz.

javascript client instagram scraping python-3-6 instagram-scraper no-authentication requests-html

Updated Jun 29, 2018
Python

istresearch / scrapy-cluster

Star

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

python redis kafka scraping distributed scrapy

Updated Jul 20, 2020
Python

iawia002 / Lulu

Star

[Unmaintained] A simple and clean video/music/image downloader 👾

python crawler scraper downloader video scraping crawling python3

Updated Oct 18, 2019
Python

sananth12 / ImageScraper

Star

✂️ High performance, multi-threaded image scraper

python scraper terminal command-line pypi scraping commandline-tool

Updated Jan 4, 2018
Python

speed / newcrawler

Star

Free Web Scraping Tool with Java

docker crawler spider scraping

Updated May 9, 2020
JavaScript

scrapy / parsel

Star

Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

css python xml scraping selectors xpath lxml

Updated Jun 18, 2020
Python

MorvanZhou / easy-scraping-tutorial

Star

Simple but useful Python web scraping tutorial code.

crawler regex scraping crawling requests asyncio scrapy beautifulsoup distributed-scraper urllib

Updated Oct 22, 2019
Jupyter Notebook

Lackoftactics / facebook_data_analyzer

Star

Analyze facebook copy of your data with ruby language. Download zip file from facebook and get info about friends ranking by message, vocabulary, contacts, friends added statistics and more