Skip to content
#

crawling

Here are 568 public repositories matching this topic...

scrapy
ferret
JekRock
JekRock commented Sep 19, 2020

Is your feature request related to a problem? Please describe.
Currently, there are services that secure website from automation tools like ferret. Some of them send 405 in response to the DOCUMENT function call that make a ferret script fail with an error even though a page is available (not the original page, but usually a page with the captcha).

Describe the solution you'd like
It

apify-js
mtrunkat
mtrunkat commented Sep 17, 2019

Main examples at Apify SDK webpage, Github repo and CLI templates should demonstrate how to manipulate with DOM and retrieve data from it.

Also add one example of scraping with Apify SDK + jQuery to https://sdk.apify.com/docs/examples/basiccrawler

Feedback from: https://medium.com/better-programming/do-i-need-python-scrapy-to-build-a-web-scraper-7cc7cac2081d

I lost an hour trying to make

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

  • Updated Nov 13, 2019
  • C#

Improve this page

Add a description, image, and links to the crawling topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the crawling topic, visit your repo's landing page and select "manage topics."

Learn more

You can’t perform that action at this time.