https://docs.scrapy.org/en/latest/topics/link-extractors.html mentions that link extractors may be used outside CrawlSpider, but it does not go into detail on how to do that.

Also, there are broken references to scrapy.link.Link, we should provide reference documentation for that class.

If you're using proxies with requests-html and rendering JS sites is all good. Once you render a website pyppeteer don't know about this proxies and will expose your IP. This is an undesired behavior when scraping with proxies.

The idea is that whenever someone passes in proxies to the session object or any method call, make pyppeteer also use these proxies. #265

Whenever CLI process gets interrupted or killed, CDP driver must (and used to) close all open tabs.
It stopped doing this.

Main examples at Apify SDK webpage, Github repo and CLI templates should demonstrate how to manipulate with DOM and retrieve data from it.

Also add one example of scraping with Apify SDK + jQuery to https://sdk.apify.com/docs/examples/basiccrawler

Feedback from: https://medium.com/better-programming/do-i-need-python-scrapy-to-build-a-web-scraper-7cc7cac2081d

I lost an hour trying to make

My project have routing based on hosts. But web driver make request to http://127.0.0.1:9080.
How can i change host?

As you can see here, half of download_history.py is not covered by unittests.
At least, I want to test DownloadHistory.get fucntion.

We can test this module by 3 steps.

• Clear utils.cache_dir / 'download-history.jsonl'.
• Execute oj d, or directly instanciate DownloadHistory and