Description

When I scrape without proxy, both https and http urls work.
Using proxy through https works just fine. My problem is when I try http urls.
In that moment I get the twisted.web.error.SchemeNotSupported: Unsupported scheme: b'' error

As I see, most of the people have this issue the other way around.

Steps to Reproduce

1. Scrape a http link with proxy

**Expected

i want get the price (follow red frame)

	c.OnHTML("div[id=price]", func(e *colly.HTMLElement) {
		fmt.Printf("test----%+v\n",e)
		price,err := strconv.ParseFloat(e.Text,64)
		//price := e.Text
		fmt.Printf("********* price----%+v\n",price)

		if err != nil {

			fmt.Pri

If you're using proxies with requests-html and rendering JS sites is all good. Once you render a website pyppeteer don't know about this proxies and will expose your IP. This is an undesired behavior when scraping with proxies.

The idea is that whenever someone passes in proxies to the session object or any method call, make pyppeteer also use these proxies. #265

问题是这样的，我想爬取商品分页的信息于是我用了个for循环，执行document = Jsoup.connect(domain+reviews+String.format(b, p)).get()改变p的值来改变评论的页码。

但是当爬完第一页后再爬取第二页评论时（没准备爬取一页评论时都会执行这句document = Jsoup.connect(domain+reviews+String.format(b, p)).get();）出现了这样的错误：
java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.s

Tabula API version: 1.2.1.18052200
Filename: 3_2019년_통계부록.pdf
Internal Server Error (500)
    
      
        Request Method:
        POST
      
      
        Request URL:
        http://127.0.0.1:8080/pdf/8a6599b3be99fda826cc0448d74f0f74dfd3d78d/data

lines must be orthogonal, vertical and horizontal

Got this while extracting table
[pdf file](https://drive.google.com/fil

What is the current behavior?

Crawling a website that uses # (hashes) for url navigation does not crawl the pages that use #

The urls using # are not followed.

If the current behavior is a bug, please provide the steps to reproduce

Try crawling a website like mykita.com/en/

What is the motivation / use case for changing the behavior?

Though hashes are not ment to chan

The developer of the website I intend to scrape information from is sloppy and has left a lot of broken links.
When I execute an otherwise effective Ferret script on a list of pages, it stops altogether at every 404.
Is there a DOCUMENT_EXISTS or anything that would help the script go on?

There are several things not accurately documented/outdated:

• -v2 is used the examples but does not work
• # duckduckgo not supported although it is in the list of supported search engines
• To get a list of all search engines --config is suggested but that just fails

My project have routing based on hosts. But web driver make request to http://127.0.0.1:9080.
How can i change host?

I'm trying to type some stuff into a page w/ artoo, and i think simulate() will do the trick.

i've never used simulate(), though, so i have no idea what the syntax is.

the github page for simulate linked in the artoo docs has no documentation, and the only docs i can find are for jquery-simulate-ext.

are there any examples I

Checklist for items that I know need worked on before the ui branch can be merged into the dev branch

• Create documentation
• Add offline unit tests
• Add online integration tests
• Add tests to run_*_tests.sh
• Can we add actual ui tests? (aka Selenium o

I learned from the API documentation that the SelectorList class contains a remove method. But I don't see it in v1.5.2.

python3 list_links.py https://www.geeksforgeeks.org/category/advanced-data-structure/

python3 download_html.py JSON/Advanced-Data-Structure.json

python3 html_to_pdf.py HTML/Advanced-Data-Structure.html

I have written two pages in Japanese.

• https://online-judge-tools.readthedocs.io/en/master/introduction.ja.html
• https://online-judge-tools.readthedocs.io/en/master/run-ci-on-your-library.html

These documents should be translated to English.

Latest RI scraper code is rather tough due to page layout. Cam pointed out that the data is available in Gsheets:

This is better.

(https://docs.google.com/spreadsheets/d/1n-zMS9Al94CPj_Tc3K7Adin-tN9x1RSjjx2UzJ4SV7Q/edit#gid=0)

we can get JSON https://docs.google.com/spreadsheets/d/1n-zMS9Al94CPj_Tc3K7Adin-tN9x1RSjjx2UzJ4SV7Q/gviz/tq?tqx=out:json&sheet=Summary

or CSV https://docs.googl

• copying existing directory tree for a language to a new named branch
• how to update the snippets file
• adding logos
• ...?