Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.
6
votes
2answers
282 views
Parsing Wikipedia table with Python
I am new to Python and recently started exploring web crawling. The code below parses the S&P 500 List Wikipedia page and writes the data of a specific table into a database.
While this script is ...
5
votes
1answer
37 views
Magpi Magazine Downloader
I have created a simple program at https://github.com/Epic0ne/magpi-downloader in Python which downloades all issues of the MagPi magazine by parsing this web page for links ending in ...
7
votes
2answers
143 views
Optimizing the speed of a web scraper
I have just written this code to scrape some data from a website. In its current state it works fine, however, going by my tests on the script, I discovered that with the amount of data I am ...
5
votes
1answer
104 views
Simple recursive web crawler
I did a simple web crawler, I know there's many better ones out there, I did this just with the learning purpose.
The problem is that I think there's some things I could improve here. I commented the ...
6
votes
1answer
104 views
Basic IMDb scraper and movie generator
I just built my first scraper and I'd like to get your thoughts on the structure and the way I went about it. The basic premise of the script:
Get a random movie from IMDb's Top 250
Ask the user ...
2
votes
1answer
67 views
Scrapy Username Spider
This currently starts out with the speed of 2000 pages/min but shortly after starting it becomes very slow with a speed of about 200 pages/min. Why is this happening? How can I improve this scraper?
<...
3
votes
2answers
218 views
Scraping HTML via async controller & classes + HTML agility pack
I've developed a simple application to grab golfer index scores from a website that has no API. The application works but is very slow, with 6 users that require updating takes 60 seconds. I've tried ...
0
votes
0answers
224 views
Python Scrapy code using Selenium
I have written some Python code that uses Scrapy and Selenium to scrap restaurant names and addresses from a website. I needed to use Selenium because the button to show more restaurants on a page is ...
5
votes
1answer
104 views
Webscraping calendar events using Python 3, with or without BeautifulSoup
I'm trying to find out why my web-scraping code with BeautifulSoup (BS) is slower than my code without BS. I would think that BS code would be faster than the other code - so, maybe I'm doing ...
5
votes
1answer
73 views
Scraping names of directors from a website
I am scraping the names of the directors from a website using Python / ScraPy. I am very new to coding (under a year and after work) - any views would be appreciated.
The reason I have a ...
4
votes
2answers
67 views
Scrape 4chan for alive images
I'm trying to learn Clojure recently and I thought writing a simple web app would be a good way to dive in.
This function gets the list of alive threads from the API and reduces, filters and maps ...
0
votes
3answers
106 views
Image hosting site image downloader using requests and BeautifulSoup
I went about this the way I have because Selenium is slow and inconvenient (opens browser) and I was unable to find the href attribute in the link element using ...
2
votes
1answer
99 views
Scrape an infinite-scroll page
My algorithm scrapes an infinite-scroll page but it takes too long. It scrolls three times but I'm wondering if there is a way to do a ScrollBottom() so no need of ...
4
votes
2answers
859 views
Simple Python job vacancies downloader
I have created a BeautifulSoup vacancies parser, which works, but I do not like how it looks. Therefore, I'd be very happy if somebody could give me any improvements.
...
3
votes
1answer
91 views
VBA - XMLHTTP web scraping
I navigate with IE, do various things, then select all results option from a list and fire on click event. Once all results have been listed, I loop through their URLs, using the following code to ...
0
votes
0answers
65 views
Scraping websites and saving to MySQL
I have the following piece of code which scrapes websites and saves some information back to MySQL.
At the moment is consuming all the memory on my machine every time it runs.
I've refactored the ...
4
votes
1answer
206 views
Web scraping VBA - Internet Explorer
The code below extracts data from one web page - I emulate search, select all results from the list and when the list appears (42000 items) I loop through these items.
I get an id value from their ...
1
vote
1answer
175 views
Basic web scrape project written in NodeJS
Here is a short program web scraping program written in Node.js. I'm just getting to grips with node and this is the first thing I've written with it. I'm liking it so far though I guess I'm kinda ...
1
vote
0answers
127 views
GUI in Tkinter to log events for a web-scraper
I'm creating a GUI with tkinter that will handle starting/stopping/and logging events for a web-scraper (scraper not created yet).
The current code is working... but I've been gathering my ...
2
votes
1answer
124 views
Sample scraping Project Gutenberg using Beautiful Soup and requests
I am trying to learn web scraping in Python using Beautiful Soup and requests. My program goes to the book page on Project Gutenberg with the given book number (Example). It then finds the link for ...
2
votes
1answer
59 views
Downloading and saving news articles
To be honest, I am pretty new to coding. I want to analyse the articles my local newspaper publishes on ther website. To do that, I have a list of URLs and from that, I want to download and save all ...
4
votes
1answer
45 views
Haikuifier (Or at least Haiku Identifier)
All the usual stuff. Style, substance, algorithm please! I'm vaguely considering plugging it into a bot, hence the lacklustre catching of exceptions right now.
...
2
votes
1answer
119 views
Translating text using Google Translate mobile site
I have this code to translate text using google translate mobile site. currently text size is limited by the request method.
Everything else seems to works just fine
I am also about to post this on ...
1
vote
0answers
164 views
Google Searching Bot with Proxy support
I have been asked by a client to program a bot which searches Google and will show how many no of results I get.
Note: I know about Google Custom Search API and it will not produce the exact output ...
0
votes
2answers
106 views
Movie data scraping
I enter in the IMDb link and YouTube trailer link in the command line to a movie and the first main program loads all the info about the movie. The second main program uses an IMDb link to the movie ...
3
votes
0answers
26 views
VIM colors downloader in Python, using multiprocessing
I recently posted this script:
VIM colors downloader in Python
But since I'm not allowed to update the code there, I wanted to get an idea on this version, that uses multiprocessing:
...
8
votes
2answers
119 views
Finding words that rhyme
Preface
I was trying to review this question on the same topic, but in the end many points I wanted to make were excellently explained by @ferada so I felt that posting my code and explaining the ...
14
votes
2answers
728 views
VIM colors downloader in Python
Recently, I wanted to change my vim colors to something new. So I went to the vim colors website and then I decided that I wanted to download ALL the colors.
So I ...
3
votes
1answer
179 views
Python Document Downloader
This is a python document (PDF) downloader I made to download some question papers automatically. However, it is very slow.
Any better way to do this?
The code:
...
9
votes
2answers
930 views
Simple Python username scraper
I started learning Python recently and I really like it, so I decided to share one of my first projects mainly in hopes of someone telling me what I can do to make it run faster (threading/...
5
votes
2answers
211 views
Multithreaded webcrawler
I've been trying to learn Java for the last day or two. This is the first project I am working on, so please bear with me. I worked on a multithreaded web crawler. It is fairly simple but I'd like to ...
-1
votes
1answer
132 views
Selenium Kijiji web scraper
I have this script working pretty well but I know that there must be many things that I could do better to make it more efficient.
...
5
votes
1answer
78 views
Python Politico API attempt
I love politics, and I love programming, so I figured why not try and combine the two for something to do? I'm making a work-in-progress (but runnable at this stage) Politico api that I call "...
1
vote
0answers
19 views
Wikipath stack in Java - Part II/IV - The implicit Wikipedia article graph
This question is the continuation of the Wikipath stack series: the two classes that - given a Wikipedia article \$A\$ - return the lists of neighbour articles. The forward node expander return the ...
0
votes
1answer
66 views
1
vote
1answer
662 views
Downloading yahoo finance stock historical data as CSV using C++
This post is a continuation of my previous post where I used jsoncpp package to fetch exchange rates from fixer.io. In this post I have reused the above code and used it to fetch stock historical ...
3
votes
1answer
93 views
Fetching specific foreign exchange rates from fixer using curl and jsconcpp in C++
I am trying to create my own algorithmic trading system using C++. I have searched the web for a nice tutorial for such systems and I didnt find any. Then I started to learn about ...
2
votes
1answer
70 views
Optimizing Java HTML parser
I wrote a program that goes through a webpage and returns matches of regex. I used it on my letterboxd.com account to go through all of my movies (over 900 entries) and then find genres field for each ...
6
votes
2answers
126 views
Crawl site getting URL and status code
I wrote a crawler that for every page visited collects the status code.
Below my solution. Is this code optimizable?
...
5
votes
1answer
227 views
Session handling using Python Requests client
I'm using this code to login to an experiment login system created by me for this purpose.
...
5
votes
1answer
124 views
Web scraping VBA and VB Script
I am working on a project on VBA where the objective is to have a "program" that fetches rates from a website called X-Rates, and outputs to excel the monthly averages of a chosen country.
Initially ...
8
votes
0answers
67 views
Regex-guided crawler that downloads regex-matching images up to a crawling level
This is one simple crawler that downloads images from websites, the website's URL to be crawled to must match the regex, as well as any image-to-download's URL.
(Also, I know, I made my own thread ...
6
votes
1answer
110 views
Simple image scraping
I wrote this code over the last few days and I learned a lot, and it works as I expect. However I suspect it's woefully inefficient:
...
10
votes
1answer
1k views
Scraping after login using Scrapy
I just finished a scraper in python using scrapy. The scraper logs in to a certain page and then scrapes a list of other pages using the authenticated session.
It retrieves the title of these pages ...
6
votes
1answer
44 views
Script to download sequentially named files, rename them, and delete smaller files
I've written a little script to download sequentially named files, rename them, and delete files smaller than an certain number of kilobytes. I came up with this but I'm not too happy. Any advice for ...
2
votes
1answer
425 views
PHP web crawler
I'm working on a "nice" crawler that start with one URL, and find the other URLs to process each page, a kind of "Google" crawler, to index pages.
I worked hard on this crawler to respect many points ...
4
votes
2answers
66 views
I'll visit the 18th
I wrote this program, which purpose is to visit the 18th link on the list of links and then on the new page visit the 18th link again.
This program works as intended, but it's a little repetitive and ...
4
votes
1answer
106 views
Recursive Web Crawler in Go
This is probably my third Go application. It essentially takes one or two command line arguments of wikipedia articles and pulls every /wiki/ link that isn't a special page, memoizes them to avoid ...
0
votes
0answers
57 views
Finding shortest paths in a Wikipedia article graph using Java - second attempt
I have improved Finding shortest paths in a Wikipedia article graph using Java.
Now I have this:
AbstractWikipediaShortestPathFinder.java:
...
9
votes
3answers
244 views
Scraping the date of most recent post from various social media services
Task
I have a large spreadsheet where each line should include:
The URL of a social media account
A field indicating whether the account is "active"
A name and UID number for each account
I have to ...
