All Questions
Tagged with web-scraping python-3.x
205 questions
5
votes
2
answers
687
views
Readability and error handling improvements for Python web scraping class
Description
I recently wrote a Python script to download files from the Library of Congress (LOC) based on a search query. The code fetches metadata, extracts file ...
- 253
3
votes
1
answer
99
views
Scraping website with Python and Selenium to collect data from dynamic website
Summary:
The code scrapes the website and collects the data to store it in CSV. It also downloads selected information that is available for download in PDF format. The details and the entire code are ...
- 269
2
votes
1
answer
72
views
A selenium web scraper to package NBA data
I'm building a selenium web scraper for basketball-reference.com that takes a player name and returns data in either a JSON format or Pandas DataFrame object. The class in question is one of many that ...
- 21
2
votes
1
answer
116
views
Search Stack Overflow and GitHub for code in a specified language
This code is designed to scrape Stack Overflow and GitHub, pulling information based on a user-specified programming language and processing the data into a format for AI learning.
It uses a number of ...
- 39
3
votes
1
answer
229
views
A simple web scraper for nature.com news articles
I have created a simple web scraper that fetches news article previews from nature.com and saves each article to a file containing the article preview text.
I am learning independently, so I would ...
- 31
3
votes
2
answers
233
views
Saving Scraped Data to a File
When scraping and saving data into a file, Which method is more efficient when saving scraped data to a file?
open the file first, scrape, and save the data all ...
- 201
3
votes
1
answer
57
views
request data and print results
On last test, the below code takes approximately 10 seconds to download then print the data from 10 url's. I wish to speed this up as much as possible as later on I plan to expand this further and use ...
- 67
3
votes
1
answer
219
views
Scrape PokeDex and display in tkinter
Hi I am new here and I just completed my first working version of a pokedex app with a GUI using tkinter. I used selenium to scrape the data from pokemondb.net, and then used pandas to clean up the ...
- 33
9
votes
2
answers
2k
views
Python script to scrape and parse the Stanford Encyclopedia of Philosophy
I wrote the following script to parse an SEP article and call pandoc to convert it to EPUB. I'd love your feedback.
There is no function but I didn't think it was worth adding. Also there is no test ...
user260644
1
vote
2
answers
122
views
Scraper to grab publicly available data
I am using Python scraper code to grab publicly available data from http://103.48.16.132/echalan/, but it takes almost 6gb of memory and more cpu. I need to run ...
- 119
3
votes
0
answers
821
views
A simple Python script that crawls information about Youtube playlists and your watch history
You will need to follow this guide.
This is a Youtube crawler that crawls information about Youtube playlists, it uses Youtube Data API v3 and it crawls the title, url, description, count and videos ...
- 3,702
3
votes
0
answers
106
views
Scraping housing dataset row by row from Estate Agent's website
I have written web scraping code with selenium with python3 for a dynamic site (it was my first project that is why I started with selenium). In my code I extracted the values and created DataFrame.
...
- 31
8
votes
1
answer
153
views
Scrape a URL for articles to save
Below is an exercise project I was doing on a certain educational site.
It is supposed to parse a given (static address in this example) URL for html data, search articles of a given type there and ...
- 145
4
votes
2
answers
632
views
Web scraping data.cdc.gov for COVID-19 Data with Selenium in Python
I'm attempting to scrape data.cdc.gov for their COVID-19 information on cases and deaths.
The problem that I'm having is that the code seems to be very inefficient. It takes an extremely long time for ...
- 55
2
votes
0
answers
169
views
Web scraping articles using asyncio
I use asyncio to speed up web scraping. I collect only title, author, tags, datetime, total comments from list view from specific website. Also, i collect these from all pages.
I would like to improve ...
- 417