All Questions
Tagged with web-scraping ruby
16 questions
3
votes
1
answer
114
views
Tentative to transform this scrape script into a class object program
From the last few days I have been studying scraping webpages and for further development I challenge myself to transform the script I have created into a class which will receive inputs from a user.
...
- 33
2
votes
1
answer
127
views
Web scraper extracts zip files and publishes them to redis
I wrote this program for a job interview a while back. I would like general feedback on style, readability, maintainability, or any obvious flaws:
...
- 179
3
votes
1
answer
94
views
Email finding bot
I'm working on a lead generation bot that helps you find the emails of people you want to reach out to.
The bot grabs your spreadsheet from gdrive, logs into several email finding tools, and collect ...
- 133
0
votes
1
answer
76
views
Skipping Google search results that point to certain sites
I have the following code that will skip certain URLs if needed:
...
- 872
4
votes
0
answers
222
views
Web scraping with Nokogiri
At work we have a need to know what printers are getting dangerously low on their toner, and paper consumption, etc..
So I've created a program that pulls the printer information off the websites the ...
- 41
3
votes
0
answers
146
views
Using Nokogiri to scrape Oscars winners from Wikipedia
I am scraping a Wikipedia page, getting info from that page and instantiating a new object with the information collected:
...
- 223
8
votes
1
answer
6k
views
Refactoring a Crawler
I've recently ported an old project and made it object-oriented. However, I've noticed that rubocop points out the following status: ...
user27606
3
votes
0
answers
578
views
Rails app that scrapes forum using Nokogiri gem
I've built a website that scrapes a guitar forum's pages and populates Rails model. I'm using rake task along with heroku scheduler to run background scrapes every hour.
On the homepage, the forum ads ...
- 325
2
votes
1
answer
121
views
Find and select image files from webpage
For some reason, I feel like this is a bit messy and could be cleaner. Any suggestions?
I'm selecting any image files ending in .png or ...
user27606
14
votes
2
answers
779
views
Nokogiri crawler
The following code works but is a mess. But being totally new to Ruby I have had big problems trying to refactor it into something resembling clean OOP code. Could you help with this and explain what ...
- 243
5
votes
1
answer
178
views
Reverse-engineering with Filepicker API
I have this script to pull data out of the Filepicker API internal. It's mostly reverse-engineering and the code seems to be ugly to me. How can this be improved?
...
- 405
3
votes
1
answer
293
views
Crawling for emails on websites given by Google API
I'm trying to build an app which crawls a website to find the emails that it has and prints them. I also want to allow the user to type "false" into the console when they want to skip the website (...
- 369
5
votes
2
answers
135
views
Scraping and analyzing recent articles on bitcoin
This script scrapes recent articles on bitcoin, does sentiment analysis, and does some mock trading based on the sentiment of the articles. I'm looking for advice on code style, and I would love to ...
- 558
5
votes
1
answer
548
views
Craigslist search-across-regions script
I'm a JavaScript developer. I'm pretty sure that will be immediately apparent in the below code if for no other reason than the level/depth of chaining that I'm comfortable with. However, I'm learning ...
- 153
4
votes
1
answer
364
views
Web scraper for job listings
Is there any room for improvement on this code?
I use mechanize to get the links of a job listing web site. There are pages with pagination (when jobs > 25) and pages without.
If there is, then the ...
- 141