All Questions
Tagged with web-scraping php
17 questions
4
votes
2
answers
356
views
PHP script that writes a JSON file with iextrading API data
iextrading has two APIs (v1, v2) which provide financial market data. Following script records a JSON file with their equity information (using a CRON job).
Would you be kind and review it for any ...
- 3,732
10
votes
1
answer
287
views
Chart showing the Percentage of Answered CR questions
Every day Duga reports the number of unanswered questions and percentage of answered questions (after she calls the API endpoint /info) in The 2nd Monitor.
I ...
- 29.6k
1
vote
1
answer
111
views
BFA protected phpmyadmin
I use a Ubuntu 16.04 Nginx server environment with phpmyadmin (PMA). All my ports are closed besides 22, 25, 80, 443, 9000 (for PHP-FPM).
A common criticism of the good software (PMA) as of 2018, is ...
- 240
0
votes
2
answers
572
views
Movie data scraping
I enter in the IMDb link and YouTube trailer link in the command line to a movie and the first main program loads all the info about the movie. The second main program uses an IMDb link to the movie ...
- 101
1
vote
1
answer
103
views
3
votes
1
answer
1k
views
PHP web crawler
I'm working on a "nice" crawler that start with one URL, and find the other URLs to process each page, a kind of "Google" crawler, to index pages.
I worked hard on this crawler to respect many points ...
- 275
6
votes
2
answers
434
views
Steam Community Market Strange Part Scraper
A while back I wrote a simple little PHP script that searches the Steam community market for any TF2 strange weapons with strange parts on the first page of results for that weapon type. It works by ...
- 163
0
votes
1
answer
267
views
CRAP index (56) in Web Scraper Engine
I am working on a Web Scraper for the first time using Test Driven Development, however I have caught myself into a huge CRAP (Change Risk Anti-Patterns) index (56) and I can not seem to find a ...
- 89
2
votes
2
answers
454
views
PHP crawler to collect comments on articles
I have code that parses through web pages finds commentaries and saves commentary info in DB. I have an array where all necessary pages are stored. I iterate through all these pages one by one and ...
- 121
5
votes
1
answer
161
views
Mixed scripting language API to determine file upload location and scrape city government website to find corresponding government official
I wrote this script as an NYC-specific-API for file upload for a mobile app. Users upload a video file and also their geographic coordinates.
I then use an external API to get the corresponding ...
- 1,845
2
votes
1
answer
467
views
Reducing execution time of an HTML parsing script
The script is intended to return an array with texts containing specific words in English and the equivalent texts in Polish from EUR-Lex - a website with EU documents.
The script downloads the page ...
- 21
1
vote
1
answer
197
views
Formatting HTML for use in a locally hosted iframe
This formats HTML for use in a locally hosted iframe so that you can manipulate the content in the iframe freely, without running into cross domain issues. It uses Goutte to retrieve the HTML. I'd ...
- 43
2
votes
0
answers
2k
views
Scraping HTML using PHP
Because a website I need data from doesn't have any API or RSS feed for their service status, I use a Web Scraper I built using PHP to grab the data I need and structure it as JSON. However I want to ...
- 121
8
votes
1
answer
286
views
Optimize web-scraping of Moscow grocery website
This code works fine, but I believe it has optimization problems. Please review this.
Also, please keep in mind that it stops after each iteration of the loop ...
- 183
1
vote
1
answer
2k
views
Scraping links from provided URLs
This script takes the list of provided URLs and scrapes the present links in each URL. For each scraped link, Facebook share, tweet, Google Plus is found out.
For example, if the provided URL is www....
- 111