This book uses practical examples and step-by-step instructions to guide you through the basic techniques required for web scraping with PHP. This will provide the knowledge and foundation upon which to build web scraping applications for a wide variety of situations relevant to today's online data-driven economy.
Preparing your development environment (Simple), explains how to install and configure necessary software for development environment – IDE (Eclipse), PHP/MySQL (XAMPP) browser plugins for capturing live HTTP Headers, and Web Developer for setting environment variables.
Making a simple cURL request (Simple), explains how to request a web page using cURL, instructions and code for making a cURL request, and downloading a web page. The recipe also explains how it works, what is happening, and what the various settings mean. It also covers various options in cURL settings, and how to pass parameters in a GET request.
Scraping elements using XPath (Simple), explains how to convert a scraped page to a DOM object, how to scrape elements from a page based on tags, CSS hooks (class/ID), and attributes, and how to make a simple cURL request. It also discusses the instructions and code for completing a task, explains what XPath expressions and DOM are, and how the scrape works.
The custom scraping function (Simple), introduces a custom function for scraping content, which is not possible using XPath or regex. It also covers the instructions and code for the custom function, scrapeBetween()
.
Scraping and saving images (Simple), covers the instructions and code for scraping and saving images as a local copy, and also verifying whether those images are valid.
Submitting a form using cURL (Intermediate), covers how to capture and analyze HTTP headers, how to submit (POST) a form, for example, a login form using cURL and cookies, or a web page with a form. It also covers the instructions on how to read HTTP headers for necessary info required to POST, instructions and code for posting using PHP and cURL, explanation of what is happening, how headers are being posted, and how to post multipart/upload forms.
Traversing multiple pages (Intermediate), explains topics such as identifying pagination, navigating through multiple pages, and associating scraped data with its source page.
Saving scraped data to a database (Intermediate), discusses creating a new MySQL database, using PDO to save the scraped data to a MySQL database, and accessing it for future use.
Scheduling scrapes (Simple), discusses how to schedule the execution of scraping scripts for complete automation.
Building a reusable scraping class (Advanced), introduces basic object oriented programming (OOP) principles to build a scraping class, which can be expanded upon and reused for future web scraping projects.
Bonus recipes covers topics such as how to recognize a pattern using regular expressions, how to verify the scraped data, how to retrieve and extract content from e-mails, and how to implement multithreaded scraping using multi-cURL. These recipes are available at http://www.packtpub.com/sites/default/files/downloads/4760OS_Bonus_recipes.pdf.
Any basic knowledge of PHP or HTML will be useful, though not necessary
The following are the requirements:
Eclipse
Apache, PHP, and MySQL (XAMPP)
Download, installation, and configuration instructions are included in the Preparing your development environment (Simple) recipe.
This book is aimed at those who are new to web scraping, with little or no previous programming experience. Basic knowledge of HTML and the Web is useful, but not necessary.
In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.
Code words in text are shown as follows: " We create the curlPost()
function, which is used to make a cURL request."
A block of code is set as follows:
<form action="/account" accept-charset="UTF-8" method="post" id="packt-login-form">
New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: " Select Daily, and then click on Next."
Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.
To send us general feedback, simply send an e-mail to <[email protected]>
, and mention the book title via the subject of your message.
If there is a book that you need and would like to see us publish, please send us a note in the SUGGEST A TITLE form on www.packtpub.com or e-mail <[email protected]>
.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the files e-mailed directly to you.
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/support, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded on our website, or added to any list of existing errata, under the Errata section of that title. Any existing errata can be viewed by selecting your title from http://www.packtpub.com/support.
Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at <[email protected]>
with a link to the suspected pirated material.
We appreciate your help in protecting our authors, and our ability to bring you valuable content.
You can contact us at <[email protected]>
if you are having a problem with any aspect of the book, and we will do our best to address it.