Tell me more ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.
up vote 1 down vote favorite

I am planning to create a website like what http://dohop.com is doing that will allow a user to pull the airlines price rate and date from the http://airasia.com website. Currently the site will only allow the user to view the flight schedule for one day and if they wish to view x days ahead they need to repost the data.

I would like to collect the data in x days ahead and group them in table so that user can view all the flight prices & schedule variances in one screen without reposting.

I have checked the AirAsia.com site and they currently don't have any API support which would allow me to extract their data. While they are using 'aspx' for their website and POST method.

Can anyone give me some guidance on what is the method, approach or technique for me to harvest the data?

share|improve this question
it's called Screen-scraping - I'm sure Google will turn up many articles on this. I've seen ppl try to screen-scape a asp site before and it was really hard - all kind of wierdly encrypted tokens passed back and forth. Good luck. – James Jun 5 '11 at 7:46

2 Answers

up vote 5 down vote

There's a reason they don't have an API - they want to protect their data. In their terms of use they specifically state they own the copyright to all the data on their site:

This Website and the content herein, including flight schedules, routes, fares, text, graphics, button icons, audio and video clips, digital downloads, data compilations, logos and information regarding the status of AirAsia flights, etc. are proprietary to AirAsia or its content suppliers. You may use this Website and the content only for personal and non-commercial purposes subject to applicable copyright laws.

I'd suggest you avoid trying to scrape the site as it quite illegal.

share|improve this answer
1  
I'm not a legal expert, and if you are, please correct me, but I believe that while a database as a whole can be subject to copyright protection, individual pieces of information within the database may not be. I'd consult a lawyer regarding that issue. – Chuck May 30 '11 at 12:43
2  
I'm not a lawyer but I have a lot of experience with content copyright. Their terms specifically state that the data you want to retrieve is proprietary and can't be used like you intend. – Eran Galperin May 30 '11 at 13:10
Depends where you are tho. There are fair use provisions that will depend on your country. Also often ppl claim copyright over things that can't actually be copyrighted. I'm not saying it's legal, or that OP shouldn't expect to have to hire a lawyer at some point if his site becomes popular and AirAsia notice - I'm just saying its not necessarily completely illegal. See if AirAsia turn up in skyscanner.com - while they try to get data from companies with agreement they will happily screen-scrape airlines worldwide if they can't. – James Jun 5 '11 at 7:40
Also, there is a difference between "illegal" and "in trouble" - specifically, it depends what AirAsia think of it. From what it sounds like, OP's site may actually help AirAsia. Of course, whether they have the common sense to recognise that is another matter. – James Jun 5 '11 at 7:44
up vote 1 down vote

I'm not saying I robbed the bank, all I'm suggesting is that if I were do to it I would use cURL and more specifically PHP cURL.

http://davidwalsh.name/execute-http-post-php-curl

But a word of warning, as Eran Galperin did point out, this might go against their terms of service, and if you do use cURL to extract the data and use it as a commercial advantage you will not be a happy camper.

share|improve this answer
Even though this might answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. – Anna Lear♦ Jun 5 '11 at 1:40
There are libraries in PHP for parsing HTML. I hope you're using this too. – Ross Jun 5 '11 at 6:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.