Tagged Questions
10
votes
2answers
2k views
Good Free/Paid Web Scraping Library in C#.NET which handles javascripts [closed]
I am looking for a good paid/free web scraping library with .NET support which has decent support for JavaScript processing and offers very good performance.
It should have its own browser engine and ...
7
votes
6answers
8k views
How do you Screen Scrape?
When there is no webservice API available, your only option might be to Screen Scrape, but how do you do it in c#?
how do you think of doing it?
7
votes
4answers
2k views
Any Good Open Source Web Crawling Framework in C#
Iam building a shopping comparison engine and I need to build a crawling engine to perform the daily data collection process.
I have decided to build the crawler in C#. I have a lot of bad experience ...
5
votes
1answer
3k views
Headless browser for C# (.NET)?
I am (was) a Python developer who is building a GUI web scraping application. Recently I've decided to migrate to .NET framework and write the same application in C# (this decision wasn't mine).
In ...
4
votes
6answers
9k views
I need a Powerful Web Scraper library [closed]
I need a powerful web scraper library for mining contents from web. That can be paid or free both will be fine for me. Please suggest me a library or better way for mining the data and store in my ...
4
votes
5answers
955 views
How to do really mutithreaded web mining with IE/.Net/C#?
I want to mine large amounts of data from the web using the IE browser. However, spawning lots and lots of instances of IE via WatiN crashes the system. Is there a better way of doing this? Note that ...
4
votes
3answers
996 views
How do I scrape only the <body> tag off of a website
I'm working on a webcrawler. At the moment i scrape the whole content and then using regular expression i remove <meta>, <script>, <style> and other tags and get the content of the ...
4
votes
1answer
2k views
Html Agility Pack: Find Comment Node
I am scraping a website that uses Javascript to dynamically populate the content of a website with the Html Agility pack.
Basically, I was searching for the XPATH "\\div[@class='PricingInfo']", but ...
4
votes
1answer
291 views
How to use ScrapySharp to parse elements in an html document?
Here's the project official "Documentation":
https://bitbucket.org/rflechner/scrapysharp/wiki/Home
No matter what I try, I can't find the CssSelect() method that the library is supposed to add to ...
4
votes
1answer
256 views
Vbulletin scraper library?
Does anyone know of a library (preferably written in C#.NET) that scrapes VBulletin pages and just gets the main information for me? Needs to support logins and searching.
If you don't know one for ...
3
votes
2answers
473 views
Masking your web scraping activities to look like normal browser surfing activities?
I'm using the Html Agility Pack and I keep getting this error. "The remote server returned an error: (500) Internal Server Error." on certain pages.
Now I'm not sure what this is, as I can use ...
3
votes
1answer
4k views
ASP HttpWebRequest and Redirect
OK, I have a client doing a POST to a server with some data. The server receives the post, and answers with a redirect. The problem is that the client does not redirects. Also, I've tried to check the ...
3
votes
1answer
132 views
Html agility cant get the results
I have this list of URLs on following webpage, I am wondering how do I grab the URLs and add them to a ArrayList?
http://www.manta.com/mb?search=U.S.+Cellular&refine_company_loctype=B
I tried ...
3
votes
3answers
370 views
Perform Web Data Extraction
I am looking to get Today's prices from link
I have installed HTMLAgilityPack, but I cannot grasp once capturing the document table how to extracct the row whose first td element contains todays date ...
2
votes
2answers
749 views
C# Form freeze when processing information.
I wrote a personal web scrapper for myself that scraps artist information. the code works but when I press the button and it start processing the while loop, the GUI freezes. I got the textBoxes to ...
