current community

your communities

more stack exchange communities

company blog
Stack Exchange Inbox Reputation and Badges
tour help
Stack Overflow is a community of 4.7 million programmers, just like you, helping each other.

Join them; it only takes a minute:

Sign up
Join the Stack Overflow community to:
  1. Ask programming questions
  2. Answer and help your peers
  3. Get recognized for your expertise
up vote 0 down vote favorite

I have a dataset of links to newspaper articles that I want to do some research on. However, the links in the dataset end with .ece extension (which is a problem for me because of some api restrictions)

http://www.telegraaf.nl/telesport/voetbal/buitenlands/article22178882.ece

and

http://www.telegraaf.nl/telesport/voetbal/buitenlands/22178882/__Wenger_vreest_het_ergste__.html

are links to the same page. Now I need to convert all the .ece links into .html links. I didn't find an easier way to do it, but to parse the page and find the original .html link. The problem is that the link is buried inside an html meta element, and I can't get to it using tree.xpath.

<meta content="http://www.telegraaf.nl/telesport/voetbal/buitenlands/22178882/__Wenger_vreest_het_ergste__.html"

Unfortunately, I am not well acquainted with regex, and don't know how to extract a link using it. Basically, every link I need will starts with:

<meta content="http://www.telegraaf.nl/

I need the full link (i.e., http://www.telegraaf.nl/THE_REST_OF_THE_LINK). Also, I'm using BeautifulSoup for parsing. Thanks.

share|improve this question
    
Do you at least know how to use regex if the regex string was given to you? – Nick Humrich Sep 29 '14 at 17:28
    
well, I know that I'd have to use the re module. re.findall (r"expression", "string")? – Zlo Sep 29 '14 at 17:42
    
how are you getting the links? – Padraic Cunningham Sep 29 '14 at 17:43
    
@Padraic Cunningham, I have a file exclusively with .ece links. When I opened the page's source code, I've found that they store the .html link in the meta element. What I need is to get that link (.html) from the source code. – Zlo Sep 29 '14 at 17:48
    
@Zlo I can think you can probably adapt my answer then... I've assumed they're both in the the same file... but you can tweak it to how you want... – Jon Clements♦ Sep 29 '14 at 17:52
up vote 1 down vote accepted

Here is a really simple regex to get you started.

This one will extract all links

\<meta content="(http:\/\/www\.telegraaf\.nl.*)"

This one will match all html links

\<meta content="(http:\/\/www\.telegraaf\.nl.*\.html)"

To use this with what you have, you can do the following:

import urllib2
import re

replacements = dict()
for url in ece_url_list:
    response = urllib2.urlopen(url)
    html = response.read()
    replacements[url] = re.findall('\<meta content="(http:\/\/www\.telegraaf\.nl.*\.html)"', html)[0]

Note: This assumes that each source code page always includes an html link in this meta tag. It expects one and only one.

share|improve this answer
    
Thanks, this is what I needed! – Zlo Sep 29 '14 at 19:22
up vote 1 down vote

Use BeautifulSoup to find matching content attributes, then replace as such:

from bs4 import BeautifulSoup
import re

html = """
    <meta content="http://www.telegraaf.nl/telesport/voetbal/buitenlands/article22178882.ece" />
    <meta content="http://www.telegraaf.nl/telesport/voetbal/buitenlands/22178882/__Wenger_vreest_het_ergste__.html" />
"""

soup = BeautifulSoup(html)
# reference table of url prefixes to full html link
html_links = {
    el['content'].rpartition('/')[0]: el['content'] 
    for el in soup.find_all('meta', content=re.compile('.html$'))
}
# find all ece links, strip the end of to match links, then adjust
# meta content with looked up element
for el in soup.find_all('meta', content=re.compile('.ece$')):
    url = re.sub('(?:article(\d+).ece$)', r'\1', el['content'])
    el['content'] = html_links[url]

print soup
# <meta content="http://www.telegraaf.nl/telesport/voetbal/buitenlands/22178882/__Wenger_vreest_het_ergste__.html"/>
share|improve this answer
up vote 0 down vote 
(.*?)(http:\/\/.*\/.*?\.)(ece)

Try this.Replace by $2html.

See demo.

http://regex101.com/r/nA6hN9/24

share|improve this answer
    
I dont think that will work. The OP shows that the html links are different then the ece links. They are not just .html. In other words, a simple find replace on .ece would not work. He needs to replace the whole link, with the whole second link. – Nick Humrich Sep 29 '14 at 17:24
    
@Humdinger m lost now .will wait for clarification from OP – vks Sep 29 '14 at 17:26
    
@Humdinger, that's correct. The .html link is a bit different than the .ece one, so a simple replacement doesn't work. – Zlo Sep 29 '14 at 17:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.