current community

your communities

more stack exchange communities

company blog
Stack Exchange Inbox Reputation and Badges
tour help
Dismiss
Announcing Stack Overflow Documentation

We started with Q&A. Technical documentation is next, and we need your help.

Whether you're a beginner or an experienced developer, you can contribute.

Sign up and start helping → Learn more about Documentation →
up vote 2 down vote favorite

I have a large data set of urls and I need a way to parse words from the urls eg:

realestatesales.com -> {"real","estate","sales"}

I would prefer to do it in python. This seems like it should be possible with some kind of english language dictionary. There might be some ambiguous cases, but I feel like there should be a solution out there somewhere.

share|improve this question
1  
What about words that are NOT in the dictionary, eg. imgur.com? – Achrome Jun 13 '13 at 17:28
    
More ambiguity than you would think... realestatesales – Andrew Clark Jun 13 '13 at 17:29
    
I can deal with some ambiguity. Maybe a good system would come up with the most likely parsing.... – user1893354 Jun 13 '13 at 17:32
up vote 2 down vote accepted

This is a problem is word segmentation, and an efficient dynamic programming solution exists. This page discusses how you could implement it. I have also answered this question on SO before, but I can't find a link to the answer. Please feel free to edit my post if you do.

share|improve this answer
up vote 3 down vote

Ternary Search Trees when filled with a word-dictionary can find the most-complex set of matched terms (words) rather efficiently. This is the solution I've previously used.
You can get a C/Python implementation of a tst here: http://github.com/nlehuen/pytst

Example: 

import tst
tree = tst.TST()
#note that tst.ListAction() assigns each matched term to a list
words = tree.scan("MultipleWordString", tst.ListAction())

Other Resources:

The open-source search engine called "Solr" uses what it calls a "Word-Boundary-Filter" to deal with this problem you might want to have a look at it.

share|improve this answer
up vote 2 down vote

This might be of use to you: http://www.clips.ua.ac.be/pattern

It's a set of modules which, depending on your system, might already be installed. It does all kinds of interesting stuff, and even if it doesn't do exactly what you need it might get you started on the right path.

share|improve this answer
    
any particular module you would recommend? – user1893354 Jun 13 '13 at 17:35
    
Off the top of my head you could combine a Wordlist with a generated synset from WordNet using the respective modules from here: clips.ua.ac.be/pages/pattern-en then search for substrings (words) potentially included in your superstrings (URLs). This method would not be time-efficient, but the concepts it requires might help you seek a better solution. – Rune Jun 13 '13 at 17:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.