Skip to content
#

punctuation

Here are 35 public repositories matching this topic...

Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. Ucto comes with tokenisation rules for several languages and can be easily extended to suit other languages. It has been incorporated for tokenizing Dutch text in Frog, our Dutch morpho-syntactic processor. http://ilk.uvt.nl/ucto --

  • Updated Jul 12, 2021
  • C++

Improve this page

Add a description, image, and links to the punctuation topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the punctuation topic, visit your repo's landing page and select "manage topics."

Learn more