Unsupervised text tokenizer focused on computational efficiency

Ravencoin Core integration/staging tree

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.

👑 spaCy building blocks and visualizers for Streamlit apps

Rule-based token, sentence segmentation for Russian language

Fast and customizable text tokenization library with BPE and SentencePiece support

Simple NLP in Rust with Python bindings

Language Modeling and Text Classification in Malayalam Language using ULMFiT

An unofficial Sudachi clone in Rust (incomplete) 🦀

Collection of Wongnai's datasets

High performance tokenizers for natural language processing and other related tasks

Natural Language Processing Toolkit in Golang

Multi Cloud Data Tokenization Solution By Using Dataflow and Cloud DLP

🦜 Containerized HTTP API for industial-strength NLP via spaCy and sense2vec

Tokenize, encrypt/decrypt, mask your data on the fly with Vaulty proxy

Multilingual tokenizer that automatically tags each token with its type

POS Tagger, lemmatizer and stemmer for french language in javascript

Rosette API Client Library for Python

Simple and customizable text tokenization gem.

Smart Language Model

Custom Russian tokenizer for spaCy

A Fast and Accurate Neural Thai Word Segmenter

The Unicode Cookbook for Linguists

A tokenizer based on Unicode text segmentation (UAX 29), for Go