tokenizer
A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.
Here are 462 public repositories matching this topic...
-
Updated
May 14, 2018 - Swift
-
Updated
May 18, 2020 - Python
안녕하세요 Noun extractor을 잘 사용하고 있는 학생입니다!
다름 아니라 사용 중에 의문이 하나 들어서 질문 드리게 되었습니다.
input으로 사용하는 doublespace txt 파일의 sentence length가 얼마가 되어야 많은 범위의 어절을 커버하게 되나요?
제가 몇가지 샘플을 만들어서 사용해 보았는데, 인풋 데이터가 적으면 적을수록 명사를 잘 못 뽑는 것 같습니다. (비지도학습 기반의 모델이라 당연하지만요 ㅎㅎ)
예를 들어서, num sentence가 약 1만개일 경우 50~55%의 어절이 커버되었다고 출력됩니다.
[Noun Extractor] 54.52 % eojeols are covered
num sentence가 약 10만개일 경우 60~65%의 어절이 커버되었다
-
Updated
Mar 8, 2020 - Go
-
Updated
Aug 7, 2018 - Scala
-
Updated
Jun 10, 2020 - Python
Prepare the parser to PHP 8 :
Work In Progress as RFC are not yet closed
Not sure if this is the right place for it, but I want people searching for it to find it.
Since I wanted one, I made a playground for Moo: https://ablingeroscar.github.io/moo-playground/ (github link)
It's not especially pretty, but it works.
Would it be possible to have the regex parser support character classes like \w within other character classes? I had a regex pattern earlier that used the character class [0-9a-zA-Z_\.-], and I attempted to simplify it with [\w\.\-]. I didn't notice this library doesn't support doing that, and was wondering just how difficult that would be to implement. For the time being i'm just expanding
The python lib pragmatic_segmenter has a list of 50+ sentence split examples that this lib fails to parse. You can use their list to test this lib.
For example:
He left the bank at 6 P.M. Mr. Smith then went to the store.
Which neurosnap/sentences
-
Updated
Mar 18, 2020 - C
-
Updated
May 26, 2020 - Java
-
Updated
May 4, 2020 - Python
For Juman++ to be widely usable, we want to have a documented and stable C API and an option to have a dynamically linked library.
That library probably should use -fvisibility=hidden and explicit visibility on exported symbols on Unixes and __declspec(dllimport/dllexport) on Windows.
The minimal API should be:
- Loading a model using a config file
- Analyzing a sentence
- Accessing
-
Updated
Dec 16, 2019 - R
note to myself
add in docs that cooccurrence.data.frame in a group by fashion which does not take into account a sequence
does not return self-occurrences and as there is no order (bag of terms) in the output term1 is always smaller than term2, need to formulate this more concisely
while cooccurrence.character goes left to right, maybe need an option right to left also
Note in Biterm Topic Modelling (https:/
After pull request #170 c++ document aligner compilation fails with the following error:
[ 44%] Linking CXX executable bin/ngram_test
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/Scrt1.o: in function `_start':
(.text+0x24): undefined reference to `main'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/ngram_test.dir/build.make:163: bin/ngram_t
-
Updated
May 25, 2018 - JavaScript
-
Updated
May 11, 2020 - PHP
-
Updated
Mar 25, 2019 - Python
-
Updated
Dec 13, 2018 - Python
-
Updated
May 22, 2020 - Python
- Wikipedia
- Wikipedia
Everything in diagrams.css should be scoped to some wrapping css class, because as is it cannot be bundled with the rest of an app's css because of styles like this:
Curre