ucto
Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. Ucto comes with tokenisation rules for several languages and can be easily extended to suit other languages. It has been incorporated for tokenizing Dutch text in Frog, our Dutch morpho-syntactic processor. http://ilk.uvt.nl/ucto --
How to download and setup ucto
Open terminal and run command
git clone https://github.com/LanguageMachines/ucto.git
git clone is used to create a copy or clone of ucto repositories.
You pass git clone a repository URL. it supports a few different network protocols and corresponding URL formats.
Also you may download zip file with ucto https://github.com/LanguageMachines/ucto/archive/master.zip
Or simply clone ucto with SSH
[email protected]:LanguageMachines/ucto.git
If you have some problems with ucto
You may open issue on ucto support forum (system) here: https://github.com/LanguageMachines/ucto/issuesSimilar to ucto repositories
Here you may see ucto alternatives and analogs
TypeScript ruby pyret-lang go awesome-cheatsheets android-dev-sources natural-language-processing lectures spaCy HanLP gensim MatchZoo tensorflow-nlp Awesome-pytorch-list mimesis spacy-models CN-VScode-Docs oh Eve lang Repo-2017 stanford-tensorflow-tutorials rust Nim red scala-js haxe enso crystal ChatterBot