WorksApplications/SudachiPy
Python version of Sudachi, a Japanese tokenizer.
kazuma-t·4y ago·September 27, 2021📋 Changes
- When multiple user dictionaries with user-defined parts of speech are used, the user-defined POS IDs of the second and subsequent user dictionaries become invalid (`IndexError: list index out of range`)
kazuma-t·4y ago·September 10, 2021📋 Changes
- Words containing digits cannot be properly registered in split information
- Slow to build user dictionary
- Some katakana words are analyzed as OOV
kazuma-t·5y ago·March 26, 2021📋 Changes
- Added option -s to specify dictionary type
- Added argument to Dictionary class to specify dictionary type
- Removed the option to create a link
t-yamamura·5y ago·January 4, 2021📋 Changes
- https://github.com/WorksApplications/SudachiPy/pull/151 Fix `-a` option (`print all of the fields`) (Error reported in https://github.com/WorksApplications/SudachiPy/issues/150)
kazuma-t·5y ago·December 25, 2020Support for new dictionary format with synonym group IDs
sorami·5y ago·June 19, 2020📋 Changes
- #134 Fix Morphemelist split (Error reported in #133)
sorami·5y ago·June 18, 2020📋 Changes
- #131 Fix connection cost lookup (spaCy accuracy degradation reported in #129)
- #130 Fix `Lattice.dump` error (Reported in #128)
sorami·6y ago·June 15, 2020📋 Changes
- #127 Don't explicitly release the memoryview in Grammar
sorami·6y ago·June 10, 2020📋 Changes
- #123 Cython based optimization
- #124 Add `__str__` functions for Moprheme, MorphemeList
sorami·6y ago·June 2, 2020📋 Changes
- #121 Fix a bug causing … is converted to "", "", "…"
- #122 Improve error messages related to dictionary setup
sorami·6y ago·April 30, 2020Speed up execution by re-using unk info #117
sorami·6y ago·February 26, 2020Upgrade a dependent Python library `dartsclone` ([rixwew/darts-clone-python](https://github.com/rixwew/darts-clone-python)) to `v0.9`.
sorami·6y ago·December 6, 2019Fix the runtime error `EOS isn't connected to BOS` when input contains `阿Q[a-zA-Z]+`. Equivalent to [Fix #45 by kazuma-t · Pull Request #113 · WorksApplications/Sudachi](https://github.com/WorksApplications/Sudachi/pull/113) in the original Java Sudachi.
sorami·6y ago·November 26, 2019📋 Changes
- Proper reading forms for Hiragana and Katakana words #104
- Proper OOV flag #106
izziiyt·6y ago·September 7, 2019cythonize dartsclone using https://github.com/rixwew/darts-clone-python
izziiyt·6y ago·July 21, 2019In README.md > If you need to apply customized `system.dic`, place [sudachi.json](https://github.com/WorksApplications/Sudachi/blob/develop/src/main/resources/sudachi.json) to anywhere you like, and overwrite `systemDict` value with the relative path from `sudachi.json` to your `system.dic`. not worked properly but fixed. This change maybe **incompatible** if you use your defined `sudachi.json`.
izziiyt·6y ago·July 7, 2019📋 Changes
- delete default dictionary utility, because of PyPI dependency
izziiyt·6y ago·July 7, 2019📋 Changes
- link command, utility to manage dictionary
- default dictionary, improve utility to install SudachiPy
- `/resources` moved into `/sudachipy`
- `tests` module independent from `/resources`
kazuma-t·6y ago·July 5, 2019This release is just for fixing the packaging issue. It does not contain any additional bug fixes from 0.2.0.
izziiyt·6y ago·July 4, 2019📋 Changes
- interface changed !! see README
- can use user dictionary
- dictionary build command
- no test need real system.dic
- apply some format rules
izziiyt·6y ago·June 19, 2019📋 Changes
- parsing character definition file
- parsing dictionary file
- other minor bugs
- add tests following Sudachi-Java
- add CI system
- introduce formatter