Switch to the treebank tokenizer to cut a sentence into words.
PunktWordTokenizer is now considered as an internal part of punkt,
and not a public API class.
See https://github.com/nltk/nltk/commit/0b91a7160717faa2fe93d42f7c6bba735f6dd48a
Details
Details
- Reviewers
dereckson - Commits
- rEPNb8669b258fa5: Update word tokenizer
Tested with Ada Palmer, the Will to Battle
Diff Detail
Diff Detail
- Lint
No Lint Coverage - Unit
No Test Coverage - Branch
- update-tokenizer (branched from master)
- Build Status
Buildable 2132 Build 2380: arc lint + arc unit