Page MenuHomeDevCentral

Update word tokenizer
ClosedPublic

Authored by dereckson on Feb 27 2018, 21:19.
Tags
None
Referenced Files
Unknown Object (File)
Fri, Apr 11, 08:49
Unknown Object (File)
Thu, Apr 10, 19:35
Unknown Object (File)
Thu, Apr 10, 19:27
Unknown Object (File)
Thu, Apr 10, 18:53
Unknown Object (File)
Sun, Apr 6, 17:43
Unknown Object (File)
Sat, Apr 5, 23:30
Unknown Object (File)
Mar 22 2025, 12:20
Unknown Object (File)
Mar 21 2025, 23:07
Subscribers
None

Details

Summary

Switch to the treebank tokenizer to cut a sentence into words.

PunktWordTokenizer is now considered as an internal part of punkt,
and not a public API class.

See https://github.com/nltk/nltk/commit/0b91a7160717faa2fe93d42f7c6bba735f6dd48a

Test Plan

Tested with Ada Palmer, the Will to Battle

Diff Detail

Lint
No Lint Coverage
Unit
No Test Coverage
Branch
update-tokenizer (branched from master)
Build Status
Buildable 2132
Build 2380: arc lint + arc unit

Event Timeline

dereckson created this revision.
This revision is now accepted and ready to land.Feb 27 2018, 21:20