Page MenuHomeDevCentral

Update word tokenizer
ClosedPublic

Authored by dereckson on Feb 27 2018, 21:19.
Tags
None
Referenced Files
Unknown Object (File)
Sat, Mar 22, 12:20
Unknown Object (File)
Fri, Mar 21, 23:07
Unknown Object (File)
Thu, Mar 20, 00:23
Unknown Object (File)
Wed, Mar 19, 03:10
Unknown Object (File)
Sat, Mar 15, 10:42
Unknown Object (File)
Sat, Mar 15, 07:01
Unknown Object (File)
Feb 27 2025, 02:41
Unknown Object (File)
Jan 22 2025, 08:07
Subscribers
None

Details

Summary

Switch to the treebank tokenizer to cut a sentence into words.

PunktWordTokenizer is now considered as an internal part of punkt,
and not a public API class.

See https://github.com/nltk/nltk/commit/0b91a7160717faa2fe93d42f7c6bba735f6dd48a

Test Plan

Tested with Ada Palmer, the Will to Battle

Diff Detail

Lint
No Lint Coverage
Unit
No Test Coverage
Branch
update-tokenizer (branched from master)
Build Status
Buildable 2132
Build 2380: arc lint + arc unit

Event Timeline

dereckson created this revision.
This revision is now accepted and ready to land.Feb 27 2018, 21:20