aboutsummaryrefslogtreecommitdiff
path: root/v2/tokenizer_test.go
AgeCommit message (Expand)Author
2020-11-13Change the public API to use []byte instead of string.Bill Neubauer
2020-11-13Scope use of phrase induction to fix some bugs.Bill Neubauer
2020-11-13Needed to make a change to number tokenization to resolve an issueBill Neubauer
2020-11-13The tokenizer for the new version of the licenseclassifier.Bill Neubauer