aboutsummaryrefslogtreecommitdiff
path: root/v2/tokenizer_test.go
AgeCommit message (Expand)Author
2022-09-16Make the public facing API be implemented in terms of io.Reader rather thanBill Neubauer
2022-09-16Rewrite the tokenization process to work on streams rather than requiring theBill Neubauer
2022-09-16Removing the Index field from the token structures.Bill Neubauer
2022-09-16Adds Copyright detection to the report generated by the classifier.Bill Neubauer
2020-11-13Change the public API to use []byte instead of string.Bill Neubauer
2020-11-13Scope use of phrase induction to fix some bugs.Bill Neubauer
2020-11-13Needed to make a change to number tokenization to resolve an issueBill Neubauer
2020-11-13The tokenizer for the new version of the licenseclassifier.Bill Neubauer