Package edu.tufts.hrilab.llm
Class Tokenizer
java.lang.Object
edu.tufts.hrilab.llm.Tokenizer
-
Constructor Summary
Constructors -
Method Summary
-
Constructor Details
-
Tokenizer
public Tokenizer()
-
-
Method Details
-
setEncoding
-
encode
-
count
-