3 Commits

Author	SHA1	Message	Date
David Koski	82f6a969d4	llm improvements ... - document the tokenizer used (https://github.com/huggingface/swift-transformers) - provide a hook for tokenizer configuration, prompt augmentation - this isn't as rich as the python equivalents but it helps a little	2024-03-01 14:46:32 -08:00
David Koski	3f02fcc1cb	expose eosToken	2024-02-26 14:58:51 -08:00
David Koski	4fad86d84b	split tokenizer code out into new file	2024-02-26 14:42:40 -08:00