5 Commits

Author	SHA1	Message	Date
John Mai	66d9202360	feat: Qwen2 support	2024-03-03 22:26:28 +08:00
David Koski	7b746cb89c	allow alternate location for tokenizer	2024-03-01 23:27:03 -08:00
David Koski	82f6a969d4	llm improvements ... - document the tokenizer used (https://github.com/huggingface/swift-transformers) - provide a hook for tokenizer configuration, prompt augmentation - this isn't as rich as the python equivalents but it helps a little	2024-03-01 14:46:32 -08:00
David Koski	3f02fcc1cb	expose eosToken	2024-02-26 14:58:51 -08:00
David Koski	4fad86d84b	split tokenizer code out into new file	2024-02-26 14:42:40 -08:00