8 Commits

Author	SHA1	Message	Date
Anchen	63531bc5fa	feat: add command r model support (#35) ... * feat: add command r model support	2024-03-26 12:43:10 -07:00
Pedro Cuenca	6c270a9d12	Remove Tokenizer wrapper (#28) ... * Remove Tokenizer wrapper Depends on https://github.com/huggingface/swift-transformers/pull/70 * Update swift-transformers revision	2024-03-18 09:07:52 -07:00
David Koski	0af2efe8a0	switch swift-tokenizers to main, remove some workarounds (#26) ... * switch swift-tokenizers to main, remove some workarounds - swift-tokenizers is getting a lot of updates and fixes, let's track main for now - remove some workarounds that are no longer needed - https://github.com/huggingface/swift-transformers/issues/63	2024-03-14 13:18:57 -07:00
John Mai	66d9202360	feat: Qwen2 support	2024-03-03 22:26:28 +08:00
David Koski	7b746cb89c	allow alternate location for tokenizer	2024-03-01 23:27:03 -08:00
David Koski	82f6a969d4	llm improvements ... - document the tokenizer used (https://github.com/huggingface/swift-transformers) - provide a hook for tokenizer configuration, prompt augmentation - this isn't as rich as the python equivalents but it helps a little	2024-03-01 14:46:32 -08:00
David Koski	3f02fcc1cb	expose eosToken	2024-02-26 14:58:51 -08:00
David Koski	4fad86d84b	split tokenizer code out into new file	2024-02-26 14:42:40 -08:00