4 Commits

Include renames

Author	SHA1	Message	Date
David Koski	82f6a969d4	llm improvements ... - document the tokenizer used (https://github.com/huggingface/swift-transformers) - provide a hook for tokenizer configuration, prompt augmentation - this isn't as rich as the python equivalents but it helps a little	2024-03-01 14:46:32 -08:00
David Koski	4fad86d84b	split tokenizer code out into new file	2024-02-26 14:42:40 -08:00
David Koski	c86d1c195e	partial fix for #1 ... - handle loading models with different names for the safetensors files (gemma) - handle merge tokens that can't be split - organize code into Load/Evaluate	2024-02-26 13:23:21 -08:00
David Koski	b6d1e14465	initial commit	2024-02-22 10:41:02 -08:00