3 Commits

Author	SHA1	Message	Date
David Koski	4fad86d84b	split tokenizer code out into new file	2024-02-26 14:42:40 -08:00
David Koski	a2ff291608	add reference to filed issue	2024-02-26 13:31:33 -08:00
David Koski	c86d1c195e	partial fix for #1 ... - handle loading models with different names for the safetensors files (gemma) - handle merge tokens that can't be split - organize code into Load/Evaluate	2024-02-26 13:23:21 -08:00