David Koski 82f6a969d4 llm improvements ...

- document the tokenizer used (https://github.com/huggingface/swift-transformers)
- provide a hook for tokenizer configuration, prompt augmentation
	- this isn't as rich as the python equivalents but it helps a little

2024-03-01 14:46:32 -08:00

1.9 KiB

Raw Blame History

View Raw