Commit Graph

1 Commits

Author SHA1 Message Date
David Koski
82f6a969d4 llm improvements
- document the tokenizer used (https://github.com/huggingface/swift-transformers)
- provide a hook for tokenizer configuration, prompt augmentation
	- this isn't as rich as the python equivalents but it helps a little
2024-03-01 14:46:32 -08:00