David Koski
4fad86d84b
split tokenizer code out into new file
2024-02-26 14:42:40 -08:00
David Koski
c7919cf7fe
fix rmsnorm for gemma
2024-02-26 14:09:48 -08:00
David Koski
a2ff291608
add reference to filed issue
2024-02-26 13:31:33 -08:00
David Koski
c86d1c195e
partial fix for #1
...
- handle loading models with different names for the safetensors files (gemma)
- handle merge tokens that can't be split
- organize code into Load/Evaluate
2024-02-26 13:23:21 -08:00
David Koski
bb7bacc077
fix for #2 -- CodeLlama crashes
...
- add replacement tokenizer class for unknown tokenizers
- fix quantization for models that don't have lm_head quantized
Requires https://github.com/ml-explore/mlx-swift/pull/28
2024-02-26 10:38:05 -08:00
David Koski
5a83d7d92b
fix broken links, clarify documentation
2024-02-22 12:46:44 -08:00
David Koski
b6d1e14465
initial commit
2024-02-22 10:41:02 -08:00
