David Koski 0af2efe8a0 switch swift-tokenizers to main, remove some workarounds (#26) ...

* switch swift-tokenizers to main, remove some workarounds

- swift-tokenizers is getting a lot of updates and fixes, let's track main for now
- remove some workarounds that are no longer needed

- https://github.com/huggingface/swift-transformers/issues/63

2024-03-14 13:18:57 -07:00

99 KiB

Raw Blame History

View Raw