Commit Graph

4 Commits

Author SHA1 Message Date
nai-degen c8d8e2e58f avoids instantiating new Claude tiktoken on every call 2023-08-08 17:38:03 -05:00
nai-degen d1d83b41fa uses accurate Claude tokenization 2023-08-08 17:29:36 -05:00
nai-degen d29c304d5a increases tokenizer failsafe to 500000 characters 2023-07-27 15:21:06 -05:00
khanon 56a4902599 Add tokenizers and configurable context size limits (khanon/oai-reverse-proxy!28) 2023-07-22 00:11:32 +00:00