tiktoken error on AWS lambda

[vicky141998]

I am using tiktoken in my project but when I deployed my project into AWS lambda I got the below error.
Unknown encoding cl100k_base. Plugins found: ['tiktoken_ext.pycache', 'tiktoken_ext.openai_public']. how to solve this

[dtschleckser]

How is your Lambda layer set up? This looks like it could be an issue with your layer packaging & the way tiktoken scans for encodings. Can you try checking tiktoken_ext/openai_public to make sure ENCODING_CONSTRUCTORS is present and defines cl100k_base as you expect?

[vicky141998]

Hello Daniel Thanks for replying, I checked that file. cl100k_base is already present.

…

On Tue, Sep 26, 2023 at 9:52 AM Daniel Schleckser ***@***.***> wrote: How is your Lambda layer set up? This looks like it could be an issue with your layer packaging & the way tiktoken scans for encodings. Can you try checking tiktoken_ext/openai_public to make sure ENCODING_CONSTRUCTORS is present and defines cl100k_base as you expect? — Reply to this email directly, view it on GitHub <#198 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A52RQRUEDGJM3T2GUVX25PLX4JJ6VANCNFSM6AAAAAA5BB7QII> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Regards, Sales*f*orce Developer Dreamwares IT Solution *Vicky Mahale* +91 9028881205 [image: ***@***.*** ***@***.***> [image: phone-icon]https://dreamwares.com/ [image: phone-icon]Nashik, MH. (India)

[azmathr]

Did you fix it? Same issue

[vicky141998]

Not Solved

…

On Wed, Feb 14, 2024 at 11:51 AM azmathr ***@***.***> wrote: Did you fix it? Same issue — Reply to this email directly, view it on GitHub <#198 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A52RQRQCYE4DYFGTO2JFWYTYTRJWVAVCNFSM6AAAAAA5BB7QIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBTGE2TCMZWGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Regards, Sales*f*orce Developer Dreamwares IT Solution *Vicky Mahale* +91 9028881205 [image: ***@***.*** ***@***.***> [image: phone-icon]https://dreamwares.com/ [image: phone-icon]Nashik, MH. (India)

[azmathr]

@dtschleckser I get error in lambda while deploying through zappa

File "/var/task/summaryserver.py", line 13, in <module>
   encoding = tiktoken.get_encoding("cl100k_base")
 File "/var/task/tiktoken/registry.py", line 64, in get_encoding
   _find_constructors()
 File "/var/task/tiktoken/registry.py", line 44, in _find_constructors