Token limit ignored

[mwinkens]

Which version of assistant are you using?

latest

Which version of Nextcloud are you using?

v28.0.4

Which browser are you using? In case you are using the phone App, specify the Android or iOS version and device please.

Firefox, Chromium

Describe the Bug

I set New Token limit to 10000, later to 1000, this was ignored (OpenAI & LocalAI integration setting)

Client error: `POST https://<localaiaddress>:8000/v1/chat/completions` resulted in a `400 Bad Request` response: {"object":"error","message":"This model's maximum context length is 32768 tokens. However, your messages resulted in 999 (truncated...)
API request error : Client error: `POST https://<localaiaddress>:8000/v1/chat/completions` resulted in a `400 Bad Request` response: {"object":"error","message":"This model's maximum context length is 32768 tokens. However, your messages resulted in 999 (truncated...) 

Expected Behavior

Cut off tokens or result in a proper error message, not simply in "Assistant failed"

To Reproduce

Send big file to local AI

[julien-nc]

LocalAI is complaining about the size of the input. The Max new tokens per request parameter in the LocalAI admin settings are limiting the number of produced tokens.

I don't know if there is a way to know the input token limit for each available models in LocalAI. In this case we could use that value to check the input even before sending a request.

[tomchiverton]

I can't even save this setting change e.g. to lower from 1000 "max new tokens" to 200, because a toast pops up saying "Failed to save OpenAI admin options: "Invalid type for key: max_tokens. Expected integer, got string"

This seems to occur for any numeric entry on the page e.g. timeout setting.

Firefox 125.0.3, Nextcloud 29.0.0, Assistant 1.0.9

[alekstos]

If you make a request manually with curl or PowerShell and remove " like here:

`"max_tokens`":4096 

instead of original

`"max_tokens`":`"4096`"

, all works fine.

>> -Body "{`"values`":{`"use_basic_auth`":false,`"api_key`":`"`",`"basic_user`":`"`",`"basic_password`":`"`",`"url`":`"http://openai.server`",`"service_name`":`"Openchat`",`"chat_endpoint_enabled`":false,`"request_timeout`":90000,`"max_tokens`":4096,`"llm_extra_params`":`"`",`"quota_period`":30,`"quotas`":[0,0,0]}}"


StatusCode        : 200
StatusDescription : OK

The problem is clearly in the formation of parameters in the source code.

[tomchiverton]

Still broken in 29.0.2