small chat feature

[not-nullptr]

would be super cute honestly !! it already has a tuned in counter and a websocket server running already. would be happy to implement this if i knew it was gonna be merged :3

[kennethnym]

that sounds like a really cool idea!! my only worry is that it's gonna be ruined by spam and the likes :(

[not-nullptr]

hmm.. well it's already running inference for MusicLM which is notoriously hard to run already. does the server have enough VRAM left for a small LLM for sentiment anaylsis, maybe gemma2:2b which is insanely small for the quality? https://ollama.com/library/gemma2:2b

[kennethnym]

i have 6gb of vram left, should be able to run a small LLM? but the gpu is basically always at 100% usage due to it constantly churning out new clips, so i don't know if it can handle another llm. i definitely CANNOT afford to spin up another gpu 😭

[not-nullptr]

i've just looked into it, running an LLM probably isn't worth it. after some local testing they're way too overbearing to censor regular conversation. might just be worth having an IP rate limit

[kennethnym]

hmm i see, i will keep this opened for now, i do want to implement this in v2, but main focus of v2 right now is a fine-tuned model + dynamic prompt generation for more variety. thank u for this cute suggestion though!

feel free to drop a PR if u want, but again the model goes first, i will review it once the model is trained :)

[aryanranderiya]

Hey you could look into Cloudflare Workers AI. You get like a 100k requests a day for free and maybe it's something that'll work out of the box? Not for fine-tuning tho