-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
.Net: ONNX 0.5.0 Add Ogahandle resource managment to Service [WIP] #9644
base: main
Are you sure you want to change the base?
.Net: ONNX 0.5.0 Add Ogahandle resource managment to Service [WIP] #9644
Conversation
Doesn't this change mean that the model is reloaded into memory every time that you run an inference? This might take several seconds each time.
|
@f2bo This won't be merged as is, I will add some extra Unit Testing to ensure it doesn't affect the status-quo, for the moment recommend using the |
Motivation and Context
The Latest 0.5.0 package also requires the caller to handle the resources with the
OgaHandler
instance, when a service is instantiated this resource needs to be also present and exposed together with the service. Otherwise a message will be sent to the console and the application will crash before finishing.Resolves .Net: Bug: Application crashes when using the ONNX connector and OnnxRuntimeGenAI v0.5.0 package #9628