Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.Net: ONNX 0.5.0 Add Ogahandle resource managment to Service [WIP] #9644

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

RogerBarreto
Copy link
Member

Motivation and Context

@RogerBarreto RogerBarreto requested a review from a team as a code owner November 11, 2024 17:40
@markwallace-microsoft markwallace-microsoft added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel labels Nov 11, 2024
@f2bo
Copy link

f2bo commented Nov 11, 2024

Doesn't this change mean that the model is reloaded into memory every time that you run an inference? This might take several seconds each time.

On the other hand, isn't having multiple OgaHandles, one for each service instance, a problem? It's not guaranteed that you will only have a singleton instance and disposing of one of the ONNX services will shut down the ONNX runtime for the process, so that all other services become invalid.
On the other hand, isn't creating multiple OgaHandles, one for each invocation of the service, a problem? The handle is disposed after the first invocation, which will shut down the ONNX runtime for the process and as a result, any subsequent invocations should fail.

@RogerBarreto
Copy link
Member Author

@f2bo This won't be merged as is, I will add some extra Unit Testing to ensure it doesn't affect the status-quo, for the moment recommend using the ONNX Connector with previous version of GenAI.

@RogerBarreto RogerBarreto marked this pull request as draft November 11, 2024 23:07
@RogerBarreto RogerBarreto changed the title .Net: ONNX 0.5.0 Add Ogahandle resource managment to Service. .Net: ONNX 0.5.0 Add Ogahandle resource managment to Service [WIP] Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kernel Issues or pull requests impacting the core kernel .NET Issue or Pull requests regarding .NET code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

.Net: Bug: Application crashes when using the ONNX connector and OnnxRuntimeGenAI v0.5.0 package
4 participants