-
Notifications
You must be signed in to change notification settings - Fork 3.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
.Net: Feat: adds support for Copilot Agent Plugins in dotnet and othe…
…r fixes. (#9436) fixes #6614 ### Motivation and Context - Adds support for Microsoft Plugins Manifest to semantic kernel in dotnet. - Fixes performance bottleneck for API Manifest loading. - Fixes broken integration tests for API manifest loading. - Adds an OpenAPI operation id normalization visitor to replace `.` by `_` so function names are valid. - Fixes performance bottleneck in OpenAPI operation loading. - Fixes experimental ID for API manifest from SKEXP0043 to SKEXP0040 after the renumbering - Upgrades OAI.net and ApiManifest references. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 On the unit tests: I'd like guidance on where to add unit tests for: - API Manifest loading (wasn't done at the time) - Microsoft Manifest loading. - Operation Id Normalization Visitor ### Why so many files? - `./kiota`: all generated files, it contains kiota workspace configuration, which comes with copies of the OAS descriptions, etc… You don’t need to review it manually. It is useful because it allows us to automatically generate/refresh the integration tests plugins. If you feel like it's adding too much noise, we can remove those, we'll loose the ability to refresh the plugins definitions. https://aka.ms/kiota - `Samples/concepts`: restored the api manifest for astronomy API to fix the API manifest integration test. Added sample Microsoft Manifest for the new integration tests. Those are automatically generated via kiota and can be automatically refreshed later. - `Src/Functions`: Microsoft manifests implementation, API manifest fixes. ### How to run the local tests Create the following JSON file `D:\github\semantic-kernel\dotnet\samples\Concepts\bin\Debug\net8.0\appsettings.Development.json` (for some reason given how the project is setup this file is not being copied automatically. I didn't to touch any of the project setup out of fear of breaking other things) ```json { "MSGraph": { "ClientId": "clientId", "TenantId": "tenantId", "Scopes": [ "Calendars.Read", "Contacts.Read", "Files.Read.All", "Mail.Read", "User.Read" ], "RedirectUri": "http://localhost" } } ``` Replace the clientId and TenantId by your own value. To create the application registration, go to https://aad.portal.azure.com, create a new application registration, new public client (add the redirect URI). In API access, add the listed Microsoft Graph delegated scopes. Grant consent after adding the scopes. During the first run, your browser will open to get the token. ### File paths and copies Like for the development settings, the project is NOT copying the sample plugin files for some reason. This is why the loading of the files in the integration tests looks at source files directly with `../../../` path segments. Happy to review that upon guidance. ### License for Astrology plugins The description is under the Apache license. I added the plugin (API and Microsoft) to restore the integration test for the former and mirror the setup to the latter. In the case of API plugins, we're only referring to it, so having a plugin is fine. In case of the Microsoft plugin, we have a full copy under the kiota configuration directory, and a sliced down version (derived work) in the example plugin. The value of this API is that it allows us to test scenarios outside of Microsoft Graph. But if the license is a challenge, we can remove those before merging. @RogerBarreto to provide more context on why those were deleted at the first place in #6005 ### Why so many commits? Incremental work during the implementation, plus regular merges from main to make sure everything was current and we wouldn't end up with conflicts, etc... Happy to rebase and squash once the initial reviews are through. --------- Signed-off-by: Vincent Biret <[email protected]> Co-authored-by: Mustafa Zengin <[email protected]>
- Loading branch information
Showing
31 changed files
with
4,353 additions
and
75 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
157 changes: 157 additions & 0 deletions
157
dotnet/samples/Concepts/Plugins/CopilotAgentBasedPlugins.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| // Copyright (c) Microsoft. All rights reserved. | ||
|
|
||
| using System.Web; | ||
| using Microsoft.SemanticKernel; | ||
| using Microsoft.SemanticKernel.Plugins.MsGraph.Connectors.CredentialManagers; | ||
| using Microsoft.SemanticKernel.Plugins.OpenApi; | ||
| using Microsoft.SemanticKernel.Plugins.OpenApi.Extensions; | ||
|
|
||
| namespace Plugins; | ||
| /// <summary> | ||
| /// These examples demonstrate how to use Copilot Agent plugins to call Microsoft Graph and NASA APIs. | ||
| /// Copilot Agent Plugins are created from the OpenAPI document and the manifest file. | ||
| /// The manifest file contains the API dependencies and their execution parameters. | ||
| /// The manifest file also contains the authentication information for the APIs, however this is not used by the extension method and MUST be setup separately at the moment, which the example demonstrates. | ||
| /// | ||
| /// Important stages being demonstrated: | ||
| /// 1. Load Copilot Agent Plugins | ||
| /// 2. Configure authentication for the APIs | ||
| /// 3. Call functions from the loaded plugins | ||
| /// | ||
| /// Running this test requires the following configuration in `dotnet\samples\Concepts\bin\Debug\net8.0\appsettings.Development.json`: | ||
| /// | ||
| /// ```json | ||
| /// { | ||
| /// "MSGraph": { | ||
| /// "ClientId": "clientId", | ||
| /// "TenantId": "tenantId", | ||
| /// "Scopes": [ | ||
| /// "Calendars.Read", | ||
| /// "Contacts.Read", | ||
| /// "Files.Read.All", | ||
| /// "Mail.Read", | ||
| /// "User.Read" | ||
| /// ], | ||
| /// "RedirectUri": "http://localhost" | ||
| /// } | ||
| /// } | ||
| ///``` | ||
| /// | ||
| /// Replace the clientId and TenantId by your own values. | ||
| /// | ||
| /// To create the application registration: | ||
| /// 1. Go to https://aad.portal.azure.com | ||
| /// 2. Select create a new application registration | ||
| /// 3. Select new public client (add the redirect URI). | ||
| /// 4. Navigate to API access, add the listed Microsoft Graph delegated scopes. | ||
| /// 5. Grant consent after adding the scopes. | ||
| /// | ||
| /// During the first run, your browser will open to get the token. | ||
| /// | ||
| /// </summary> | ||
| /// <param name="output">The output helper to use to the test can emit status information</param> | ||
| public class CopilotAgentBasedPlugins(ITestOutputHelper output) : BaseTest(output) | ||
| { | ||
| public static readonly IEnumerable<object[]> s_parameters = | ||
| [ | ||
| // function names are sanitized operationIds from the OpenAPI document | ||
| ["MessagesPlugin", "me_ListMessages", new KernelArguments { { "_top", "1" } }, "MessagesPlugin"], | ||
| ["DriveItemPlugin", "drive_root_GetChildrenContent", new KernelArguments { { "driveItem-Id", "test.txt" } }, "DriveItemPlugin", "MessagesPlugin"], | ||
| ["ContactsPlugin", "me_ListContacts", new KernelArguments() { { "_count", "true" } }, "ContactsPlugin", "MessagesPlugin"], | ||
| ["CalendarPlugin", "me_calendar_ListEvents", new KernelArguments() { { "_top", "1" } }, "CalendarPlugin", "MessagesPlugin"], | ||
|
|
||
| // Multiple API dependencies (multiple auth requirements) scenario within the same plugin | ||
| // Graph API uses MSAL | ||
| ["AstronomyPlugin", "me_ListMessages", new KernelArguments { { "_top", "1" } }, "AstronomyPlugin"], | ||
| // Astronomy API uses API key authentication | ||
| ["AstronomyPlugin", "apod", new KernelArguments { { "_date", "2022-02-02" } }, "AstronomyPlugin"], | ||
| ]; | ||
| [Theory, MemberData(nameof(s_parameters))] | ||
| public async Task RunCopilotAgentPluginAsync(string pluginToTest, string functionToTest, KernelArguments? arguments, params string[] pluginsToLoad) | ||
| { | ||
| WriteSampleHeadingToConsole(pluginToTest, functionToTest, arguments, pluginsToLoad); | ||
| var kernel = new Kernel(); | ||
| await AddCopilotAgentPluginsAsync(kernel, pluginsToLoad); | ||
|
|
||
| var result = await kernel.InvokeAsync(pluginToTest, functionToTest, arguments); | ||
| Console.WriteLine("--------------------"); | ||
| Console.WriteLine($"\nResult:\n{result}\n"); | ||
| Console.WriteLine("--------------------"); | ||
| } | ||
|
|
||
| private void WriteSampleHeadingToConsole(string pluginToTest, string functionToTest, KernelArguments? arguments, params string[] pluginsToLoad) | ||
| { | ||
| Console.WriteLine(); | ||
| Console.WriteLine("======== [CopilotAgent Plugins Sample] ========"); | ||
| Console.WriteLine($"======== Loading Plugins: {string.Join(" ", pluginsToLoad)} ========"); | ||
| Console.WriteLine($"======== Calling Plugin Function: {pluginToTest}.{functionToTest} with parameters {arguments?.Select(x => x.Key + " = " + x.Value).Aggregate((x, y) => x + ", " + y)} ========"); | ||
| Console.WriteLine(); | ||
| } | ||
| private async Task AddCopilotAgentPluginsAsync(Kernel kernel, params string[] pluginNames) | ||
| { | ||
| #pragma warning disable SKEXP0050 | ||
| if (TestConfiguration.MSGraph.Scopes is null) | ||
| { | ||
| throw new InvalidOperationException("Missing Scopes configuration for Microsoft Graph API."); | ||
| } | ||
|
|
||
| LocalUserMSALCredentialManager credentialManager = await LocalUserMSALCredentialManager.CreateAsync().ConfigureAwait(false); | ||
|
|
||
| var token = await credentialManager.GetTokenAsync( | ||
| TestConfiguration.MSGraph.ClientId, | ||
| TestConfiguration.MSGraph.TenantId, | ||
| TestConfiguration.MSGraph.Scopes.ToArray(), | ||
| TestConfiguration.MSGraph.RedirectUri).ConfigureAwait(false); | ||
| #pragma warning restore SKEXP0050 | ||
|
|
||
| BearerAuthenticationProviderWithCancellationToken authenticationProvider = new(() => Task.FromResult(token)); | ||
| #pragma warning disable SKEXP0040 | ||
|
|
||
| // Microsoft Graph API execution parameters | ||
| var graphOpenApiFunctionExecutionParameters = new OpenApiFunctionExecutionParameters( | ||
| authCallback: authenticationProvider.AuthenticateRequestAsync, | ||
| serverUrlOverride: new Uri("https://graph.microsoft.com/v1.0")); | ||
|
|
||
| // NASA API execution parameters | ||
| var nasaOpenApiFunctionExecutionParameters = new OpenApiFunctionExecutionParameters( | ||
| authCallback: async (request, cancellationToken) => | ||
| { | ||
| var uriBuilder = new UriBuilder(request.RequestUri ?? throw new InvalidOperationException("The request URI is null.")); | ||
| var query = HttpUtility.ParseQueryString(uriBuilder.Query); | ||
| query["api_key"] = "DEMO_KEY"; | ||
| uriBuilder.Query = query.ToString(); | ||
| request.RequestUri = uriBuilder.Uri; | ||
| }); | ||
|
|
||
| var apiManifestPluginParameters = new CopilotAgentPluginParameters | ||
| { | ||
| FunctionExecutionParameters = new() | ||
| { | ||
| { "https://graph.microsoft.com/v1.0", graphOpenApiFunctionExecutionParameters }, | ||
| { "https://api.nasa.gov/planetary", nasaOpenApiFunctionExecutionParameters } | ||
| } | ||
| }; | ||
| var manifestLookupDirectory = Path.Combine(Directory.GetCurrentDirectory(), "..", "..", "..", "Resources", "Plugins", "CopilotAgentPlugins"); | ||
|
|
||
| foreach (var pluginName in pluginNames) | ||
| { | ||
| try | ||
| { | ||
| #pragma warning disable CA1308 // Normalize strings to uppercase | ||
| await kernel.ImportPluginFromCopilotAgentPluginAsync( | ||
| pluginName, | ||
| Path.Combine(manifestLookupDirectory, pluginName, $"{pluginName[..^6].ToLowerInvariant()}-apiplugin.json"), | ||
| apiManifestPluginParameters) | ||
| .ConfigureAwait(false); | ||
| #pragma warning restore CA1308 // Normalize strings to uppercase | ||
| Console.WriteLine($">> {pluginName} is created."); | ||
| #pragma warning restore SKEXP0040 | ||
| } | ||
| catch (Exception ex) | ||
| { | ||
| Console.WriteLine("Plugin creation failed. Message: {0}", ex.Message); | ||
| throw new AggregateException($"Plugin creation failed for {pluginName}", ex); | ||
| } | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
38 changes: 38 additions & 0 deletions
38
...et/samples/Concepts/Resources/Plugins/ApiManifestPlugins/AstronomyPlugin/apimanifest.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| { | ||
| "applicationName": "Astronomy Plugin", | ||
| "description": "This plugin accesses Nasa API to get Astronomy Picture of the Day and Microsoft Graph to get email messages from the user's mailbox.", | ||
| "publisher": { | ||
| "name": "publisher-name", | ||
| "contactEmail": "[email protected]" | ||
| }, | ||
| "apiDependencies": { | ||
| "microsoft.graph": { | ||
| "apiDescriptionUrl": "https://raw.githubusercontent.com/microsoftgraph/msgraph-metadata/master/openapi/v1.0/graphexplorer.yaml", | ||
| "requests": [ | ||
| { | ||
| "method": "Get", | ||
| "uriTemplate": "/me/messages" | ||
| } | ||
| ] | ||
| }, | ||
| "nasa": { | ||
| "apiDescriptionUrl": "https://raw.githubusercontent.com/zengin/openapi-directory/zengin/nasa/APIs/nasa.gov/apod/1.0.0/openapi.yaml", | ||
| "authorizationRequirements": { | ||
| "clientIdentifier": "some-uuid-here", | ||
| "access": [ | ||
| { | ||
| "type": "api_key", | ||
| "content": { | ||
| } | ||
| } | ||
| ] | ||
| }, | ||
| "requests": [ | ||
| { | ||
| "method": "Get", | ||
| "uriTemplate": "/apod" | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.