How-To: OpenAIAssistantAgent File Search

To add package dependencies from the command-line use the dotnet command:

dotnet add package Azure.Identity
dotnet add package Microsoft.Extensions.Configuration
dotnet add package Microsoft.Extensions.Configuration.Binder
dotnet add package Microsoft.Extensions.Configuration.UserSecrets
dotnet add package Microsoft.Extensions.Configuration.EnvironmentVariables
dotnet add package Microsoft.SemanticKernel
dotnet add package Microsoft.SemanticKernel.Agents.OpenAI --prerelease

If managing NuGet packages in Visual Studio, ensure Include prerelease is checked.

The project file (.csproj) should contain the following PackageReference definitions:

  <ItemGroup>
    <PackageReference Include="Azure.Identity" Version="<stable>" />
    <PackageReference Include="Microsoft.Extensions.Configuration" Version="<stable>" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Binder" Version="<stable>" />
    <PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" Version="<stable>" />
    <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="<stable>" />
    <PackageReference Include="Microsoft.SemanticKernel" Version="<latest>" />
    <PackageReference Include="Microsoft.SemanticKernel.Agents.OpenAI" Version="<latest>" />
  </ItemGroup>

The Agent Framework is experimental and requires warning suppression. This may addressed in as a property in the project file (.csproj):

  <PropertyGroup>
    <NoWarn>$(NoWarn);CA2007;IDE1006;SKEXP0001;SKEXP0110;OPENAI001</NoWarn>
  </PropertyGroup>

Additionally, copy the Grimms-The-King-of-the-Golden-Mountain.txt, Grimms-The-Water-of-Life.txt and Grimms-The-White-Snake.txt public domain content from Semantic Kernel LearnResources Project. Add these files in your project folder and configure to have them copied to the output directory:

  <ItemGroup>
    <None Include="Grimms-The-King-of-the-Golden-Mountain.txt">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Include="Grimms-The-Water-of-Life.txt">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Include="Grimms-The-White-Snake.txt">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
  </ItemGroup>

Start by creating a folder that will hold your script (.py file) and the sample resources. Include the following imports at the top of your .py file:

import asyncio
import os

from semantic_kernel.agents.open_ai import AzureAssistantAgent
from semantic_kernel.contents import StreamingAnnotationContent

Additionally, copy the Grimms-The-King-of-the-Golden-Mountain.txt, Grimms-The-Water-of-Life.txt and Grimms-The-White-Snake.txt public domain content from Semantic Kernel LearnResources Project. Add these files in your project folder.

Agents are currently unavailable in Java.

# OpenAI
dotnet user-secrets set "OpenAISettings:ApiKey" "<api-key>"
dotnet user-secrets set "OpenAISettings:ChatModel" "gpt-4o"

# Azure OpenAI
dotnet user-secrets set "AzureOpenAISettings:ApiKey" "<api-key>" # Not required if using token-credential
dotnet user-secrets set "AzureOpenAISettings:Endpoint" "https://lightspeed-team-shared-openai-eastus.openai.azure.com/"
dotnet user-secrets set "AzureOpenAISettings:ChatModelDeployment" "gpt-4o"

The following class is used in all of the Agent examples. Be sure to include it in your project to ensure proper functionality. This class serves as a foundational component for the examples that follow.

using System.Reflection;
using Microsoft.Extensions.Configuration;

namespace AgentsSample;

public class Settings
{
    private readonly IConfigurationRoot configRoot;

    private AzureOpenAISettings azureOpenAI;
    private OpenAISettings openAI;

    public AzureOpenAISettings AzureOpenAI => this.azureOpenAI ??= this.GetSettings<Settings.AzureOpenAISettings>();
    public OpenAISettings OpenAI => this.openAI ??= this.GetSettings<Settings.OpenAISettings>();

    public class OpenAISettings
    {
        public string ChatModel { get; set; } = string.Empty;
        public string ApiKey { get; set; } = string.Empty;
    }

    public class AzureOpenAISettings
    {
        public string ChatModelDeployment { get; set; } = string.Empty;
        public string Endpoint { get; set; } = string.Empty;
        public string ApiKey { get; set; } = string.Empty;
    }

    public TSettings GetSettings<TSettings>() =>
        this.configRoot.GetRequiredSection(typeof(TSettings).Name).Get<TSettings>()!;

    public Settings()
    {
        this.configRoot =
            new ConfigurationBuilder()
                .AddEnvironmentVariables()
                .AddUserSecrets(Assembly.GetExecutingAssembly(), optional: true)
                .Build();
    }
}

The quickest way to get started with the proper configuration to run the sample code is to create a .env file at the root of your project (where your script is run).

Configure the following settings in your .env file for either Azure OpenAI or OpenAI:

AZURE_OPENAI_API_KEY="..."
AZURE_OPENAI_ENDPOINT="https://<resource-name>.openai.azure.com/"
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME="..."
AZURE_OPENAI_API_VERSION="..."

OPENAI_API_KEY="sk-..."
OPENAI_ORG_ID=""
OPENAI_CHAT_MODEL_ID=""

Once configured, the respective AI service classes will pick up the required variables and use them during instantiation.

Agents are currently unavailable in Java.

Instantiate the Settings class referenced in the previous Configuration section. Use the settings to also create an AzureOpenAIClient that will be used for the Agent Definition as well as file-upload and the creation of a VectorStore.

Settings settings = new();

AzureOpenAIClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(new AzureCliCredential(), new Uri(settings.AzureOpenAI.Endpoint));

The class method setup_resources() on the Assistant Agent handles creating the client and returning it and the model to use based on the desired configuration. Pydantic settings are used to load environment variables first from environment variables or from the .env file. One may pass in the api_key, api_version, deployment_name or endpoint, which will take precedence over any environment variables configured.

# Create the client using Azure OpenAI resources and configuration
client, model = AzureAssistantAgent.setup_resources()

Agents are currently unavailable in Java.

Use the AzureOpenAIClient to access a VectorStoreClient and create a VectorStore.

Console.WriteLine("Creating store...");
VectorStoreClient storeClient = client.GetVectorStoreClient();
CreateVectorStoreOperation operation = await storeClient.CreateVectorStoreAsync(waitUntilCompleted: true);
string storeId = operation.VectorStoreId;

def get_filepath_for_filename(filename: str) -> str:
    base_directory = os.path.join(
        os.path.dirname(os.path.dirname(os.path.realpath(__file__))),
        "resources",
    )
    return os.path.join(base_directory, filename)

# Upload the files to the client
file_ids: list[str] = []
for path in [get_filepath_for_filename(filename) for filename in filenames]:
    with open(path, "rb") as file:
        file = await client.files.create(file=file, purpose="assistants")
        file_ids.append(file.id)

# Get the file search tool and resources
file_search_tools, file_search_tool_resources = AzureAssistantAgent.configure_file_search_tool(file_ids=file_ids)

Agents are currently unavailable in Java.

private static readonly string[] _fileNames =
    [
        "Grimms-The-King-of-the-Golden-Mountain.txt",
        "Grimms-The-Water-of-Life.txt",
        "Grimms-The-White-Snake.txt",
    ];

filenames = [
    "Grimms-The-King-of-the-Golden-Mountain.txt",
    "Grimms-The-Water-of-Life.txt",
    "Grimms-The-White-Snake.txt",
]

Agents are currently unavailable in Java.

Now upload those files and add them to the Vector Store by using the previously created VectorStoreClient clients to upload each file with a OpenAIFileClient and add it to the Vector Store, preserving the resulting File References.

Dictionary<string, OpenAIFile> fileReferences = [];

Console.WriteLine("Uploading files...");
OpenAIFileClient fileClient = client.GetOpenAIFileClient();
foreach (string fileName in _fileNames)
{
    OpenAIFile fileInfo = await fileClient.UploadFileAsync(fileName, FileUploadPurpose.Assistants);
    await storeClient.AddFileToVectorStoreAsync(storeId, fileInfo.Id, waitUntilCompleted: true);
    fileReferences.Add(fileInfo.Id, fileInfo);
}

Agents are currently unavailable in Java.

We will utilize the AzureOpenAIClient again as part of creating the OpenAIAssistantAgent:

Console.WriteLine("Defining assistant...");
Assistant assistant =
    await assistantClient.CreateAssistantAsync(
        settings.AzureOpenAI.ChatModelDeployment,
        name: "SampleAssistantAgent",
        instructions:
                """
                The document store contains the text of fictional stories.
                Always analyze the document store to provide an answer to the user's question.
                Never rely on your knowledge of stories not included in the document store.
                Always format response using markdown.
                """,
        enableFileSearch: true,
        vectorStoreId: storeId);

// Create agent
OpenAIAssistantAgent agent = new(assistant, assistantClient);

# Create the assistant definition
definition = await client.beta.assistants.create(
    model=model,
    instructions="""
        The document store contains the text of fictional stories.
        Always analyze the document store to provide an answer to the user's question.
        Never rely on your knowledge of stories not included in the document store.
        Always format response using markdown.
        """,
    name="SampleAssistantAgent",
    tools=file_search_tools,
    tool_resources=file_search_tool_resources,
)

# Create the agent using the client and the assistant definition
agent = AzureAssistantAgent(
    client=client,
    definition=definition,
)

Agents are currently unavailable in Java.

Console.WriteLine("Creating thread...");
AssistantThread thread = await assistantClient.CreateThreadAsync();

Console.WriteLine("Ready!");

try
{
    bool isComplete = false;
    do
    {
        // Processing occurs here
    } while (!isComplete);
}
finally
{
    Console.WriteLine();
    Console.WriteLine("Cleaning-up...");
    await Task.WhenAll(
        [
            assistantClient.DeleteThreadAsync(thread.Id),
            assistantClient.DeleteAssistantAsync(assistant.Id),
            storeClient.DeleteVectorStoreAsync(storeId),
            ..fileReferences.Select(fileReference => fileClient.DeleteFileAsync(fileReference.Key))
        ]);
}

print("Creating thread...")
thread_id = await agent.create_thread()

try:
    is_complete: bool = False
    while not is_complete:
        # Processing occurs here

finally:
    print("Cleaning up resources...")
    if agent is not None:
        [await agent.delete_file(file_id) for file_id in agent.file_search_file_ids]
        await agent.delete_thread(thread_id)
        await agent.delete()

Agents are currently unavailable in Java.

Console.WriteLine();
Console.Write("> ");
string input = Console.ReadLine();
if (string.IsNullOrWhiteSpace(input))
{
    continue;
}
if (input.Trim().Equals("EXIT", StringComparison.OrdinalIgnoreCase))
{
    isComplete = true;
    break;
}

await agent.AddChatMessageAsync(thread.Id, new ChatMessageContent(AuthorRole.User, input));
Console.WriteLine();

user_input = input("User:> ")
if not user_input:
    continue

if user_input.lower() == "exit":
    is_complete = True
    break

await agent.add_chat_message(
    thread_id=thread_id, message=ChatMessageContent(role=AuthorRole.USER, content=user_input)
)

Agents are currently unavailable in Java.

private static string ReplaceUnicodeBrackets(this string content) =>
    content?.Replace('【', '[').Replace('】', ']');

# No special handling required.

Agents are currently unavailable in Java.

List<StreamingAnnotationContent> footnotes = [];
await foreach (StreamingChatMessageContent chunk in agent.InvokeStreamingAsync(thread.Id))
{
    // Capture annotations for footnotes
    footnotes.AddRange(chunk.Items.OfType<StreamingAnnotationContent>());

    // Render chunk with replacements for unicode brackets.
    Console.Write(chunk.Content.ReplaceUnicodeBrackets());
}

Console.WriteLine();

// Render footnotes for captured annotations.
if (footnotes.Count > 0)
{
    Console.WriteLine();
    foreach (StreamingAnnotationContent footnote in footnotes)
    {
        Console.WriteLine($"#{footnote.Quote.ReplaceUnicodeBrackets()} - {fileReferences[footnote.FileId!].Filename} (Index: {footnote.StartIndex} - {footnote.EndIndex})");
    }
}

footnotes: list[StreamingAnnotationContent] = []
async for response in agent.invoke_stream(thread_id=thread_id):
    footnotes.extend([item for item in response.items if isinstance(item, StreamingAnnotationContent)])

    print(f"{response.content}", end="", flush=True)

print()

if len(footnotes) > 0:
    for footnote in footnotes:
        print(
            f"\n`{footnote.quote}` => {footnote.file_id} "
            f"(Index: {footnote.start_index} - {footnote.end_index})"
        )

Agents are currently unavailable in Java.

using Azure.AI.OpenAI;
using Azure.Identity;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Agents;
using Microsoft.SemanticKernel.Agents.OpenAI;
using Microsoft.SemanticKernel.ChatCompletion;
using OpenAI.Assistants;
using OpenAI.Files;
using OpenAI.VectorStores;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

namespace AgentsSample;

public static class Program
{
    private static readonly string[] _fileNames =
        [
            "Grimms-The-King-of-the-Golden-Mountain.txt",
            "Grimms-The-Water-of-Life.txt",
            "Grimms-The-White-Snake.txt",
        ];

    /// <summary>
    /// The main entry point for the application.
    /// </summary>
    /// <returns>A <see cref="Task"/> representing the asynchronous operation.</returns>
    public static async Task Main()
    {
        // Load configuration from environment variables or user secrets.
        Settings settings = new();

        // Initialize the clients
        AzureOpenAIClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(new AzureCliCredential(), new Uri(settings.AzureOpenAI.Endpoint));
        //OpenAIClient client = OpenAIAssistantAgent.CreateOpenAIClient(new ApiKeyCredential(settings.OpenAI.ApiKey)));
        AssistantClient assistantClient = client.GetAssistantClient();
        OpenAIFileClient fileClient = client.GetOpenAIFileClient();
        VectorStoreClient storeClient = client.GetVectorStoreClient();

        // Create the vector store
        Console.WriteLine("Creating store...");
        CreateVectorStoreOperation operation = await storeClient.CreateVectorStoreAsync(waitUntilCompleted: true);
        string storeId = operation.VectorStoreId;

        // Upload files and retain file references.
        Console.WriteLine("Uploading files...");
        Dictionary<string, OpenAIFile> fileReferences = [];
        foreach (string fileName in _fileNames)
        {
            OpenAIFile fileInfo = await fileClient.UploadFileAsync(fileName, FileUploadPurpose.Assistants);
            await storeClient.AddFileToVectorStoreAsync(storeId, fileInfo.Id, waitUntilCompleted: true);
            fileReferences.Add(fileInfo.Id, fileInfo);
        }

        // Define assistant
        Console.WriteLine("Defining assistant...");
        Assistant assistant =
            await assistantClient.CreateAssistantAsync(
                settings.AzureOpenAI.ChatModelDeployment,
                name: "SampleAssistantAgent",
                instructions:
                        """
                        The document store contains the text of fictional stories.
                        Always analyze the document store to provide an answer to the user's question.
                        Never rely on your knowledge of stories not included in the document store.
                        Always format response using markdown.
                        """,
                enableFileSearch: true,
                vectorStoreId: storeId);

        // Create agent
        OpenAIAssistantAgent agent = new(assistant, assistantClient);

        // Create the conversation thread
        Console.WriteLine("Creating thread...");
        AssistantThread thread = await assistantClient.CreateThreadAsync();

        Console.WriteLine("Ready!");

        try
        {
            bool isComplete = false;
            do
            {
                Console.WriteLine();
                Console.Write("> ");
                string input = Console.ReadLine();
                if (string.IsNullOrWhiteSpace(input))
                {
                    continue;
                }
                if (input.Trim().Equals("EXIT", StringComparison.OrdinalIgnoreCase))
                {
                    isComplete = true;
                    break;
                }

                await agent.AddChatMessageAsync(thread.Id, new ChatMessageContent(AuthorRole.User, input));
                Console.WriteLine();

                List<StreamingAnnotationContent> footnotes = [];
                await foreach (StreamingChatMessageContent chunk in agent.InvokeStreamingAsync(thread.Id))
                {
                    // Capture annotations for footnotes
                    footnotes.AddRange(chunk.Items.OfType<StreamingAnnotationContent>());

                    // Render chunk with replacements for unicode brackets.
                    Console.Write(chunk.Content.ReplaceUnicodeBrackets());
                }

                Console.WriteLine();

                // Render footnotes for captured annotations.
                if (footnotes.Count > 0)
                {
                    Console.WriteLine();
                    foreach (StreamingAnnotationContent footnote in footnotes)
                    {
                        Console.WriteLine($"#{footnote.Quote.ReplaceUnicodeBrackets()} - {fileReferences[footnote.FileId!].Filename} (Index: {footnote.StartIndex} - {footnote.EndIndex})");
                    }
                }
            } while (!isComplete);
        }
        finally
        {
            Console.WriteLine();
            Console.WriteLine("Cleaning-up...");
            await Task.WhenAll(
                [
                    assistantClient.DeleteThreadAsync(thread.Id),
                    assistantClient.DeleteAssistantAsync(assistant.Id),
                    storeClient.DeleteVectorStoreAsync(storeId),
                    ..fileReferences.Select(fileReference => fileClient.DeleteFileAsync(fileReference.Key))
                ]);
        }
    }

    private static string ReplaceUnicodeBrackets(this string content) =>
        content?.Replace('【', '[').Replace('】', ']');
}

import asyncio
import os

from semantic_kernel.agents.open_ai import AzureAssistantAgent
from semantic_kernel.contents import StreamingAnnotationContent

"""
The following sample demonstrates how to create a simple,
OpenAI assistant agent that utilizes the vector store
to answer questions based on the uploaded documents.
"""


def get_filepath_for_filename(filename: str) -> str:
    base_directory = os.path.join(
        os.path.dirname(os.path.dirname(os.path.realpath(__file__))),
        "resources",
    )
    return os.path.join(base_directory, filename)


filenames = [
    "Grimms-The-King-of-the-Golden-Mountain.txt",
    "Grimms-The-Water-of-Life.txt",
    "Grimms-The-White-Snake.txt",
]


async def main():
    # Create the client using Azure OpenAI resources and configuration
    client, model = AzureAssistantAgent.setup_resources()

    # Upload the files to the client
    file_ids: list[str] = []
    for path in [get_filepath_for_filename(filename) for filename in filenames]:
        with open(path, "rb") as file:
            file = await client.files.create(file=file, purpose="assistants")
            file_ids.append(file.id)

    vector_store = await client.beta.vector_stores.create(
        name="assistant_search",
        file_ids=file_ids,
    )

    # Get the file search tool and resources
    file_search_tools, file_search_tool_resources = AzureAssistantAgent.configure_file_search_tool(
        vector_store_ids=vector_store.id
    )

    # Create the assistant definition
    definition = await client.beta.assistants.create(
        model=model,
        instructions="""
            The document store contains the text of fictional stories.
            Always analyze the document store to provide an answer to the user's question.
            Never rely on your knowledge of stories not included in the document store.
            Always format response using markdown.
            """,
        name="SampleAssistantAgent",
        tools=file_search_tools,
        tool_resources=file_search_tool_resources,
    )

    # Create the agent using the client and the assistant definition
    agent = AzureAssistantAgent(
        client=client,
        definition=definition,
    )

    print("Creating thread...")
    thread = await client.beta.threads.create()

    try:
        is_complete: bool = False
        while not is_complete:
            user_input = input("User:> ")
            if not user_input:
                continue

            if user_input.lower() == "exit":
                is_complete = True
                break

            await agent.add_chat_message(thread_id=thread.id, message=user_input)

            footnotes: list[StreamingAnnotationContent] = []
            async for response in agent.invoke_stream(thread_id=thread.id):
                footnotes.extend([item for item in response.items if isinstance(item, StreamingAnnotationContent)])

                print(f"{response.content}", end="", flush=True)

            print()

            if len(footnotes) > 0:
                for footnote in footnotes:
                    print(
                        f"\n`{footnote.quote}` => {footnote.file_id} "
                        f"(Index: {footnote.start_index} - {footnote.end_index})"
                    )

    finally:
        print("\nCleaning up resources...")
        [await client.files.delete(file_id) for file_id in file_ids]
        await client.beta.threads.delete(thread.id)
        await client.beta.assistants.delete(agent.id)


if __name__ == "__main__":
    asyncio.run(main())

You may find the full code, as shown above, in our repo.

Agents are currently unavailable in Java.